{"id":"ae652a7a-f68b-4151-a9e3-73fb1c631376","arxiv_id":"2506.23590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CAI reduces object hallucination in LVLMs by injecting caption-query attention patterns into selected attention heads at inference time.","lead":"The paper introduces CAI, a training-free method that reduces object hallucination in vision-language models by steering attention heads toward patterns seen when the model is asked to caption an image. The method adds little inference cost and reports accuracy gains on four hallucination benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shift vector in Eq. 9 uses unmasked attention outputs, so it may encode caption/non-caption text differences rather than visual signal; the claimed visual-perception mechanism is not yet supported.","rationale":"The reader's weakest assumption was that the average shift vector is a safe image-agnostic steering direction. I agree that generalization across images and queries is a risk, but I identify a more specific and more easily testable flaw: the shift vector is not even constrained to visual information, despite the probing procedure being explicitly designed to isolate visual signal. This is directly evidenced by the mismatch between Eq. 5-6 (which masks text-token attention) and Eq. 9 (which uses unmasked outputs). The proposed concrete test would settle whether the mechanism is genuinely visual or partly textual. If the latter, the paper's central explanation would need revision, though the empirical method might still be useful. I do not recommend rejecting the paper outright because the reported improvements could still hold with a different mechanism, and the test is straightforward. The conditional acceptance recommended by the reader remains appropriate, but the acceptance conditions should include this visual-only shift ablation in addition to held-out hyperparameter validation and artifact release.","tokens_in":18113,"tokens_out":5374,"duration_ms":68122,"concrete_test":"Recompute the shift vectors from the visual-only, text-masked attention outputs defined in Eq. 6, i.e., S_vis^{(l,h)} = (1/B) Σ_b (\\hat{O}_b^{(l,h)} - \\hat{O}'_b^{(l,h)}), using the same 1000 LLaVA pretraining pairs and the same top-K heads selected by the masked classifiers. Run CAI on the POPE Adversarial set and CHAIR with this visual-only shift and with α=1.5, K=100. If S_vis yields comparable gains to the original S, the visual mechanism is supported; if the original S works but S_vis fails, the method is relying on text-prior differences. Also report the cosine similarity between S and S_vis averaged over the intervened heads.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CAI steers attention heads toward a vision-centric state by adding a shift vector S^{(l,h)} learned from caption versus non-caption queries. The probing stage (Eq. 5-6) deliberately masks text-token attention to isolate visual information, yet the shift vector itself is computed in Eq. 9 from the *unmasked* original attention outputs O_b^{(l,h)} and O'^{(l,h)}. Since caption queries and non-caption queries differ in wording (e.g., 'Describe this image in detail' vs. 'Is there a helmet?'), the average difference O_b - O'_b can include large linguistic-prior components—such as attention to tokens correlated with 'image', 'photo', 'describe', or generic sentence-structure differences—rather than purely visual grounding. This vector is then added to every token's attention output in Eq. 10 for arbitrary test inputs. Consequently, the observed hallucination reduction could arise from injecting a caption-style textual bias (e.g., making answers more generic or conservative) instead of from enhancing fine-grained visual perception, which is the paper's stated mechanism. This is a load-bearing risk because the headline contribution is a *visual-attention* intervention; if the shift is dominated by text-query differences, the explanation for the gains is unsupported, even if the empirical numbers reproduce.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAI, a training-free, plug-and-play intervention for reducing object hallucination in large vision-language models. The authors observe that caption-style queries elicit stronger visual attention than non-caption queries in certain attention heads, and they exploit this by (1) searching over a small set of caption queries to find one with minimal attention shift, (2) training per-head SVM classifiers on masked attention outputs to identify caption-sensitive heads and computing average attention-output shift vectors from caption versus non-caption queries, and (3) adding scaled shift vectors to the selected heads during inference. The method is evaluated on POPE, MME, CHAIR, and MMHal-Bench using LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT, reporting consistent improvements over several training-free baselines with minimal additional latency.","tokens_in":18520,"tokens_out":8111,"duration_ms":91401,"significance":"If the reported gains are reproducible, CAI would be a useful lightweight contribution to the hallucination-mitigation toolbox: it requires no LVLM fine-tuning, has a low inference overhead documented in Table 4, and is applied across three models and four benchmarks from a single set of pretrain-derived shift vectors. The paper's strengths are its clear three-stage formulation, the cross-model application of the same shift vectors, and the explicit latency comparison against contrastive-decoding baselines. However, the manuscript provides no code or random seeds, the two intervention hyperparameters are tuned on a test benchmark, and the mechanistic claim that the shift vector is vision-centric is undermined by the use of unmasked attention outputs in Eq. 9. The consistent direction of the gains is encouraging, but the current evidence is not yet statistically or methodologically strong enough for the SOTA claims made in the paper.","major_comments":[{"comment":"The hyperparameters α and K are selected by maximizing accuracy on the MS-COCO Adversarial POPE test set in Figure 7 and Tables 5–6, and the same test set is then used to report the adversarial POPE row of Table 1; consequently, the headline adversarial numbers are fitted values rather than independent evaluations. This is load-bearing for the SOTA claim on POPE. Please report a validation-based selection procedure (for example, tuning on a held-out split or reporting the full grid over all settings) and add error bars or significance tests, since no repeated runs are reported anywhere in the paper.","section":"§5.4, Appendix D, Table 1"},{"comment":"The probing stage explicitly masks text-token attention in Eqs. 5–7 to isolate visual information, but the shift vector S^{(l,h)} in Eq. 9 is computed from the unmasked original attention outputs O_b^{(l,h)} and O'^{(l,h)}. Because caption and non-caption queries differ in wording and length, this average difference can encode language-prior components—such as attention to query tokens or generic sentence-structure differences—so the claim that S is a 'vision-centric optimization direction' is not yet supported. Please compute the shift from the masked outputs used for probing, or provide a control experiment (for example, length- and topic-matched non-caption queries, or a token-level decomposition of S) that isolates the visual contribution.","section":"§3.3, Eq. 9"},{"comment":"No error bars, confidence intervals, or significance tests are reported, and several claimed gains are small relative to the sample sizes: MMHal-Bench contains only 96 questions, and the VH-rate difference for Qwen-VL-Chat is 38.0 versus 38.4 for two of the baselines, while the CHAIR results are reported for a single model. Without repeated runs or an uncertainty estimate, the paper's central claim of consistent improvements is not statistically grounded. Please report multiple decoding seeds or bootstrap confidence intervals, or explicitly state that the gains are descriptive rather than statistically tested.","section":"§4.2, Figure 4, Table 3"},{"comment":"The paper states that the images used for probe and shift computation are 'outside the domain of the benchmark datasets,' but the probe samples come from the LLaVA pretraining dataset, which is built from image-caption corpora that include MS-COCO, while POPE and CHAIR evaluate on MS-COCO images. The claim of cross-domain discrepancy is therefore not justified without specifying the exact image splits (for example, COCO train versus val) and any overlap filtering. Please clarify the data sources and assess whether the COCO-based pretraining distribution materially overlaps the evaluation benchmarks.","section":"§4.2, Implementation Details"}],"minor_comments":[{"comment":"Please define the norm used to turn the attention-weight shift matrix into the scalar minimized in Eq. 4; as written, Eq. 4 appears to minimize a matrix-valued quantity over a finite index set.","section":"§3.2, Eqs. 3–4"},{"comment":"The main POPE experiments are described as using direct sampling decoding, but the ablation in Figure 7 and Appendix D is described as using greedy decoding; please clarify which decoding scheme is used for hyperparameter selection and for each reported result.","section":"§A.2 and §5.4"},{"comment":"The method is described as 'training-free,' but Stage 2 trains an SVM classifier per attention head; please qualify this as 'no LVLM fine-tuning' or otherwise clarify the terminology.","section":"§1 and §3.3"},{"comment":"Please specify the dimension of S^{(l,h)} and state explicitly whether the same shift vector is added to every token position; the current notation suggests a last-token vector broadcast over all positions, which should be stated.","section":"§3.4, Eq. 10"},{"comment":"Minor typographical issues include 'methodes' in §4.1, 'performence' in §4.2, and 'A verage' in Table 7; these should be corrected.","section":"Throughout"},{"comment":"Please use consistent baseline labels with the other tables ('Greedy' versus 'Regular') and state explicitly that the VH Rate column is a percentage.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main concerns are the test-set tuning of α and K and the mismatch between the masked probing and the unmasked shift computation. Both are fixable within a revision, but as written the SOTA claims are stronger than the evidence supports. If the authors cannot provide a validation-based selection or a masked-shift control, the central mechanism claim should be downgraded accordingly. The paper is otherwise within scope and represents a useful incremental contribution to training-free hallucination mitigation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—here's my take on arXiv:2506.23590. The paper proposes CAI, a training-free attention intervention that adds precomputed attention-output shifts derived from caption vs. non-caption queries to selected heads at inference. The genuinely new piece is using caption/non-caption query differences to locate and steer attention heads; that observation is not in ITI or PAI. The method is cheap, plug-and-play, and the tables show consistent gains across LLaVA-1.5, Qwen-VL-Chat, and LLaVA-NeXT on POPE, MME, CHAIR, and MMHal-Bench. That is real empirical signal, and the cross-model transfer from LLaVA pretraining data is a plus.\n\nThe soft spots are proportionate. First, the mechanism story is under-supported. The probing stage masks text-token attention (Eq. 5–6) to isolate visual information, but the shift vector itself (Eq. 9) is computed from the unmasked attention outputs. So the average difference between caption and non-caption queries can include linguistic-prior components—word choices like 'describe' vs. 'is there'—not just visual grounding. The paper calls S a 'vision-centric optimization direction,' but that label is not established. The observed hallucination reduction may partly come from injecting a caption-style textual bias. This is the central explanation, so it should be fixed or softened. Second, α and K were selected on the MS-COCO Adversarial POPE test set, which inflates those specific numbers; no error bars or significance tests are reported, and MMHal-Bench is only 96 questions. Third, no code or data is released, so the head selection and shift vectors are hard to reproduce.\n\nThe consistent direction of improvements across models and benchmarks is evidence the method does something useful. The stress-test worry is a fair one, not a manufactured flaw. If the authors rerun with held-out hyperparameters and show the shift vector's effect isn't just a text-prior injection, the paper would be solid.\n\nWho this is for: people working on LVLM hallucination and inference-time interventions. It deserves a serious referee—the idea is simple, cheap, and mostly works—but the mechanism claim needs revision and the eval needs rigor.\n\nI'd send it to review with major-revision expectations.","headline":"A cheap, training-free attention intervention that consistently reduces LVLM hallucination, but the stated visual mechanism is undercut by the shift vector's unmasked computation and the headline numbers are partly fitted.","tokens_in":18916,"tokens_out":2739,"would_cite":false,"duration_ms":30732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAI, a training-free method, mitigates object hallucination in large vision-language models by adding precomputed caption-query attention shifts to a handful of caption-sensitive attention heads at inference time.","keywords":["object hallucination","large vision-language models","attention intervention","caption-sensitive attention heads","inference-time intervention","hallucination mitigation","training-free method"],"falsifier":"Collect a held-out set of images, compute the per-image caption-to-noncaption attention-output shift, and check whether hallucination improvements persist on images whose per-image shift points opposite to CAI's precomputed shift; if they do, the fixed steering direction is not the active ingredient.","tokens_in":17945,"feed_emoji":"👁️","tokens_out":11485,"duration_ms":100562,"temperature":0.7,"pith_summary":"Large vision-language models routinely invent objects that are not in the image, and existing fixes either need expensive training data or slow down inference. This paper shows that the same models pay noticeably more attention to visual tokens when the prompt asks for a caption than when it asks a direct question, and that this gap can be reused as a steering signal. The proposed method, CAI, finds the attention heads that differ most between caption and non-caption queries, precomputes their output shift on a small set of pretraining samples, and adds that shift back at inference time. On POPE, MME, CHAIR, and MMHal-Bench, the intervention improves hallucination metrics across three open models with almost no added latency. The claim is that a fixed, image-agnostic shift toward the caption state is enough to restore fine-grained visual perception.","feed_headline":"One attention shift from caption queries cuts object hallucination","feed_subtitle":"Adding precomputed caption-query shifts to top attention heads improves four hallucination benchmarks without retraining.","key_machinery":"The central object is the caption-sensitive attention head, defined as a head whose output at the last token differs strongly between caption and non-caption queries after text-related attention scores are masked. The method uses per-head binary classifiers to rank heads, then computes an attention output shift vector $S^{(l,h)}$ as the average of (caption output minus non-caption output) over a probing set. At inference this shift is added to the top-$K$ heads' outputs, scaled by a strength parameter $\\alpha$, through the gated update $H^{l+1} = H^l + \\sum_h (O^{(l,h)} + I^{(l,h)} \\alpha S^{(l,h)}) W_o^l$. The shift is the load-bearing artifact: it encodes where the model's own visual attention goes when it is asked to describe rather than interrogate an image.","core_discovery":"CAI's discovery is that caption queries put vision-language models into a reliably more visual state: 65.92% of attention heads and 30 of 32 layers increase their visual attention when the prompt is a caption. The paper treats this state as a target and steers non-caption inference toward it. Concretely, it trains one binary classifier per head to separate caption from non-caption query outputs, keeps the top-K heads by classification accuracy, and computes the average output difference between the two query types over 1,000 pretraining pairs. At inference, that difference is added to the selected heads' outputs before the output projection. The empirical claim is that this single precomputed shift makes LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT hallucinate less on both discriminative and generative benchmarks while leaving other capabilities largely intact.","pith_inferences":["The paper leaves untested whether adapting the shift per image or per query would do better; if the pretraining-average shift is not the optimal direction, a lightweight per-image estimate could improve gains further.","The same 'expert query' pattern could transfer to other failure modes: find a query that reliably activates the desired behavior, identify the heads that separate it from a generic query, and steer those heads at inference.","Because CAI modifies attention outputs rather than logits, combining it with contrastive decoding is a natural next experiment the paper does not run."],"forward_implications":["On POPE, CAI raises accuracy by 3.30 to 6.58 points and F1 by 3.57 to 8.10 points over the vanilla model across LLaVA-1.5-7b, Qwen-VL-Chat, and LLaVA-NeXT.","On the MME hallucination subset, total scores increase by 50.0 to 76.2 points depending on the model, and on the full MME set LLaVA-1.5-7b improves on 13 of 14 tasks.","On MS-COCO CHAIR, sentence-level hallucination (CHAIRs) drops by 3.6 points and instance-level hallucination (CHAIRi) by 1.27 points, even when the prompt is itself a caption-style request.","On MMHal-Bench, the hallucination rate falls from 63.5% to 51.0% for LLaVA-1.5-7b and from 41.1% to 38.0% for Qwen-VL-Chat, while the informativeness score rises.","Inference cost stays near the vanilla model: time to first token changes from 99.8 ms to 102.2 ms and time per output token from 36.0 ms to 36.5 ms on LLaVA-1.5-7b, unlike contrastive decoding methods that multiply latency."],"supporting_citations":[{"why":"Supplies the inference-time intervention recipe that CAI adapts from text models to vision-language models.","marker":"Li et al., 2024"},{"why":"Provides the MS-COCO images used for the quantitative attention analysis and for computing the shift vectors.","marker":"Lin et al., 2014"},{"why":"Baseline contrastive decoding method whose accuracy and latency are compared with CAI.","marker":"Leng et al., 2024"},{"why":"Baseline attention-head intervention method that CAI is compared against on hallucination benchmarks.","marker":"Liu et al., 2024d"},{"why":"Baseline latent-space steering method used for comparison.","marker":"Liu et al., 2024c"},{"why":"The primary base model LLaVA-1.5-7b on which CAI is developed and evaluated.","marker":"Liu et al., 2024a"}],"fun_headline_variants":["Caption attention shift reduces object hallucination in LVLMs","Training-free fix: use caption attention to curb hallucination","Steer LVLMs with caption queries to stop object hallucination","Caption-aware attention tweak cuts object hallucination","Zero-training plug-in: caption attention shift cuts hallucination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average attention-output difference between caption and non-caption queries, computed once on 1,000 pretraining samples, is a safe image-agnostic steering direction for every test image and query.","fun_headline_variants_meta":{"raw":{"variants":["Caption attention shift reduces object hallucination in LVLMs","Training-free fix: use caption attention to curb hallucination","Steer LVLMs with caption queries to stop object hallucination","Caption-aware attention tweak cuts object hallucination","Zero-training plug-in: caption attention shift cuts hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3709,"prompt_tokens":877,"completion_tokens":2832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2751}},"tokens_in":493,"tokens_out":2832,"duration_ms":20696,"temperature":1.0,"reasoning_tokens":2751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:36:19.387120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of images, compute the per-image caption-to-noncaption attention-output shift, and check whether hallucination improvements persist on images whose per-image shift points opposite to CAI's precomputed shift; if they do, the fixed steering direction is not the active ingredient.","supporting_citations":[],"review_version":1}