{"id":"e54ada44-6831-4b41-859e-f99d04902d20","arxiv_id":"2506.11073","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An inference-time attention-shift intervention aligns non-English queries' cross-modal attention with English, cutting multilingual object hallucination in LVLMs on POPE and MME.","lead":"CLAIM is a near training-free method that reduces multilingual object hallucination in vision-language models by shifting attention patterns of non-English queries toward English at inference. It reports average gains of about 13.6% on POPE and substantial gains on MME hallucination subsets across several languages and three open models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline POPE gain is partly in-sample: hyperparameters α and K are tuned on the POPE-COCO popular subset (Appendix A), which is included in the reported POPE average (Table 1).","rationale":"We read the paper as claiming a training-free attention intervention that transfers English visual perception to non-English queries, with measured gains on POPE and MME. The shift-vector transfer assumption (Eq. 9 applied to arbitrary queries) is plausible and is directly tested by the POPE and MME evaluations: different query types (yes/no presence, attribute, counting) all improve, and MME was not used to construct the shift. The weakest point in the evidence is instead the selection protocol: the two free parameters α and K are tuned on POPE-COCO popular, and that same subset is part of the headline POPE average. This is an admitted in-sample evaluation component, not a speculative risk. The reader's verdict already marks the paper CONDITIONAL; our analysis supports that verdict and does not move it. We therefore recommend UNCHANGED: the concern is real but the MME results provide partially independent support, so rejection would be too strong. If the proposed re-computation shows the POPE gain is largely driven by the tuned subset, the correct verdict would become REJECT or a more stringent CONDITIONAL with the numeric claim revised.","tokens_in":20426,"tokens_out":10408,"duration_ms":120231,"concrete_test":"Recompute the aggregate POPE improvement after removing all POPE-COCO popular rows from Table 1 for both models; if the average gain falls below the reported 13.56% by a large margin (e.g., >3 points), the headline is selection-inflated. As a stronger check, rerun the full protocol with α and K chosen on a held-out validation set of ~200 COCO images that are disjoint from POPE and MME evaluation images, then report the out-of-sample POPE and MME averages. If the out-of-sample gains remain comparable, the tuning concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's headline POPE improvement is not a clean out-of-sample number. Appendix A states that the intervention intensity α (search space {0.5,...,4.5}) and the number of heads K ({50,...,300}) are selected by sequential search on the POPE-COCO popular subset. Table 1 then reports the POPE-COCO popular rows as part of the average over all POPE settings that backs the 13.56% abstract claim. The condition that must hold for the central claim to be supported is that the intervention's benefit is not concentrated in the exact subset used for model selection. The paper does not report a validation set disjoint from the evaluation benchmarks, nor error bars over seeds, so we cannot tell whether the tuned subset is an outlier. The MME numbers are less affected because MME was not used for tuning, but the same α and K carry over to MME, so MME is not a fully independent selection of hyperparameters. If the POPE average drops materially when the popular subset is excluded, the headline quantitative claim is optimistic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLAIM, an inference-time intervention that mitigates multilingual object hallucination in LVLMs by aligning cross-modal attention patterns of non-English queries to those of English. The method first identifies language-specific cross-modal attention heads using linear probes, then estimates language shift vectors as the mean English-minus-target difference in attention outputs over 1,000 COCO images, and finally adds scaled shift vectors to the selected heads during inference. Experiments on POPE and MME with LLaVA-1.5 and Qwen-VL-Chat report consistent improvements across five to six non-English languages, with an average improvement of 13.56% on POPE and 21.75% on MME hallucination subsets claimed in the abstract.","tokens_in":20635,"tokens_out":5985,"duration_ms":56459,"significance":"If the central claim holds, the paper offers a valuable, near training-free alternative to fine-tuning approaches for multilingual hallucination, with a mechanistic explanation in terms of cross-modal attention divergence. The evaluation is external (POPE, MME), uses multiple models and languages, and includes comparisons to VCD and PAI; the method is falsifiable and relatively cheap to apply. The attention-level analysis and intervention analysis are thoughtful and help interpret why the method works. The main weakness is that the headline POPE number is partly in-sample because the two hyperparameters are tuned on the POPE-COCO popular subset, which is part of the reported average; this needs to be addressed before the quantitative claims can be taken at face value.","major_comments":[{"comment":"The hyperparameters α and K are selected by sequential search on the POPE-COCO popular subset (Appendix A), and this same subset is included in the POPE average reported in Table 1 and used for the 13.56% improvement in the abstract. Because the tuning subset is part of the reported evaluation distribution, the headline POPE gain is partly in-sample. Please report the average excluding the COCO-popular rows, or perform hyperparameter selection on a held-out validation split disjoint from all evaluation benchmarks, and verify that the improvement is not concentrated in the tuned subset.","section":"Appendix A, Table 1, Abstract"},{"comment":"Head selection uses masked attention outputs bO (Eq. 6) that exclude text tokens, whereas the language shift vectors (Eq. 9) and the intervention (Eq. 10) act on the unmasked outputs O. The paper justifies this by preserving the original representation space, but it does not address whether the heads selected under masking are the right heads to intervene on with shifts computed from the unmasked distribution. Please provide an analysis (e.g., head-selection agreement between masked and unmasked outputs) or a theoretical argument that the mismatch does not undermine the intervention's mechanism.","section":"§3.2–§3.3"},{"comment":"No variance estimates or repeated-run results are reported, so it is unclear whether the performance gaps (which can be as small as a few points) are significant relative to run-to-run variation. Please report standard deviations over at least three seeds, or provide bootstrap confidence intervals, particularly for the hyperparameter robustness analysis in Figure 8, which currently shows only the tuning subset.","section":"§4, Table 1, Figure 8"}],"minor_comments":[{"comment":"The reported improvements (13.56%, 21.75%, 17.5%, 9.8%) should be clarified as absolute percentage points or relative improvements, and the exact set of languages and rows over which each average is computed should be specified.","section":"Abstract and §4.3"},{"comment":"The text contains typos: 'Rencent' (Introduction), 'Perfomance' (Table 2 caption), 'theses' (§5.4), and 'a issue' (§2); please proofread.","section":"Throughout"},{"comment":"The Avg columns in Table 1 exclude the English column; please state this explicitly in the table caption or in the text.","section":"Table 1"},{"comment":"The x-axis labels (e.g., 'Total Average') and y-axis units should be defined in the caption so the reader can interpret the scale.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a multimodal/ML venue. The main concern is the in-sample tuning for the headline POPE result; if the authors fix that and add variance estimates, the paper could be acceptable. The comparisons to VCD and PAI are adequate, though a comparison with a translation-based baseline (e.g., translating non-English queries to English) would strengthen the claim that the intervention itself is responsible for the improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first training-free mitigation for multilingual object hallucination I know of, and the central mechanism—probe-selected cross-modal attention heads plus language shift vectors—is genuinely new as a combination. The empirical pattern is consistent enough that the method is probably real. But the 13.56% POPE headline is partly in-sample: alpha and K are tuned on the POPE-COCO popular subset, which is part of the reported POPE average. That does not kill the paper, but it makes the headline number optimistic until someone re-runs with a held-out validation split.\n\nWhat the paper does well: the method is cheap, plug-and-play, and mechanically motivated—non-English queries shift cross-modal attention away from relevant image regions, so adding an English-minus-target shift to the right heads is a sensible intervention. The analysis showing attention divergence concentrated in intermediate layers is useful. The results span three models, seven languages, and two benchmark families, and MME provides some genuinely held-out support because it was not used for tuning (though alpha and K carry over, so it is not fully independent). The training-size ablation and cross-language shift transfer experiments are good signs. The authors also state the obvious limitations: open models only, internal layer access required, and possible reinforcement of English-centric bias.\n\nSoft spots, in order of softness. First, the in-sample tuning above: no validation split disjoint from the benchmarks, no error bars, no seeds. Second, the shift vector is estimated from one template ('What is it in the image?') and then applied to yes/no, attribute, and counting queries. That transfer assumption is untested but not implausible. Third, no code or data is shipped, so the reported numbers are not auditable—reproducibility matters here because translation quality and benchmark construction are manual steps. These are all fixable, not fatal.\n\nMy take: this deserves a serious referee. The core idea is novel enough and the evidence strong enough to warrant replication. I would ask the authors to add a held-out hyperparameter split, report variance across seeds, and release code plus the translated benchmarks before trusting the exact figures. I'd cite this if I worked on hallucination mitigation.","headline":"Genuinely new inference-time method with a plausible mechanism, but the headline POPE gain is partly in-sample via hyperparameter tuning, so the numbers are promising rather than proven.","tokens_in":21185,"tokens_out":2386,"would_cite":true,"duration_ms":24947,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that multilingual object hallucination in LVLMs can be mitigated at inference time by shifting the attention outputs of language-specific cross-modal heads toward English, with average gains of 13.56 points on POPE and…","keywords":["multilingual object hallucination","large vision-language models","attention intervention","training-free mitigation","cross-lingual alignment","POPE benchmark","MME benchmark","attention heads"],"falsifier":"Compute the shift from the caption template, then evaluate on Spanish questions whose syntax differs sharply from the template—for example attribute or spatial-relation questions on images outside COCO; if accuracy does not improve over the baseline, the transfer assumption fails.","tokens_in":20258,"feed_emoji":"🖼️","tokens_out":3541,"duration_ms":34916,"temperature":0.7,"pith_summary":"The paper claims that a large share of multilingual object hallucination in vision-language models comes from non-English queries failing to route visual attention the way English queries do, and that this can be corrected without retraining. Its proposed method, CLAIM, finds the attention heads whose cross-modal outputs differ most between English and a target language, estimates a per-head language shift vector from 1,000 captioning prompts, and adds that shift at inference time. On two LVLMs and six non-English languages it reports average accuracy gains of 13.56 percentage points on POPE and 21.75 percentage points on MME's hallucination subsets. If true, this offers a lightweight, plug-and-play alternative to supervised fine-tuning and preference optimization for multilingual hallucination.","feed_headline":"Attention shift cuts multilingual AI vision hallucination by 13.6%","feed_subtitle":"A training-free intervention aligns non-English queries with English visual attention, improving POPE and MME scores across six languages.","key_machinery":"The central object is the language shift vector $S^l_h$, a per-head, per-layer mean difference in attention outputs between English and target-language caption queries over 1,000 COCO images, computed by Equation (9) with the single template “What is it in the image?”. During inference, Equation (10) adds the scaled shift $\\alpha S^l_h$ to the attention outputs of the heads selected by probe classification accuracy, so that non-English queries are processed with English-like visual perception.","core_discovery":"CLAIM establishes that cross-modal attention outputs carry a language-specific offset: for semantically identical queries, the attention output of certain heads differs systematically between English and other languages, and this offset can be measured once and then applied during generation. The paper identifies language-specific cross-modal attention heads by training linear probes to predict query language from masked attention outputs, estimates the shift as the mean English-minus-target difference over COCO caption queries, and intervenes by adding a scaled shift to the attention outputs of selected heads. This realigns non-English attention toward English-like perception, reducing object-level and attribute-level hallucination without updating any model weights.","pith_inferences":["The same shift-based intervention could be applied to other token divisions, such as separating image regions or audio chunks, to steer multimodal attention without training.","A fixed shift averaged over 1,000 images could be replaced by a per-query adaptive shift computed on the fly, which may improve robustness on out-of-distribution images and non-caption-style questions.","Because CLAIM aligns non-English attention toward English patterns, it may reinforce English-centric biases, as the paper itself notes; a natural extension is aligning toward a multilingual consensus representation instead.","A testable extension is estimating shifts from task-specific query distributions and checking whether the resulting vectors transfer across languages and benchmarks better than the single caption-template estimate."],"forward_implications":["CLAIM works without supervised fine-tuning or preference optimization, so it avoids the cost of building annotated multilingual image-text datasets.","The method improves both low-resource and high-resource languages, and a shift estimated from one language pair transfers partially to unseen languages.","The intervention reduces not only object-level hallucination in existence and count questions but also attribute-level hallucination in color and position questions.","Attention divergence between languages is strongest in intermediate layers, indicating where cross-modal integration and language mapping happen in LVLMs."],"supporting_citations":[{"why":"Supplies the inference-time intervention recipe of adding a vector to internal representations to steer model behavior.","marker":"(Li et al., 2024)"},{"why":"VCD is the decoding-time hallucination baseline that CLAIM compares against and outperforms.","marker":"(Leng et al., 2024)"},{"why":"POPE provides the object-presence hallucination benchmark used for the main evaluation.","marker":"(Li et al., 2023)"},{"why":"MME provides the hallucination subsets and full perception/cognition benchmark used in the experiments.","marker":"(Fu et al., 2023)"},{"why":"MHR is the prior SFT/DPO approach to multilingual hallucination that CLAIM positions itself against.","marker":"(Qu et al., 2024)"},{"why":"Establishes the attention-based, training-free direction for hallucination mitigation that motivates CLAIM.","marker":"(Liu et al., 2025)"},{"why":"Supports the claim that multilingual models map non-English inputs into an English semantic space in intermediate layers.","marker":"(Wendler et al., 2024)"}],"fun_headline_variants":["No training needed: attention shift cuts AI vision hallucination by 13.6%","Align attention, cut hallucination: training-free fix for multilingual vision AI","Zero training, big drop: attention shift cuts multilingual vision hallucination","Attention trick reduces AI vision hallucinations in non-English languages","No fine-tuning: attention shift reduces multilingual vision errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a shift vector averaged over 1,000 captioned COCO images with the single template “What is it in the image?” transfers to arbitrary unseen questions, images, and tasks at inference.","fun_headline_variants_meta":{"raw":{"variants":["No training needed: attention shift cuts AI vision hallucination by 13.6%","Align attention, cut hallucination: training-free fix for multilingual vision AI","Zero training, big drop: attention shift cuts multilingual vision hallucination","Attention trick reduces AI vision hallucinations in non-English languages","No fine-tuning: attention shift reduces multilingual vision errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001212,"raw_usage":{"total_tokens":4961,"prompt_tokens":886,"completion_tokens":4075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":3985}},"tokens_in":502,"tokens_out":4075,"duration_ms":29114,"temperature":1.0,"reasoning_tokens":3985,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:16:58.593723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the shift from the caption template, then evaluate on Spanish questions whose syntax differs sharply from the template—for example attribute or spatial-relation questions on images outside COCO; if accuracy does not improve over the baseline, the transfer assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the attention-based, training-free direction for hallucination mitigation that motivates CLAIM."}],"review_version":1}