{"id":"a0949d09-253b-4659-b0c0-d893a593f62e","arxiv_id":"2507.05056","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"INTER is a training-free logit-correction method that adds Harsanyi interaction scores to selected keyword tokens, lowering hallucination on six LVLM benchmarks.","lead":"This paper introduces INTER, a decoding-time method that makes vision-language models lean on their own estimated image-text interaction when generating key words, which reduces hallucinated objects and details on several benchmarks. It is worth reading because a training-free hallucination patch matters for practical multimodal AI, though the reported gains rest on a model-tuned threshold and lack error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The interaction term in Eq. 3 rests on unvalidated counterfactual baselines: if empty-prompt and random-noise-image states do not isolate modality absence, the claimed mechanism behind INTER is not established.","rationale":"The reader's weakest-assumption identification is exactly the load-bearing point: Eq. 3's interaction is defined through empty-text and random-noise counterfactuals, and the paper provides no evidence that these states isolate modality contributions. I agree this is the most important concern because it sits underneath all three insights, the locator, and the modifier. The paper does have genuine supporting evidence: consistent improvements across multiple LVLMs and decoding strategies, an IGL ablation in Table 7, a k-robustness analysis in Section 10, and use of standard benchmarks. Those facts support an empirical decoding intervention but not the specific mechanism claimed. The proposed concrete test would settle whether the counterfactual construction matters: if alternative neutral baselines give the same gains, the method is robust even if the interpretation is loose; if they do not, the central explanation fails. Because the paper's empirical case is broad enough to warrant conditional acceptance rather than rejection, and because the counterfactual concern is testable rather than fatal as stated, I recommend keeping the reader's CONDITIONAL verdict. Other concerns noted by the reader, such as the benchmark-tuned threshold and missing error bars, are real but secondary; they would strengthen the conditional verdict rather than change it.","tokens_in":29151,"tokens_out":3633,"duration_ms":52716,"concrete_test":"Run a controlled counterfactual ablation on a fixed subset (e.g., POPE-COCO with LLaVA-v1.5 under Nucleus sampling, and CHAIR with the same model). Keep all INTER hyperparameters fixed, and compare four variants of Eq. 3: (1) the published empty-prompt / random-noise baselines; (2) a length-matched neutral prompt (e.g., 'Please continue.' or repeating the padding token) instead of the empty prompt; (3) a gray image or a heavily blurred version of the original image instead of random Gaussian noise; (4) both replacements together. For each variant, report the mean |I({v,p})|, the fraction of steps where the IGL gate activates, and the POPE F1 and CHAIR CS/CI improvements over the unmodified baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that adding the Harsanyi interaction logit I({v,p}) to keyword tokens, per Eq. 5, reduces hallucination by making decoding \"reapply multimodal interaction understanding.\" Every insight and every component (IGL variance gate and IPM logit modification) depends on the value of I(A) computed by Eq. 3. For A = {v,p}, I(A)_yt = L(v,p)_yt - L(v,empty-text)_yt - L(noise,p)_yt + L(noise,empty)_yt. This is only interpretable as the joint image-text interaction if the two counterfactual states represent controlled absence of one modality. They do not. Removing the prompt entirely changes the task: the model is no longer answering the question, so L(v, empty-text) conflates task mismatch with modality absence. Replacing the image with random Gaussian noise is also not a neutral \"no image\" baseline; LVLMs are not trained on pure noise, so L(noise, p) can reflect out-of-distribution artifacts rather than text-only processing. If these states are invalid, then I(A) is not a true interaction, the locator's variance gate may simply detect prompt-sensitive or OOD-sensitive tokens, and the modifier may be performing a prompt-contrast or noise-contrast correction whose success does not validate the claimed explanation. The broad benchmark gains in Tables 2-13 do not resolve this, because many decoding interventions that alter the logit distribution reduce hallucination without relying on a valid interaction decomposition. The manuscript itself provides no validation of the counterfactual states; Section 5.1 only states that random noise and empty text are used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free decoding intervention, INTER, for reducing hallucinations in large vision-language models (LVLMs). The authors formalize the LVLM decoding logits as a cooperative game over two players, the image v and the prompt p, and use the Harsanyi dividend to define a multimodal interaction logit I({v,p}) for each candidate token. From this quantity they derive three insights: LVLMs do use image-text interactions, these interactions concentrate on a few keyword tokens, and stronger interactions correlate with more accurate responses. Based on these insights, INTER combines an Interactive Guided Locator (IGL), which uses the variance of interaction values to detect keyword tokens, and an Interaction Probability Modifier (IPM), which adds the interaction logits to the original logits before softmax at keyword positions. Experiments on POPE, MME, MM-Bench, MMStar, CHAIR, and LLaVA-Bench across several LVLMs and decoding strategies report consistent improvements over nucleus sampling, beam search, VCD, and OPERA.","tokens_in":29450,"tokens_out":4862,"duration_ms":58413,"significance":"If the proposed mechanism were fully validated, INTER would be a notable contribution: it is training-free, model-agnostic, and appears to improve both VQA and captioning benchmarks across a range of LVLMs, including InternVL2.5-MPO. The paper also ships an unusually broad evaluation matrix, covering multiple models, decoding strategies, and benchmarks, which is a genuine strength. However, the central mechanistic claim depends entirely on the validity of the Harsanyi interaction term defined in Eq. (3). The current counterfactual baselines for 'image-only' and 'text-only' states are not justified, and the insights are to some extent built into the construction of the metric. The empirical gains are interesting and may survive as a heuristic, but the paper does not yet establish that they arise from multimodal interaction understanding rather than from a generic prompt-contrast or noise-contrast correction.","major_comments":[{"comment":"The interaction term I({v,p}) is computed from logits L(v, empty-text), L(noise, p), and L(noise, empty-text). These counterfactual inputs do not isolate modality absence: removing the prompt changes the task and the expected answer distribution, while Gaussian noise is outside the visual encoder's training distribution. Consequently I({v,p}) conflates task mismatch and out-of-distribution artifacts with genuine multimodal interaction. Because Eq. (3) is the basis for Insights 1–3, the IGL locator, and the IPM modifier, the paper's central explanatory claim that INTER works by 'reapplying multimodal interaction understanding' is not supported by the present evidence. The empirical gains could equally arise from a prompt-contrast or noise-contrast correction. The authors should validate the counterfactual states against alternative no-image and no-prompt baselines, or reframe the method as a contrastive decoding heuristic and substantiate that framing.","section":"§3.2, Eq. (3); §5.1"},{"comment":"Insight 1 is near-tautological as stated. For any four distinct logit vectors, the alternating sum in Eq. (3) is generically nonzero, so reporting mean |I(A)| > 0 does not establish that LVLMs 'capture multimodal interactions'; it only shows that the four logit vectors differ. A null model is needed before the existence claim can be distinguished from arbitrary input sensitivity. For example, the authors could compare the measured interaction values with those obtained under random image-prompt pairings or with a dummy-player baseline, and show that real interactions are significantly larger.","section":"§3.3, Table 1"},{"comment":"The design loop uses the same benchmarks for insight, threshold selection, and evaluation. Insight 3 is derived from MME yes/no correlations, and the interaction-guidance threshold k is tuned on MME and CHAIR (Figs. 7–8), while MME and CHAIR are also among the six evaluation benchmarks. The reported gains are therefore at risk of selection on the test set. The authors should report results with k fixed a priori or chosen on a separate validation set, or provide a sensitivity analysis on a holdout benchmark not used in any design step.","section":"§3.5, Fig. 3; §10, Figs. 7–8"}],"minor_comments":[{"comment":"The heading contains a typo: 'substasks' should be 'subtasks'.","section":"§14"},{"comment":"The binary selector β is described in the IGL paragraph but not explicitly defined in Eq. (5); adding β = 1{D_yt(I(A)_yt) > k} directly below the equation would improve readability.","section":"§4, Eq. (5)"},{"comment":"The paper says IGL 'prevents interference from image-text interactions to contextual connectives, thereby preserving linguistic coherence,' but no coherence metric or linguistic-quality analysis is reported; GPT-4o scores on LLaVA-Bench measure accuracy and detail, not coherence.","section":"§4, 'Overall Mechanism'"},{"comment":"The claim that runtime overhead is 'negligible' because all subset evaluations are batched is not supported by measurements; please report wall-clock or latency comparisons for at least one model.","section":"§17"},{"comment":"The text states that INTER achieves 'the highest reduction of 34.6% and 18.9% on CS and CI respectively,' but these numbers are not directly traceable in Table 4; the specific baseline and setting should be identified.","section":"Table 4 and §5.2"},{"comment":"The statement that the Harsanyi dividend 'satisfies the efficiency, linearity, dummy, symmetry axioms' is imprecise: those axioms apply to the Shapley value, and the Harsanyi dividend is the unique decomposition from which the Shapley value is assembled, not itself an axiomatic value.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is accurate and lands on the central load-bearing point of the paper. The empirical breadth is real, but the mechanistic interpretation and the design-loop circularity need to be addressed before the paper can be accepted. If the authors can validate the interaction decomposition or explicitly reposition INTER as a contrastive decoding heuristic, the manuscript could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two distinct things: it offers a game-theoretic analysis of multimodal interactions in LVLMs, and it uses that analysis to build a training-free decoding correction that consistently reduces hallucination across six benchmarks and five model families. The second part is the real contribution. Adding the Harsanyi interaction logit to selected high-variance tokens is simple, computationally cheap, and appears to generalize across nucleus, greedy, beam, VCD, and OPERA. The evaluation is broad: POPE, MME, MM-Bench, MMStar, CHAIR, LLaVA-Bench, and they report per-subtask numbers. The threshold k is examined for robustness. That is a solid empirical package, and the improvements look real.\n\nWhat I don't buy is the mechanism. Equation 3 defines the image-text interaction as L(v,p) - L(v,empty) - L(noise,p) + L(noise,empty). This is only a controlled interaction term if the two counterfactual states isolate absence of a single modality. They don't. Empty text changes the task from answering a question to continuing a prompt-free string; the resulting logit difference conflates task mismatch with modality absence. Random Gaussian noise is not a neutral \"no image\" baseline; LVLMs are not trained on pure noise, so the noise-only term can reflect out-of-distribution artifacts. The paper provides no validation that these alternatives are controlled ablations. And Insight 1 is nearly tautological: for any non-constant logit function, the finite difference in Eq. 3 is generically nonzero, so Table 1 showing positive absolute values doesn't demonstrate that the model \"captures multimodal interactions\" in any meaningful sense.\n\nSince the locator and the modifier both depend on I(A), the whole method may be performing a prompt-contrast or noise-contrast correction rather than an interaction boost. That would still be a useful decoding intervention, but the paper's claimed explanation would be unsupported. The empirical gains don't resolve this, because many logit corrections reduce hallucination without a valid interaction decomposition. Also, the threshold k is selected using the same benchmarks later used for evaluation, and although they show stability across a range, that is tuning on the test set. There are no error bars or significance tests despite reporting averages over five runs, and some subtask numbers go the wrong way.\n\nWho is this for? Anyone building training-free hallucination mitigations or doing contrastive decoding will want to know about INTER as a strong baseline. The game-theoretic insights are suggestive at best. The paper deserves a serious referee; it should go through peer review with a request for validation of the counterfactual baselines, a fixed threshold protocol or proper nested selection, and variance reporting. I would not desk-reject it.","headline":"A useful empirical decoding patch for LVLM hallucination wrapped in a game-theoretic story whose central interaction term rests on unvalidated counterfactual baselines.","tokens_in":30012,"tokens_out":2667,"would_cite":true,"duration_ms":30126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding game-theory interaction scores to key tokens cuts LVLM hallucinations.","keywords":["hallucination mitigation","large vision-language models","Harsanyi dividend","multimodal interaction","decoding strategy","training-free","game theory","sampling correction"],"falsifier":"Compute $I(A)$ using a different null state for the missing modality—for example, a black or blurred image instead of random noise, or a padded duplicate of the prompt instead of an empty string—and rerun INTER on POPE. If the hallucination reduction largely disappears or flips sign, the interaction term is an artifact of the particular counterfactual rather than a genuine multimodal signal.","tokens_in":28958,"feed_emoji":"🧠","tokens_out":5303,"duration_ms":51741,"temperature":0.7,"pith_summary":"This paper claims that large vision-language models, like people, lean on the combined image-text meaning of a sample rather than on either modality alone, and that this multimodal interaction mainly shapes the few content words of an answer. The authors measure that interaction with the Harsanyi dividend from game theory and find three things: the interaction exists, it is concentrated on keyword tokens, and stronger interaction on a keyword predicts a more accurate answer. They then turn the measurement into a decoding rule, INTER, which adds the interaction logit to keyword tokens during sampling. Because the rule needs no training and no extra data, it is a drop-in correction for any autoregressive vision-language model and any existing decoding strategy; on six benchmarks it improves scores by up to 3.4% on average across five models.","feed_headline":"Sampling tweak cuts vision-language hallucinations by 3.4%","feed_subtitle":"Adding a game-theory interaction score to key words reduces made-up objects and attributes across six benchmarks.","key_machinery":"The load-bearing object is the Harsanyi dividend $I(A|\\{v,p\\})_{y_t}$, a game-theoretic measure of how much the coalition of image and text contributes to the logit of candidate token $y_t$ beyond what each modality contributes alone. It is computed by taking the logit on the full input minus the logits on the image-only and text-only counterfactual inputs plus the logit on the empty input, per Eq. 3. INTER uses two modules built from it: the Interactive Guided Locator flags a step as a keyword step when the variance of $I(A)$ over the vocabulary exceeds a threshold $k$, and the Interaction Probability Modifier adds $I(A)$ to the logits of those steps before softmax. The variance gate is what keeps the correction on content words and off grammatical connectives, preserving fluency.","core_discovery":"The central discovery is that multimodal interaction information, computed as the Harsanyi dividend of the image and prompt players for each candidate token, is a usable, training-free steering signal for generation. Concretely, the paper proposes to replace the sampling distribution at keyword positions with $P_t = \\mathrm{SoftMax}(M_\\theta(v,p,y_{<t}) + \\beta \\cdot I(A))$, where $I(A)$ is the image-text interaction logit for $A=\\{v,p\\}$ and $\\beta$ is a 0/1 gate set by whether the variance of $I(A)$ across the vocabulary exceeds a threshold. The authors establish the three premises behind this rule—interactions exist in LVLM decoding, they are concentrated on keywords, and they correlate positively with correctness—and then demonstrate that the rule reduces object- and attribute-level hallucinations on POPE, MME, MM-Bench, MMStar, CHAIR, and LLaVA-Bench across five LVLMs and multiple decoding strategies, including the current state-of-the-art InternVL2.5-MPO.","pith_inferences":["If the interaction logits are a genuine steering signal, they could be combined with other training-free methods such as contrastive decoding by summing corrections, potentially yielding larger gains than either alone; the paper does not test this combination.","The variance gate appears to be an implicit part-of-speech detector, flagging content words without explicit tagging; a testable extension is whether using explicit POS tags instead of the variance threshold gives the same or better hallucination reduction.","The MME correlation analysis suggests that interaction strength on the 'yes'/'no' keyword predicts answer correctness; this could be turned into a hallucination detector that flags low-interaction keyword decisions at inference time.","Because the counterfactual inputs are cheap to batch, the approach may extend to other multimodal settings such as audio-image or video-text models, where the same Harsanyi construction would measure different interaction coalitions."],"forward_implications":["Adding interaction logits to keyword tokens reduces object and attribute hallucinations on POPE, MME, MM-Bench, MMStar, CHAIR, and LLaVA-Bench without any training.","The correction composes with existing decoding strategies, improving nucleus sampling, beam search, greedy search, VCD, and OPERA when applied on top of them.","The method transfers across model families (InstructBLIP, LLaVA-v1.5, Qwen-VL, mPLUG-Owl2) and across parameter scales from 1B to 26B.","The three verified insights imply that hallucination is partly a decision-time failure to reuse multimodal interaction knowledge the model already has, not only a knowledge gap."],"supporting_citations":[{"why":"Defines the Harsanyi dividend used to quantify the multimodal interaction contribution of image and text to each candidate token.","marker":"[22]"},{"why":"Shapley value axioms that give the Harsanyi dividend its theoretical support as an interaction measure.","marker":"[48]"},{"why":"Prior application of game-theoretic interaction analysis to vision-language models that this paper extends to hallucination.","marker":"[56]"},{"why":"Visual contrastive decoding baseline that INTER is compared against and built on top of.","marker":"[16]"},{"why":"OPERA decoding baseline whose over-trust penalty and CHAIR sampling setup are used for comparison and evaluation.","marker":"[25]"},{"why":"Nucleus sampling, one of the base decoding strategies INTER corrects.","marker":"[23]"},{"why":"POPE benchmark used to measure object hallucination reductions.","marker":"[33]"},{"why":"CHAIR benchmark used to quantify sentence- and object-level hallucination in captions.","marker":"[46]"}],"fun_headline_variants":["Training-free sampling fix reduces LVLM hallucinations by 3.4%","No training, no extra data: INTER cuts hallucination by 3.4%","Interaction-guided sampling: no training, 3.4% less hallucination","New decode rule: use image-text interaction to kill made-up answers","Use game-theory interaction signal to guide LVLM decoding, no extra data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the counterfactual inputs—an empty text prompt paired with the image, and a random-noise image paired with the text—cleanly isolate what each modality alone contributes, so that subtracting them leaves a true measure of image-text interaction; if those artificial states do not isolate modality contributions, the locator and modifier are steering by an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Training-free sampling fix reduces LVLM hallucinations by 3.4%","No training, no extra data: INTER cuts hallucination by 3.4%","Interaction-guided sampling: no training, 3.4% less hallucination","New decode rule: use image-text interaction to kill made-up answers","Use game-theory interaction signal to guide LVLM decoding, no extra data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000724,"raw_usage":{"total_tokens":3266,"prompt_tokens":982,"completion_tokens":2284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2184}},"tokens_in":598,"tokens_out":2284,"duration_ms":18193,"temperature":1.0,"reasoning_tokens":2184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:33:04.502017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $I(A)$ using a different null state for the missing modality—for example, a black or blurred image instead of random noise, or a padded duplicate of the prompt instead of an empty string—and rerun INTER on POPE. If the hallucination reduction largely disappears or flips sign, the interaction term is an artifact of the particular counterfactual rather than a genuine multimodal signal.","supporting_citations":[{"cited_title":"A simplified bar- gaining model for the n-person cooperative game","cited_arxiv_id":null,"evidence_quote":"Defines the Harsanyi dividend used to quantify the multimodal interaction contribution of image and text to each candidate token."},{"cited_title":"A value for n-person games","cited_arxiv_id":null,"evidence_quote":"Shapley value axioms that give the Harsanyi dividend its theoretical support as an interaction measure."},{"cited_title":"Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View","cited_arxiv_id":"2405.17201","evidence_quote":"Prior application of game-theoretic interaction analysis to vision-language models that this paper extends to hallucination."},{"cited_title":"Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation","cited_arxiv_id":null,"evidence_quote":"OPERA decoding baseline whose over-trust penalty and CHAIR sampling setup are used for comparison and evaluation."},{"cited_title":"The curious case of neural text degeneration","cited_arxiv_id":null,"evidence_quote":"Nucleus sampling, one of the base decoding strategies INTER corrects."}],"review_version":1}