{"id":"7f1c8e6f-1047-4015-a218-89891aed053b","arxiv_id":"2608.11847","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LookBack scores LVLM responses by calibrating token likelihood with an attention-based visual lookback score and weighting by visual relevance, improving Best-of-N selection over baselines.","lead":"LookBack is a training-free scoring method that ranks candidate answers from vision-language models by combining each word's likelihood with how strongly the model attended to image tokens while generating that word. It improves Best-of-N answer selection on four image-grounded benchmarks, with the largest average gains over confidence-only baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A_t in Eq. 3 averages attention over all vision tokens and all layers, so it may reward attention to any image content rather than to evidence supporting the token; a mismatched-image control would test whether LookBack measures actual grounding.","rationale":"The central claim requires that A_t is a usable proxy for visual reference usage. This is the least secure condition because Eq. 3's definition is content-agnostic, and the paper's own qualitative results show the proxy alone selects irrelevant entities; only the combination with confidence rescues selection. The proposed mismatched-image control is decisive: a grounding score must distinguish the correct image from a mismatched one. If LookBack does not degrade under mismatched images, the method is not doing what the title and abstract claim, and the empirical wins are a correlation with attention magnitude rather than a mechanism. The concern is addressable, since the authors have the sampling infrastructure and can rerun the scorer under the control condition. The paper has some independent support, including oracle headroom in Table 2 and the POS sanity check in Fig. 4, and the reader's CONDITIONAL verdict remains appropriate: the result is plausible but not yet established without this control or a validation-split description for the per-model hyperparameters.","tokens_in":17435,"tokens_out":10716,"duration_ms":114335,"concrete_test":"Run a mismatched-image control on the diagnostic subsets (VQAv2 and CHAIR, 1,000 instances, N=25): keep the question and the same candidate responses, but condition the LVLM on a different image sampled from the same benchmark; recompute LookBack with the paper's per-model (alpha, lambda) and measure selection accuracy. If A_t reflects reference to the evidence supporting each response, accuracy should drop substantially under mismatched images relative to the correct-image condition. If the accuracy gap is small, or top-1 agreement between conditions is high, then A_t is tracking 'attention to any vision token' rather than visual grounding, and the central claim is unsupported. Report the gap and a paired bootstrap confidence interval.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Eq. 3 defines the visual lookback score A_t as the fraction of attention directed to the full set P_v of vision tokens, averaged over every layer and head. This collapses the image into a single scalar: a token receives a high A_t if the model attends to any visual token, including a global/CLS token, an unrelated background patch, or an image region that does not support the claimed object or relation. The POS analysis in Fig. 4 shows only that content words attend to vision tokens more than function words on average; it never compares A_t between correct and hallucinated tokens. The qualitative examples in Figs. 9-10 actually demonstrate the failure mode: visual lookback alone highlights irrelevant entities (cars, house, truck; child, toothbrush), and only the combination with confidence rescues selection. The Limitations section concedes that 'a model can strongly attend to image tokens while still generating an incorrect or unsupported response.' Thus the central mechanism, visual reference usage, is not established by A_t; the score may simply reward confident responses that allocate attention somewhere in the image. If that is the case, the reported gains are a heuristic correlation rather than a grounding signal, and the 'consistently improves' claim is unlikely to transfer to new benchmarks or architectures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies Best-of-N response selection for LVLMs. It first presents a diagnostic showing that an output-space confidence scorer (Self-Certainty) is largely insensitive to the input image: score distributions barely shift when the image is removed, and top-1 agreement between image-present and image-absent scoring is 0.36–0.64. It then proposes LOOKBACK, a training-free scorer that adds a token-level visual lookback score A_t (the fraction of attention from each output token to vision tokens, averaged over layers and heads) to the log-likelihood, and aggregates tokens with a visual-relevance distribution q_lambda derived from A_t. The final score is a weighted average of log p_t + alpha log A_t under q_lambda. Experiments on VQAv2, CHAIR, AMBER, and HallusionBench with LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL3-8B report consistent improvements in Best-of-N selection over SC, USC, CLIPScore, and VAUQ, with small scoring overhead.","tokens_in":17693,"tokens_out":3168,"duration_ms":29434,"significance":"The paper's motivation is clearly demonstrated: the image-removal diagnostic for Self-Certainty is simple and visually convincing. The proposed score is elegant, parameter-light in form (two hyperparameters), does not require external models, and the paper ships code and a reproducible experimental protocol. If the reported gains hold out-of-sample, LOOKBACK would be a useful, lightweight addition to the LVLM response-selection toolbox, and the attention-based calibration idea is worth publishing. However, the significance is currently limited by the evaluation protocol: the two free hyperparameters are tuned per model on the same benchmark suite that is used for the headline numbers, no variance or significance information is reported, and the central interpretation of A_t as visual reference usage is only partially validated (the authors themselves note in the Limitations that lookback is only a proxy and can reward incorrect responses).","major_comments":[{"comment":"The headline claim of consistent improvement is not supported across all cells. For InternVL3-8B on HallusionBench at N=25, LookBack (56.85) is below Random (57.19) and only ties SC (56.85). This is a clear counterexample to the abstract's 'consistently improves' and should be discussed explicitly, along with a tempered claim and per-cell analysis.","section":"§5.2, Table 1"},{"comment":"The per-model hyperparameters (alpha, lambda) are set on the same benchmarks on which the main results are reported, with no validation split and no sensitivity analysis in the main paper. This makes the reported 4.97% relative gain over random partly a result of tuning rather than a parameter-free prediction. Please provide results with fixed hyperparameters shared across models, a validation/test split, or per-benchmark sensitivity curves for all three models.","section":"§5.1, Table 1"},{"comment":"No variance or statistical significance is reported anywhere. With 1,000 sampled instances, the standard error of a percentage near 60–80 is roughly 1.2–1.5 points, so many of the reported differences (e.g., LLaVA CHAIR N=25: 74.43 vs 73.53; Qwen VQAv2 N=5: 68.23 vs 66.88) are within plausible noise. Please report standard errors or confidence intervals, and state which cells are statistically distinguishable from the best baseline.","section":"§5.2, Table 1"},{"comment":"The central claim that A_t measures 'visual reference usage' is not established. A_t averages attention over all vision tokens and all layers, so it rewards attention to any image token, including global/CLS tokens, background patches, or irrelevant distractors. The qualitative examples in Figs. 9–10 actually demonstrate this: lookback alone highlights cars/house/truck and child/toothbrush, which are not the question-relevant evidence. The POS analysis (Fig. 4) only shows that content words attend to vision tokens more than function words on average; it never compares A_t between correct and hallucinated tokens. Please add (i) a mismatched-image control (e.g., the same question paired with a different image) and (ii) a token-level analysis of A_t on hallucinated vs. correct claims, to test whether A_t reflects actual grounding rather than any image attention.","section":"§4.1, Eq. (3)"},{"comment":"The image-insensitivity diagnostic is run only for Self-Certainty and only for LLaVA-1.5-7B and Qwen2.5-VL-7B. The interpretation of the top-1 agreement (0.36–0.64) as 'barely changes selection' lacks a reference point: a vision-sensitive scorer under the same image-removal perturbation might also agree substantially because most candidates remain similar in quality. Please add a reference comparison, or alternatively show that the agreement of LookBack (or another vision-sensitive signal) under image removal is substantially lower.","section":"§3.1, Figs. 2–3"}],"minor_comments":[{"comment":"The main text states 'improves the average score from 65.37% to 68.62%', but Table 1's Average column does not contain these numbers; please clarify the aggregation across model–benchmark–N cells.","section":"§5.2 and Table 1"},{"comment":"The color scale for token-level highlighting is not described; please state what the darkness indicates (e.g., z-scored score, normalized within response).","section":"§D, Figs. 9–10"},{"comment":"The Limitations section correctly acknowledges that visual lookback is only a proxy and that attention to image tokens does not guarantee correctness; please move this caveat earlier (e.g., at the end of Sec. 3.2) so it qualifies the motivation before the method is introduced.","section":"§6, Limitations"},{"comment":"For USC, the text reports averaging over three runs but does not report the variance; since USC is stochastic, please give the standard deviation or justify that it is negligible.","section":"§B.2"},{"comment":"Please report the number of instances used per benchmark for the main experiments; the motivation analysis uses 1,000, but the main tables do not state N.","section":"§5.1, Implementation details"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable, lightweight contribution, but the evaluation protocol is the main risk: per-model hyperparameters tuned on the evaluation suite, no error bars, and one clear counterexample cell (InternVL3-8B on HallusionBench N=25) undermine the 'consistently improves' claim. A revision with held-out validation or fixed hyperparameters, significance reporting, and a mismatched-image control for the grounding interpretation would substantially strengthen the paper. I do not see grounds for rejection; the central idea is sound and the diagnostic is interesting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read LookBack (arXiv:2608.11847). The simple version is: yes, there's a useful, cheap idea here, but the paper oversells how well its attention-based score measures grounding, and the evaluation has a couple of fixable holes.\n\nWhat's actually new: they combine token-level likelihood with a 'visual lookback' score — the fraction of attention a generation step gives to vision tokens, averaged over layers and heads — and then aggregate with a soft weight that favors visually attended positions. That specific package for Best-of-N selection isn't in the cited baselines. It's training-free, uses only attention from the generation forward pass, and the authors make code available. Their diagnostic that Self-Certainty scores are nearly identical with and without the image is a nice, clean illustration that output-space confidence is largely text-prior-driven in LVLMs.\n\nThe paper is also honest about its own proxy. The Limitations section states that a model can attend strongly to image tokens while still generating an unsupported response, and the qualitative examples show visual lookback alone highlighting cars, houses, and toothbrushes — not the evidence that actually answers the question. Success comes only after combining with confidence and applying the relevance weighting.\n\nThat honesty doesn't fully rescue the central claim. Eq. 3 collapses the whole image into one scalar; high A_t just means 'some attention to some vision token.' The POS analysis shows only that content words attend more than function words, never that attend-a-lot tokens are more factually grounded. A mismatched-image control, or a direct comparison of A_t on correct vs hallucinated tokens, would be the natural way to test that.\n\nThe evaluation also needs work before I'd trust the headline. Alpha and lambda are set per model with no validation split described, so part of the gain is tuning. No error bars or significance tests anywhere. And the 'consistently improves' claim is literally false: on InternVL3-8B / HallusionBench, LookBack gets 57.02 at N=5, while CLIPScore gets 59.39, VAUQ 57.19, USC 58.15 — it loses to several baselines. So the claim should be softened to 'on average.'\n\nOverall: the method is plausible, cheap, and worth knowing about. I'd send it to peer review with the request that the authors add a proper validation split, report variance, and include a control that actually tests whether attention equals grounding. It's not a desk reject and not a breakthrough — it's a solid incremental contribution that needs a more careful evaluation.","headline":"A cheap and honest scoring trick that works on average, but the grounding claim is under-tested and the evaluation has fixable holes.","tokens_in":18208,"tokens_out":3718,"would_cite":true,"duration_ms":35364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LookBack, a training-free scorer that combines token likelihood with a visual lookback attention signal, consistently improves Best-of-N response selection for vision-language models across four benchmarks and three models.","keywords":["Best-of-N selection","visual hallucination","large vision-language models","response scoring","attention weights","visual grounding","training-free scoring","self-certainty"],"falsifier":"Zero out or randomize the attention paid to vision tokens while keeping the token probability distributions unchanged, then rerun LookBack Best-of-N selection on the same four benchmarks; if selection accuracy stays the same as with intact attention, then visual lookback is not the mechanism behind the reported gains.","tokens_in":17258,"feed_emoji":"🖼️","tokens_out":8747,"duration_ms":74991,"temperature":0.7,"pith_summary":"The paper claims that choosing among sampled responses of a vision-language model fails when the scorer only measures text confidence, because that confidence barely changes when the image is removed. To fix this, it proposes LookBack, a training-free scorer that combines each token's likelihood with a visual lookback score, the fraction of attention that token pays to image tokens, and weights tokens by how visually relevant they are. Across four image-grounded benchmarks and three different vision-language models, LookBack selects better Best-of-N responses than existing linguistic and vision-side baselines, at negligible extra cost. A sympathetic reader would care because it suggests model-internal attention is a usable, low-cost calibration signal for visual grounding, without reward models or extra inference.","feed_headline":"Attention to image tokens improves LVLM answer selection","feed_subtitle":"Training-free scorer beats confidence-only baselines on four benchmarks and three models.","key_machinery":"The load-bearing object is the token-level visual lookback score $A_t$, defined in Eq. (3) as the mean over layers and heads of the fraction of attention weight from the prediction step that falls on vision-token positions $P_v\\subset C_t$. It feeds two places: the lookback-calibrated token score $u_t = \\log(p_t) + \\alpha\\log(A_t)$, which lowers the score of high-likelihood tokens that do not attend to the image; and the visual relevance distribution $q_\\lambda(t) = A_t^\\lambda/\\sum_j A_j^\\lambda$, the closed-form solution of an entropy-regularized relevance maximization (Proposition 1) that concentrates response-level weight on visually attended positions. The final score $S(y|x,v)=\\sum_t q_\\lambda(t)u_t$ is equivalent to maximizing a visual-relevance-weighted geometric mean of $p_t A_t^\\alpha$ (Corollary A.0.1), which makes it length-normalized. All quantities come from the generation forward pass, so no extra inference or external model is needed.","core_discovery":"On the paper's own terms, the central discovery is that output-space confidence in LVLMs is largely image-insensitive: the distribution of Self-Certainty scores barely shifts when the image is removed, and the same top candidate is selected 0.36–0.64 of the time, far above the 1/25 random rate. Visual lookback score, defined as the mean over layers and heads of the fraction of attention each generation step directs to vision tokens, is higher for visually referential words (nouns, adjectives, numerals) and lower for function words, exactly opposite to confidence. LookBack combines the two by scoring each token as $\\log(p_t) + \\alpha\\log(A_t)$ and aggregating under the distribution $q_\\lambda(t)\\propto A_t^\\lambda$. The result is that Best-of-N selection improves consistently across VQAv2, CHAIR, AMBER, and HallusionBench and over the three tested LVLMs, with an average gain from 65.37% to 68.62% over random selection, requiring no auxiliary verifier, training, or extra inference pass.","pith_inferences":["A natural testable extension is to turn the calibrated token score into a training signal, for example by training a model to increase $A_t$ for high-likelihood tokens, potentially reducing hallucination without a reward model; the paper only uses the score for selection.","The paper averages $A_t$ over all layers and heads, an easy extension would be to select or learn layer and head subsets, since attention calibration varies across architectures and the authors' own limitations note this dependence.","The entropy-regularized weighting suggests an adaptive $\\lambda$ per response: responses with attention concentrated on a few tokens might need sharper weighting, while diffuse attention might need smoother weighting to avoid over-penalizing valid low-attention tokens.","The per-model hyperparameters differ widely ($\\alpha=7$ for LLaVA versus $0.25$ for InternVL3), so cross-model transfer of hyperparameters without tuning is untested; a further study could examine whether $\\alpha$ tracks attention magnitude."],"forward_implications":["Best-of-N selection for vision-language models can be improved without reward models or extra inference, by reusing attention weights computed during sampling.","The diagnostic itself, removing the input image and measuring top-1 agreement of confidence scores, is a reusable sanity test for whether any LVLM response scorer is actually sensitive to the image.","Because the scorer needs internal attention weights, the method is usable on open-architecture LVLMs but not on black-box API-only models.","The same source-aware scoring principle should transfer to any generation task where high-confidence tokens must refer back to a designated part of the context, such as retrieved documents, tool outputs, or instruction tokens."],"supporting_citations":[{"why":"Self-Certainty, the output-space confidence baseline that the paper diagnoses as image-insensitive and that LookBack extends and must outperform.","marker":"Kang et al., 2025"},{"why":"CLIPScore, the vision-side baseline using a pretrained vision-language encoder that LookBack is compared against.","marker":"Hessel et al., 2021"},{"why":"VAUQ, an attention-masking uncertainty baseline that LookBack is compared against and shown to be cheaper than.","marker":"Park et al., 2026"},{"why":"VQAv2, one of the four image-grounded benchmarks used for Best-of-N selection evaluation.","marker":"Goyal et al., 2017"},{"why":"CHAIR, the object hallucination benchmark used in the diagnostic analysis and main evaluation.","marker":"Rohrbach et al., 2018"},{"why":"AMBER, the multidimensional hallucination benchmark used in the main results.","marker":"Wang et al., 2023a"},{"why":"HallusionBench, the discriminative visual-grounding benchmark used for scaling and main results.","marker":"Guan et al., 2024"},{"why":"LLaVA-1.5-7B, one of the three LVLMs on which LookBack is evaluated.","marker":"Liu et al., 2024"},{"why":"Qwen2.5-VL-7B, one of the three LVLMs on which LookBack is evaluated.","marker":"Bai et al., 2025"},{"why":"InternVL3-8B, one of the three LVLMs on which LookBack is evaluated.","marker":"Zhu et al., 2025"}],"fun_headline_variants":["LookBack: visual attention scores LVLM responses","Image-aware scoring beats confidence for LVLMs","Training-free scorer uses visual attention for LVLMs","LVLM scoring improved by visual reference usage","LookBack: where and how to score LVLM responses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole construction rests on the assumption that the fraction of attention a token pays to image tokens faithfully measures how much that token actually relies on the image; if attention is miscalibrated or visual information flows through other pathways, the lookback calibration stops working.","fun_headline_variants_meta":{"raw":{"variants":["LookBack: visual attention scores LVLM responses","Image-aware scoring beats confidence for LVLMs","Training-free scorer uses visual attention for LVLMs","LVLM scoring improved by visual reference usage","LookBack: where and how to score LVLM responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1256,"prompt_tokens":921,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":262}},"tokens_in":537,"tokens_out":335,"duration_ms":4299,"temperature":1.0,"reasoning_tokens":262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:24:23.850309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Zero out or randomize the attention paid to vision tokens while keeping the token probability distributions unchanged, then rerun LookBack Best-of-N selection on the same four benchmarks; if selection accuracy stays the same as with intact attention, then visual lookback is not the mechanism behind the reported gains.","supporting_citations":[],"review_version":1}