{"id":"85e2acfc-ebef-4631-8285-449070374232","arxiv_id":"2607.03738","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"MLLMs attend to the relevant modality at the exact generation steps that need it; causal blocking of those peaks breaks answers while timed boosting improves multimodal performance.","lead":"Multimodal LLMs reallocate attention to image or text tokens precisely when generating the corresponding answer parts, and revisit instructions at task switches. Blocking those peaks cripples answers while a simple timed boost of the right modality lifts accuracy on spatial-reasoning and multi-part VQA tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified; global averaging (Eq. 1) is the softest point but is already mitigated by layer-group consensus and successful interventions.","rationale":"The reader correctly isolates the uniform layer/head average (Eq. 1) as the weakest modeling assumption. The paper already supplies the two pieces of evidence that keep the assumption from being load-bearing: (i) the stratified analysis in A.2 demonstrating that layers agree on routing direction, and (ii) the fact that intervening on the very same average produces the predicted causal outcomes (blocking failures of the four profiled types; timed-boost accuracy lifts of +28.5 % / +8.6 % / +2.7 % on VSR). Automatic POS tagging and absence of released code are real practical limitations, yet they do not reverse any reported numerical result. Consequently the ACCEPT verdict with high confidence remains appropriate; no adjustment is warranted.","tokens_in":29905,"tokens_out":526,"duration_ms":17248,"concrete_test":"Re-run the VSR boosting experiment of Table 3, applying the same multiplicative factor β only to the late-layer group defined in A.2 (instead of all layers/heads). If the VT-Acc gains remain within 2–3 points of the reported global-boost numbers, the average is confirmed as a sufficient proxy; if the gains collapse, layer-specific targeting is required and the current intervention over-states generality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that modality-specific attention peaks (image at image-derived tokens, instruction at hand-offs, previous tokens rising) are functionally causal—rests on the scalar α_t(c_k) obtained by uniform averaging of post-softmax attention over all layers and heads (Eq. 1). If early and late layers routed information in opposing directions, both the observed peaks and the global blocking/boosting interventions could be misleading. Appendix A.2, however, shows directional consensus: early/mid/late groups all exhibit positive relative attention to the relevant modality at the same semantic tags, with late layers merely amplifying the spikes. Moreover, total blocking of the averaged image stream at Fruit Concept (Table 1) and timed multiplicative boosting of the averaged stream at ISR/TSR (Table 3) produce exactly the predicted causal effects (accuracy collapse or large gains). Thus the averaging assumption, while simplifying, is empirically supported for the claims made; no stronger internal inconsistency appears.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies token-level attention dynamics during autoregressive generation in MLLMs (OTaT), tracking attention from the currently generating token to semantic chunks (image, text, instruction, previous tokens). Using a diagnostic Fruit-Math task plus ChartQA and VSR, it reports consistent patterns across LLaVA-OneVision and Qwen2.5-VL models (0.5B–7B): image attention peaks at image-derived concepts, instruction attention rises at task hand-offs, and previous-token attention grows over decoding. Total attention blocking at these peaks produces predicted failures (language priors, cross-modal leakage, denial, recovery); timed multiplicative boosting of the relevant modality at the right step improves VSR accuracy substantially, especially for the weaker model.","tokens_in":30202,"tokens_out":1214,"duration_ms":16949,"significance":"If the patterns and causal effects hold, the work supplies a useful complementary ‘when’ perspective to existing layer/head (‘where’) interpretability of MLLMs, with concrete functional validation via total blocking and a simple, architecture-agnostic test-time boost that yields large gains on an entangled spatial-reasoning task. Strengths include multi-family/multi-size consistency, explicit causal interventions (Tables 1–3), qualitative failure-mode profiling, layer-group consensus checks (App. A.2), and an immediately usable intervention. The contribution is primarily empirical and methodological rather than theoretical, but it is actionable for both analysis and reliability improvements.","major_comments":[{"comment":"Eq. (1) and §2.1 define the core scalar α_t(c_k) by uniform averaging of post-softmax attention over all layers and heads. While App. A.2 shows directional consensus across early/mid/late groups and the blocking/boosting results (Tables 1, 3) are consistent with the scalar being functionally meaningful, the paper should more explicitly quantify how much of the causal effect is carried by late layers alone (e.g., by repeating total blocking only on late layers). Without that, the claim that the global average is a sufficient intervention target remains only partially stress-tested.","section":"§2.1, Eq. (1); App. A.2"},{"comment":"The timed boosting intervention (§4.3, Table 3) and the dataset-level bar plots both rely on knowing semantic role tags (Fruit Concept, ISR/TSR, Handoff, etc.) in advance. For free-form generation this knowledge is unavailable; the paper notes the limitation (App. C) but does not demonstrate an automatic, model-internal detector of the critical steps. A minimal experiment showing that a simple heuristic (e.g., first occurrence of a format keyword or entropy spike) recovers most of the gain would strengthen the practical claim.","section":"§4.3; App. C"},{"comment":"Fruit and VSR accuracy, answer rates, and error profiles (Tables 1–3, Fig. 7) are obtained via Gemini 2.5 Pro prompts (App. D.3–D.4) with only ‘spot-checks’. Because these numbers underwrite the causal claims, the manuscript should report inter-annotator agreement on a non-trivial human-labeled subset (or release the exact scored outputs) so that the magnitude of the reported drops and gains can be independently verified.","section":"§3; Tables 1–3; App. D"}],"minor_comments":[{"comment":"Fig. 1D–E and the per-token plots in App. A.1 are dense; adding a short legend or color-blind-safe palette note would help readers who cannot zoom.","section":"Fig. 1; App. A.1"},{"comment":"The distinction between ‘lazy’ and ‘total’ blocking (Fig. 2, §2.2.1) is clear, but the main text could briefly restate why intermediate-token leakage is possible under lazy blocking before presenting Table 1.","section":"§2.2.1"},{"comment":"In §4.1 the claim that attention scores are influenced by both y_t and y_{t−1} is interesting; a one-sentence pointer to the corresponding spike at EOG in the per-token plots would make the observation easier to locate.","section":"§4.1"},{"comment":"App. A.4 (Math-Sport on the base LLM) is valuable; a single sentence in the main text noting that the same patterns appear in the text-only setting would better advertise the generality.","section":"§4.1; App. A.4"},{"comment":"Minor typographical inconsistencies appear in the appendix (e.g., ‘oranges’ misspelled in one recovery example, occasional missing spaces around citations). A light proof-reading pass would suffice.","section":"App. A.5"}],"recommendation":"minor_revision","confidential_remarks":"The work is solid empirical interpretability with a usable intervention; it fits a methods/interpretability track well. The heavy reliance on a closed proprietary model (Gemini) for both tagging and scoring is the main reproducibility soft spot; if the journal values open evaluation pipelines, the authors should be asked to mitigate it. Novelty relative to concurrent attention-dynamics papers (Dex-AR, DyRate, AdaptInfer) is adequately differentiated in §5, so I see no citation or priority issue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is the first systematic look at multimodal attention as a function of the semantic role of the token being generated, not just which layer or head is active. Across LOV and Qwen2.5-VL (0.5B–7B) they show the same schedule: image attention peaks exactly when the model names the visual concept, instruction is re-read at task hand-off, and previous tokens accumulate weight as generation proceeds. They then prove the peaks matter with a total-blocking operator (not the lazy mask used elsewhere) and convert the schedule into a multiplicative boost that lifts VSR joint accuracy by ~28 points on the weaker model and a few points on the stronger one.\n\nWhat works: the Fruit-Math diagnostic cleanly separates modalities so the peaks are unambiguous; the same schedule reappears on ChartQA and VSR; the four failure modes under image block (prior, leakage, denial, recovery) are documented with confusion matrices and examples; Appendix A.2 shows early/mid/late layers agree on direction even if late layers dominate magnitude, so the global average in Eq. 1 is not hiding opposing circuits. Boosting is architecture-agnostic and the γ plateau is wide.\n\nSoft spots are real but secondary. Everything still rides on post-softmax averages and Gemini tagging/scoring (they spot-check, but code is not released). Math accuracy on Fruit-Math is near floor, so the text-side story is thinner. The boost is a post-hoc sweep, not a learned policy. None of that undercuts the central empirical claim.\n\nThis is for people who care about multimodal interpretability or test-time reliability. It is solid enough for a serious referee; I would bring it to reading group and expect to cite the OTaT framing and the total-block results.","headline":"Clean token-level map of when MLLMs attend to image vs text, causally validated and turned into a simple timed boost that actually moves accuracy.","tokens_in":30712,"tokens_out":461,"would_cite":true,"duration_ms":5468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Multimodal models shift attention to the image or text exactly when the next token needs that modality, and boosting it on schedule improves their answers.","keywords":["multimodal LLMs","attention dynamics","autoregressive generation","token-level interpretability","attention blocking","attention boosting","vision-language models","test-time intervention"],"falsifier":"Measure fruit or spatial-relation accuracy after total blocking of image attention only at the concept token and after timed boosting only at the same token; if accuracy is statistically unchanged relative to the unblocked and always-boost baselines, the claimed causal role of the timed peaks is false.","tokens_in":30831,"feed_emoji":"🎯","tokens_out":855,"duration_ms":16249,"temperature":0.7,"pith_summary":"The paper shows that multimodal large language models do not attend uniformly while they generate. Instead, attention to the image spikes precisely when the model produces an image-derived word, attention to the instruction rises at the moment it switches tasks, and attention to its own previous tokens steadily grows. These patterns appear consistently across four open-weight models from two families and across both diagnostic and real benchmarks that force switches between vision and language inside one response. Blocking the peaks causes systematic failures—language priors, cross-modal leakage, denial of the image, or recovery—while a simple multiplicative boost of the right modality at the right generation step raises accuracy on visual-spatial reasoning. The work therefore supplies both a temporal account of how multimodal context is used and a practical test-time lever for improving it.","feed_headline":"Boost the right modality at the right token to lift accuracy","feed_subtitle":"Models spike image attention exactly when naming what they see; timed boosts raise spatial scores.","key_machinery":"One Token at a Time (OTaT) analysis: average post-softmax attention from the currently generating token onto semantic chunks (image, text, instruction, previous tokens), normalize by mean subtraction, aggregate over role-tagged output tokens, then intervene by total blocking or multiplicative reweighting of the resulting scalar.","core_discovery":"Across four open-weight MLLMs, attention to image tokens peaks at the generation steps that require image-derived information, instruction tokens are revisited at task hand-offs, and attention to previously generated tokens increases throughout decoding; total blocking of these peaks produces language-prior fallback, leakage, denial or recovery, while selective boosting of the relevant modality at those steps measurably improves multimodal task performance.","pith_inferences":["Because late layers dominate the observed spikes, a global boost still hits the critical computation even without head- or layer-selective targeting.","Role-aware boosting could be stacked with existing head- or layer-selective interventions for larger gains than either alone.","Post-hoc semantic tagging of free-form outputs would let the same analysis be run without forcing a fixed response format.","Cross-modal leakage under blocking in the smallest model may signal incomplete modality separation that larger models later suppress."],"forward_implications":["A training-free, token-timed attention boost can raise accuracy on tasks that require modality switches inside one answer.","Failure to answer the second part of a multi-part prompt can be diagnosed by checking whether instruction attention spikes at the hand-off token.","The same temporal routing appears in pure language models given two text tasks, indicating the pattern is inherited from the base LLM.","Disrupted visual attention produces stereotyped failure modes (prior, leakage, denial, recovery) that can be used as reliability diagnostics."],"fun_headline_variants":["Image attention spikes exactly at tokens needing visual info","Instruction tokens get revisited during task hand-offs","Blocking image peaks yields language-prior fallback or leakage","Boost relevant modality at peak steps to raise task scores","Attention to prior tokens rises steadily through decoding"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Averaging attention uniformly across every layer and head still yields a scalar that is functionally meaningful enough that blocking or boosting it alone can change the model’s answers.","fun_headline_variants_meta":{"raw":{"variants":["Image attention spikes exactly at tokens needing visual info","Instruction tokens get revisited during task hand-offs","Blocking image peaks yields language-prior fallback or leakage","Boost relevant modality at peak steps to raise task scores","Attention to prior tokens rises steadily through decoding"]},"model":"grok-4.5","effort":"low","cost_usd":0.004294,"raw_usage":{"total_tokens":1293,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":55,"cost_in_usd_ticks":42940000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":468,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":55,"duration_ms":3731,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T00:16:35.601307+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Measure fruit or spatial-relation accuracy after total blocking of image attention only at the concept token and after timed boosting only at the same token; if accuracy is statistically unchanged relative to the unblocked and always-boost baselines, the claimed causal role of the timed peaks is false.","supporting_citations":[],"review_version":1}