{"id":"14ec0d74-bdb7-41e2-b8bc-ec245e8ce435","arxiv_id":"2506.05344","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.","lead":"Multimodal AI models use far fewer attention heads for seeing than for language, and this paper exploits that by shrinking the memory cache for little-used heads. The result is a cache-compression method that claims up to 1.38x faster generation and roughly half the memory use while keeping accuracy on vision-language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The visual-head score in Eq. 3-4 may reward generic attention concentration rather than content-specific visual alignment, and no criterion justifies the '<5%' sparsity claim; a label-permutation control would settle this.","rationale":"The paper's strongest evidence for visual heads is the contrast between SparseMM and Random Head under low cache budgets, plus the masking experiments in Sec. 4.4. Those results are correlational: they show the score ranking matters, not that it measures visual specialization. The Reader's conditional verdict already rests on the same premise, and I do not see an internal inconsistency that would justify rejection. The missing piece is a control that rules out generic attention concentration as the source of the score. The permutation test I propose is cheap and decisive: with random labels, a content-specific score should collapse to chance overlap and lose its accuracy advantage, whereas a generic-concentration score would survive. If it survives, the central claim is supported and SparseMM's head budgets are genuinely visual; if it fails, the paper should be revised to claim a practical head-wise cache prior rather than a discovered visual-head phenomenon. The threshold for 'visual head' should also be stated explicitly and tested for sensitivity, since the '<5%' claim has no stated cutoff.","tokens_in":17472,"tokens_out":4939,"duration_ms":64610,"concrete_test":"Permutation control for Algorithm 1: for each OCR image, shuffle the (output token -> patch set) assignment while preserving the size of each target patch set, compute the permuted score matrix S_perm over the same 1000 Synthdog images, and (1) report the Jaccard overlap between the original and permuted top-5% head sets; (2) run SparseMM at budget 256 with S_perm replacing S on DocVQA and OCRBench. If overlap is high and S_perm matches original accuracy, the identification is not content-specific; if overlap is near chance and accuracy drops to the Random-head level, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a small, identifiable subset of heads (<5%) carries visual understanding and that OCR-aligned argmax hits (Eq. 3-4, Algorithm 1) isolate that subset. This premise is load-bearing for both the scientific conclusion and for SparseMM, because Eq. 8 converts these scores into cache budgets. The weakest point is that a 'hit' only requires the argmax position of a head to fall inside the bbox-aligned image-token set. In OCR-heavy inputs, text patches cover a substantial fraction of the image, and many heads have generic concentration behavior (e.g., attending to a salient text region, a sink token, or any visually dense patch) that would register as a hit regardless of whether the head computes the specific token-to-patch correspondence. The score also ignores attention magnitude: a head placing 30% of its mass on the target patch is treated identically to one placing 100%, so the score measures coverage of the target region, not causal contribution to the emitted token. Separately, the paper never states the threshold on the score at which a head becomes 'visual'; Fig. 2 is qualitative, so the 'approximately less than 5%' statement has no stated falsification criterion. If the hit-based score is mostly generic concentration, SparseMM is a head-wise cache-allocation method with a tuned prior, not evidence of visual specialization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates attention-head specialization in multimodal large language models (MLLMs) and claims that fewer than 5% of decoder attention heads, termed visual heads, drive visual understanding. The authors introduce a training-free identification procedure based on OCR-aligned argmax attention hits (Eq. 3-4), report that these heads are sparse across Vicuna- and Qwen2-based backbones, and use the resulting head scores to allocate asymmetric KV-cache budgets in a method called SparseMM. Experiments across DocVQA, OCRBench, TextVQA, ChartQA, TextCaps, MMBench, GQA, and VQAv2 show that SparseMM generally outperforms SnapKV, PyramidKV, AdaKV, and a random-head baseline at low cache budgets, with reported latency and memory gains.","tokens_in":17726,"tokens_out":7868,"duration_ms":86104,"significance":"If the sparsity claim is correct, the paper offers a simple, training-free route to identify modality-specialized heads, which is of scientific interest for interpretability of MLLMs and of practical value for inference acceleration. Strengths include: the identification method is training-free and model-agnostic; the paper includes a random-head control that is consistently worse; the main results span three backbones and many benchmarks; code is open-sourced; and the appendix provides numerical tables. However, the central claim of '<5%' is not quantitatively defined, and the hit-based score may confound generic attention concentration with visual specialization; these issues are load-bearing for the paper's framing and need to be resolved.","major_comments":[{"comment":"The claim that 'approximately less than 5%' of heads are visual is never tied to a threshold on the visual score. Eq. (3) produces a continuous score, but the paper does not state what score cutoff, percentile, or other rule separates visual from non-visual heads; Fig. 2 is a qualitative heatmap with no scale or cutoff marked. Without a stated decision rule, the sparsity claim is not falsifiable, and the connection to the budget allocation in Eq. (8) is loose. Please define the criterion explicitly (e.g., a score threshold calibrated against a null distribution, or a top-k selection) and report the resulting number of heads per layer/model.","section":"Sec. 3.2, Algorithm 1, Abstract"},{"comment":"The hit criterion in Eq. (4) uses only the argmax position and ignores how much attention mass is placed on the aligned image token. A head that puts 30% of its mass on the target is counted identically to one that puts 100%. Moreover, because OCR images have text regions covering a large fraction of the image, a head with generic concentration on salient patches or attention sinks may accumulate hits by chance. To support the interpretation that these heads are specifically responsible for visual understanding, please add a control, for example a permutation test that randomly reassigns token-to-region correspondences, or a baseline of attention to random image patches, and show that observed scores exceed chance.","section":"Sec. 3.2, Eq. (4)"},{"comment":"The experimental comparisons are reported as point estimates without error bars, multiple seeds, or significance tests, and several differences at moderate budgets are small (e.g., Table 6 at 256 tokens: SparseMM 64.52 vs AdaKV 64.65 on GQA; at 512 tokens both achieve 81.52 on MMBench, with PyramidKV at 81.53). In addition, the hyperparameter rho is tuned on OCRBench (Table 3) and the benchmark suite is dominated by OCR-style tasks, raising a risk of selection bias. Please report variance across at least three seeds and either fix rho on a separate validation set or show sensitivity across benchmarks for all evaluated values.","section":"Sec. 4.2, Tab. 3, Fig. 4"}],"minor_comments":[{"comment":"In Eq. (3) and the surrounding text, the normalization of the visual score matrix is not specified; 'These scores are then normalized' (end of Sec. 3.2) is vague. Please state the normalization formula and whether it is applied per layer, per head, or globally.","section":"Sec. 3.2"},{"comment":"Algorithm 1 does not specify how output tokens that do not match any OCR text/bbox pair (e.g., function words, punctuation) are handled; this could bias the hit counts. Clarify the matching procedure and the treatment of non-matching tokens.","section":"Algorithm 1"},{"comment":"The efficiency numbers in the abstract (1.38x acceleration, 52% memory reduction) are not directly traceable to a table: Table 2 reports 52.9/37.1 = 1.43x latency reduction, and the Sec. 4.3 memory example gives 32.87 to 17.38 GB (47%). Please state the exact configuration for the abstract numbers.","section":"Abstract, Sec. 4.3, Tab. 2"},{"comment":"Typos: 'sparity' in the abstract should be 'sparsity'; 'Mm asking' in the Fig. 7 caption should be 'Masking'. Additionally, the statement in Sec. 4.2 that the random head method is comparable to SnapKV for the MHA model is not fully supported by Table 5 at low budgets on DocVQA (e.g., 0.4977 vs 0.4868 at 64 tokens), and should be qualified.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper would be considerably stronger if the authors added a null-model control for the visual score and a quantitatively defined threshold for the sparsity claim. If the authors cannot provide these, the paper may be reframed as a pure engineering contribution (head-aware cache allocation) with a more modest interpretation. The current manuscript's central scientific claim is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SparseMM is a genuinely useful engineering contribution wrapped in a slightly overclaimed scientific story. The cache-allocation scheme works in the reported experiments, the code is out, and the OCR-anchored head scoring is a clever idea. But the 'less than 5% of heads are visual' claim has no stated threshold, and the evaluation skips FastV and ElasticCache, the two baselines that matter most. I'd send it out for review, but I'd want those issues fixed before I'd cite the head-sparsity claim.\n\nWhat's new: the visual score in Eq. 3-4 is a simple, training-free way to rank heads by OCR-aligned activation, and SparseMM's three-part budget (local window + uniform baseline + score-proportional) is a sensible synthesis of SnapKV and AdaKV with a visual prior. The experiments cover three backbones including GQA, and the robustness section shows the identified heads transfer across OCR datasets. The masking analysis (Fig. 7) is the strongest piece: masking the high-scoring heads degrades OCRBench and TextVQA more than masking random heads. That is real evidence the score is picking out functionally important heads. Credit also for shipping code and full numerical tables.\n\nThe soft spots are real but addressable. First, the 'less than 5%' claim is unfalsifiable as stated: no threshold on the score separates visual from non-visual heads. Second, the hit definition only checks whether the argmax token falls in the target patch; a head with 30% of its mass on the patch counts the same as one with 100%, so generic salience or sink attention can inflate scores. Third, rho=0.1 is tuned on OCRBench, which is also the main benchmark where the method looks best, so there is a mild selection effect. Fourth, no error bars or multi-seed results are reported, which matters for a method with tuned hyperparameters. Finally, FastV and ElasticCache are both absent from the comparisons; ElasticCache is cited but not evaluated, and it is arguably the closest prior art for MLLM cache management.\n\nOn the circularity concern: the visual-head identity is defined from the same OCR score matrix used to compute the sparsity percentage, and the benchmarks are OCR-dominated. But the multiple-choice results (MMBench, VQAv2) show the method still holds up, so the story generalizes more than the pure circularity worry suggests. It would still help to show a permutation control, e.g., shuffling the visual scores and demonstrating SparseMM degrades to the level of random allocation.\n\nWho should read this: anyone working on MLLM inference efficiency or attention analysis. It deserves a serious referee, because the empirical result is concrete, reproducible, and plausibly useful. My recommendation: engage, but push for a stated threshold, the missing baselines, and at least one control experiment before accepting the broader scientific claim.","headline":"A useful, reproducible KV-cache allocation method behind an under-validated 'visual head' story; the <5% claim lacks a threshold and FastV/ElasticCache are missing.","tokens_in":18282,"tokens_out":3333,"would_cite":true,"duration_ms":36407,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In multimodal LLMs, fewer than 5% of attention heads carry visual understanding, and SparseMM skews KV-cache budgets toward those heads to accelerate generation without losing accuracy.","keywords":["visual heads","head sparsity","KV cache compression","multimodal large language models","attention analysis","OCR grounding","asymmetric cache allocation","inference acceleration"],"falsifier":"Compute the same visual scores on a text-only control: run the OCR task with the image replaced by a blank or scrambled patch grid while keeping the same output tokens, and check whether the top-scoring visual heads still receive frequent argmax hits on image-token positions; if they do, the score does not isolate visual function. Alternatively, shuffle the mapping between bounding boxes and image patches and show that the head ranking barely changes, which would indicate the score tracks attention concentration rather than visual content.","tokens_in":1634,"feed_emoji":"⚡","tokens_out":1911,"duration_ms":77655,"temperature":0.7,"pith_summary":"This paper claims that multimodal large language models inherit a sparse visual faculty: fewer than 5% of attention heads carry essentially all of the model's visual understanding, across both multi-head and grouped-query attention architectures. The authors propose a training-free OCR-anchored procedure that scores each head by how often its strongest attention lands on the image patch corresponding to the text token being generated. They then use those scores to allocate asymmetric KV-cache budgets, keeping more cache for visual heads and compressing the rest, and report that this preserves benchmark accuracy while accelerating decoding and cutting memory. The significance is that visual semantics in MLLMs can be identified cheaply before inference and exploited for efficiency without retraining.","feed_headline":"Fewer than 5% of attention heads do the visual work in MLLMs","feed_subtitle":"An OCR-derived head score lets SparseMM cut cache by 52% and speed up generation 1.38x without retraining","key_machinery":"The machinery is the visual-score matrix S, computed without training by matching each generated OCR token to its ground-truth bounding box, mapping that box to image tokens, and incrementing a head's score when its argmax attention lands inside those tokens. The second piece is the SparseMM cache allocator, which splits each head's budget into a local window of the newest tokens, a small uniform baseline share, and a score-proportional share of the remaining cache; the surviving keys in each head are then chosen by top-k attention within a 32-token observation window. Together these convert a static head-level sparsity measurement into a per-head, per-layer cache budget.","core_discovery":"The central claim is that visual understanding in MLLMs is concentrated in a sparse set of attention heads, termed visual heads, which emerge consistently across model families and attention designs. The authors demonstrate this by defining a visual score per head: over OCR images, for each generated token they locate the aligned image patch, and a head scores a hit if its argmax attention falls on one of those image tokens; scores are normalized and aggregated over 1,000 images. The claim is that heads in the top few percent of this score matrix are genuinely visual: masking them hurts OCR and VQA performance far more than masking the same number of random heads, and they generalize to non-OCR tasks. On this basis SparseMM gives visual heads a larger share of the KV-cache budget and reports parity with full-cache performance at a small fraction of the cache, with concrete efficiency gains.","pith_inferences":["The single-argmax criterion could conflate visual heads with generic attention concentration, such as attention sinks or language priors; a natural control is to run the same scoring on text-only inputs and check whether the same heads still receive frequent argmax hits on image-token positions.","The paper's own ablation shows that removing the uniform baseline (rho = 0) collapses performance for the Mistral model, so the reported gains depend on the hybrid allocation rather than on visual-head scores alone.","The identification stage costs 1,000 OCR forward passes; whether SparseMM is an end-to-end win depends on amortizing that one-time profiling cost over many downstream inferences for the same model.","A testable extension is to replace OCR supervision with caption-based alignment signals to see whether the same sparse head set emerges, which would show whether the phenomenon is intrinsic to the base LLM or shaped by the instruction-tuning data."],"forward_implications":["If the sparsity claim is right, MLLM inference can be accelerated without retraining by concentrating KV-cache budget on the small visual-head subset.","On DocVQA, LLaVA-NeXT-Vicuna-7B matches full-cache accuracy at 20% of the cache and Qwen2-VL-7B-Instruct at 5.3% of the cache.","With 32K input tokens, LLaVA-NeXT-Vicuna-7B keeps decoding latency nearly constant, achieving roughly a 1.87x speedup and cutting peak memory from 32.87 GB to 17.38 GB; overall the paper reports 1.38x real-time acceleration and a 52% memory reduction.","Visual heads identified on OCR transfer to object recognition and scene understanding, so the budget skew applies beyond document tasks.","Masking the top 5% of visual heads causes a larger performance drop than masking an additional 5%, consistent with a sparse but indispensable distribution."],"supporting_citations":[{"why":"Supplies the 1,000 Synthdog OCR images used to compute the visual-score matrix.","marker":"[18]"},{"why":"Provides the observation-window top-k KV-cache selection mechanism that SparseMM adapts.","marker":"[22]"},{"why":"Inspires dynamic per-head budget allocation and serves as a main baseline.","marker":"[13]"},{"why":"Hierarchical cache-budget baseline that SparseMM is compared against.","marker":"[4]"},{"why":"Defines the LLaVA-NeXT model family used in the main experiments.","marker":"[27]"},{"why":"Provides the Qwen2-VL-7B-Instruct GQA backbone used in experiments.","marker":"[41]"},{"why":"Supplies the Vicuna-7B MHA backbone for LLaVA-NeXT-Vicuna-7B.","marker":"[8]"},{"why":"Supplies the Mistral-7B GQA backbone for LLaVA-NeXT-Mistral-7B.","marker":"[17]"},{"why":"DocVQA benchmark where parity is reported at 5.3% of the cache.","marker":"[35]"}],"fun_headline_variants":["Only 5% of attention heads drive MLLM visual understanding","SparseMM finds visual heads, cuts cache 52%, speeds up 1.38x","Training-free SparseMM targets the <5% visual heads for speed","MLLM visual work needs only 5% of heads, SparseMM exploits it","SparseMM: sparse visual heads boost inference 1.38x, save memory"],"cache_read_input_tokens":20352,"weakest_assumption_plain":"The argument assumes that repeatedly observing a head's argmax attention land on the OCR-aligned image patch measures that head's causal contribution to visual understanding; if argmax hits instead reflect general attention concentration, such as attention sinks or text priors, the sparsity phenomenon and the cache advantage would not be specifically visual.","fun_headline_variants_meta":{"raw":{"variants":["Only 5% of attention heads drive MLLM visual understanding","SparseMM finds visual heads, cuts cache 52%, speeds up 1.38x","Training-free SparseMM targets the <5% visual heads for speed","MLLM visual work needs only 5% of heads, SparseMM exploits it","SparseMM: sparse visual heads boost inference 1.38x, save memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000444,"raw_usage":{"total_tokens":2255,"prompt_tokens":959,"completion_tokens":1296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1189}},"tokens_in":575,"tokens_out":1296,"duration_ms":12421,"temperature":1.0,"reasoning_tokens":1189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:21:18.916461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same visual scores on a text-only control: run the OCR task with the image replaced by a blank or scrambled patch grid while keeping the same output tokens, and check whether the top-scoring visual heads still receive frequent argmax hits on image-token positions; if they do, the score does not isolate visual function. Alternatively, shuffle the mapping between bounding boxes and image patches and show that the head ranking barely changes, which would indicate the score tracks attention concentration rather than visual content.","supporting_citations":[{"cited_title":"Snapkv: Llm knows what you are looking for before generation","cited_arxiv_id":null,"evidence_quote":"Provides the observation-window top-k KV-cache selection mechanism that SparseMM adapts."},{"cited_title":"Llava-next: Improved reason- ing, ocr, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the LLaVA-NeXT model family used in the main experiments."},{"cited_title":"Qwen2-vl: To see the world more clearly","cited_arxiv_id":null,"evidence_quote":"Provides the Qwen2-VL-7B-Instruct GQA backbone used in experiments."},{"cited_title":"Docvqa: A dataset for vqa on document images","cited_arxiv_id":null,"evidence_quote":"DocVQA benchmark where parity is reported at 5.3% of the cache."}],"review_version":1}