{"id":"d8ce16e6-a782-4b4f-b2a2-89d9a3d1519b","arxiv_id":"2412.01818","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VisPruner uses visual-encoder attention plus similarity-based duplicate removal to prune visual tokens before the language model, sustaining high accuracy at extreme reduction ratios without training.","lead":"Vision-language models spend most of their computation on image tokens. This paper introduces VisPruner, a training-free method that picks which image tokens to keep using attention from the visual encoder and removes duplicates, so a model can run on far fewer visual tokens while keeping most of its accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is only tested on coarse QA/hallucination benchmarks; question-independent diversity in CLIP feature space may not preserve task-critical small-detail or background tokens, so the reported win may not generalize.","rationale":"The paper is well-executed: the attention-shift and attention-dispersion analyses are compelling, the ablations show both components matter, and the code is provided. My concern is not about internal consistency but about external validity of the headline claim. The method's only selection criteria are [CLS] saliency and visual similarity, and neither is guaranteed to preserve question-relevant small-detail or background tokens. The reader's weakest assumption is essentially the same concern, so I agree with the conditional verdict. A benchmark or oracle-retention test on detail- and layout-sensitive tasks would settle whether the claim generalizes or should be scoped to coarse QA and hallucination tasks.","tokens_in":29780,"tokens_out":7703,"duration_ms":84683,"concrete_test":"Re-run VisPruner, FastV, and SparseVLM at 32 and 64 retained tokens on a fine-grained captioning benchmark (NoCaps or Flickr30k with CIDEr) and a spatial-reasoning benchmark (VSR or BLINK), reporting full-model-normalized scores. Additionally, on a region-annotated VQA set, measure the retention rate of ground-truth answer-relevant patches among VisPruner's retained tokens stratified by object size; if answer-region retention is not clearly above random for small-object questions, the diversity criterion is not preserving the task-critical details the central claim requires.","verdict_should_be":"UNCHANGED","load_bearing_attack":"VisPruner's central claim is that [CLS] attention from the visual encoder plus similarity-based diversity is a more reliable pruning signal than text-visual attention. The load-bearing premise is that question-independent selection can retain the tokens actually needed for the answer. This is not established. The important-token budget at 32/576 tokens is tiny, and the diverse-token budget from Eq. (4) and Algorithm 1 is smaller still; diversity is computed by cosine similarity in CLIP feature space, which is semantically coarse and known to be foreground-biased. For small objects, fine text, and background layout, a patch can be visually redundant yet task-critical, and the algorithm has no mechanism to keep it. Table 1's ten benchmarks are dominated by coarse QA and object-presence/hallucination tasks; none require detailed captioning or fine-grained spatial reasoning. Thus the consistent margin over FastV, SparseVLM, and VisionZip may reflect benchmark task distribution rather than a general advantage of visual-cue pruning, which is the paper's broader claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies token pruning in vision-language models and argues that text-visual attention inside the language model is a poor pruning signal, because of positional bias (attention shift) and flat attention concentration (attention dispersion). The authors propose VisPruner, a training-free, plug-and-play method that selects a small set of important visual tokens using the visual encoder's [CLS] attention and then supplements them with diverse tokens chosen by a cosine-similarity based duplicate-removal procedure. The method is evaluated on LLaVA-1.5, LLaVA-NeXT, Video-LLaVA, Qwen-VL, InternVL, and CogVLM across image VQA, hallucination, and video QA benchmarks, and compared with ToMe, FastV, SparseVLM, LLaVA-PruMerge, and VisionZip. The main empirical claim is that at 32 retained tokens on LLaVA-1.5-7B, VisPruner retains 91.5% of full-model average performance while pruning 94.4% of visual tokens, consistently outperforming text-visual attention baselines and the visual merging baseline VisionZip. The paper also reports FLOPs and latency reductions.","tokens_in":29882,"tokens_out":4845,"duration_ms":47958,"significance":"If the reported results are firm, the paper makes a useful and practical contribution: it challenges the dominant text-visual attention paradigm for VLM token pruning, provides a simple visual-cue based alternative that requires no training, and demonstrates the approach across several model families. The analysis of attention shift and dispersion is a concrete, testable observation that other researchers can build on. The paper also gives credit to concurrent work sharing similar conclusions and provides code, which supports reproducibility. However, the paper's broader claim that visual cues are generally more reliable than text-visual attention is only tested on a specific set of benchmarks, and several load-bearing numerical claims need correction or additional statistical support before the conclusions can be taken at face value.","major_comments":[{"comment":"The abstract and introduction state that VisPruner 'can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%' and that pruning 94.4% of visual tokens in LLaVA-1.5-7B reduces inference FLOPs by 'more than 95%'. Supplementary Table 6 for LLaVA-1.5-7B at 95% token reduction reports FLOPs of 0.76 T vs 8.02 T (a 90.5% reduction) and CUDA time of 77.15 ms vs 107.26 ms (a 28% reduction). The 91% FLOP and 75% latency figures match LLaVA-NeXT-7B in Table 8, not LLaVA-1.5-7B. The efficiency claims must be corrected and attributed to the appropriate model.","section":"Abstract, Section 1, Section 5.7, Supplementary Table 6"},{"comment":"No standard deviations or repeated-run statistics are reported, and several of the claimed margins over VisionZip are small. For example, at 128 retained tokens, SQA-IMG differs by 0.2 points (69.1 vs 68.9), TextVQA by 0.2 points (57.0 vs 56.8), MMBench-CN by 0.6 points (57.3 vs 56.7), and MM-Vet by 1.1 points (33.7 vs 32.6). Without variance estimates, the statement that VisPruner consistently outperforms VisionZip is not statistically supported.","section":"Section 5.2, Table 1"},{"comment":"The hyperparameter r in Eq. (4) determines the split between attention-based important tokens and similarity-based diverse tokens, but its value at each operating point is never reported and no sensitivity analysis is provided. The ablation in Fig. 5 only compares the 'Important' variant (which appears to use the entire budget for attention-selected tokens) with the full VisPruner; it does not isolate the effect of r on the final performance. Since r is a free parameter of the method, the main results are not fully reproducible without reporting its value and stability.","section":"Section 4.1, Section 4.2, Section 5.6, Fig. 5"},{"comment":"The ten image benchmarks are dominated by visual question answering and object-presence/hallucination tasks. None of them requires detailed captioning or fine-grained spatial layout reasoning. VisPruner's diverse-token component is question-independent and based on cosine similarity in CLIP feature space, so the claim that visual-cue pruning is generally superior to text-visual attention is not established for tasks in which small-detail or background tokens are task-critical. The authors should either add such benchmarks (e.g., NoCaps or Flickr30k captioning, spatial reasoning subsets) or explicitly scope the claims to the tested task distribution.","section":"Section 5.1, Section 5.2, Section 4.2"},{"comment":"The pseudocode is difficult to reconcile with the text and with Eq. (4). The variable r is used both as the important-token ratio in Eq. (4) and as the per-iteration removal count in Algorithm 1. The input 'normalized' is not defined, the update r = min(r, remaining_idx.shape[0] - n) is not explained, and the termination condition is implicit. The algorithm should be rewritten with distinct variable names, a clear initialization and termination condition, and a stated guarantee that the output size exactly equals R(1-r) diverse tokens.","section":"Algorithm 1, Section 4.2"}],"minor_comments":[{"comment":"The sentence 'Based on the analysis, We propose VisPruner' has an incorrect capitalization; 'We' should be 'we'.","section":"Abstract"},{"comment":"The upper bound row for LLaVA-1.5-13B contains extra duplicated values ('85.99 107.26 85.88') that do not match the column layout; the table should be reformatted.","section":"Supplementary Table 6"},{"comment":"The caption does not define what 'Important' means in terms of r or the token budget; please specify the hyperparameter setting used for the ablation.","section":"Fig. 5 caption"},{"comment":"For Qwen-VL, InternVL, and CogVLM, VisPruner is only compared with FastV; the absence of SparseVLM and VisionZip on these architectures should be acknowledged, since the main tables compare only with FastV.","section":"Section 5.5"},{"comment":"There is a typo: 'adpot' should be 'adopt'.","section":"Supplementary Section 8.1"},{"comment":"The page list for reference [9] is out of order ('1, 3, 5, 6, 7, 8, 2'); please clean up the reference metadata.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VisPruner is a solid empirical paper on training-free token pruning for VLMs. The core idea—use CLIP [CLS] attention to pick important tokens, then add back similarity-diverse tokens—is simple and works better than text-attention-based pruning across multiple architectures and reduction ratios. What's actually new is the diagnosis: the authors show that text-visual attention in the LLM suffers from both a positional shift (due to RoPE) and a dispersion problem, making it a poor pruning signal. This analysis is concrete and well-supported by the figures. The practical bonus is that pruning happens before the LLM, so it is compatible with FlashAttention, which the efficiency tables confirm.\n\nThe experimental work is extensive: LLaVA-1.5, LLaVA-NeXT, Video-LLaVA, Qwen-VL, InternVL, CogVLM, ten image benchmarks, three video benchmarks, and comparisons against FastV, SparseVLM, ToMe, PruMerge, and VisionZip. The ablations show both components contribute. They also release code, which makes the results reproducible.\n\nThe soft spots are mostly about rigor and scope. There are no error bars or repeated trials; every number is a single run. The r hyperparameter in Eq. (4) is not swept, and the paper gives no hint of how sensitive the method is to it. More substantively, the benchmark suite is dominated by question-answering and object-presence/hallucination tasks. MME and MMBench cover some fine-grained recognition, but there is no detailed captioning benchmark (NoCaps, COCO Captions) and no dedicated fine-grained spatial reasoning set. The stress-test concern that question-independent diversity in CLIP space may not preserve small, task-critical patches is plausible. However, it is not fatal: the diversity component is designed for background preservation, and the ablation shows it helps on POPE and TextVQA. The concern is about generalization of the reported win to tasks like detailed captioning, which the paper does not address.\n\nOverall, this is a worthwhile contribution. The diagnosis of text-visual attention is useful in itself, and the method is likely to be cited. I would send it to review, with requests for an r-sweep, error bars or variance estimates, and at least one detailed-captioning benchmark. If those hold up, the paper's claim will be much stronger.","headline":"VisPruner is a genuinely useful, training-free pruning method backed by a solid analysis of why text-attention pruning fails; the weaknesses are missing robustness details and benchmark coverage, not the core idea.","tokens_in":30477,"tokens_out":2834,"would_cite":true,"duration_ms":27037,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that text-visual attention in large vision-language models is a misleading pruning signal and that a training-free method using the visual encoder's own attention plus token diversity keeps 91.5% of LLaVA-1.5-7B's average…","keywords":["token pruning","vision-language models","visual attention","CLS attention","training-free inference","inference acceleration","attention shift","token diversity"],"falsifier":"Run the identical 32-token pruning comparison on a benchmark built around small background details, such as a detailed captioning or spatial-relation benchmark: if VisPruner's accuracy falls toward random or drops below FastV at the same token budget, the claim that visual-encoder attention plus diversity preserves needed information is falsified.","tokens_in":1564,"feed_emoji":"✂️","tokens_out":1772,"duration_ms":64392,"temperature":0.7,"pith_summary":"The paper is trying to establish that the attention scores text tokens pay to image tokens inside a vision-language model are the wrong signal to use when pruning visual tokens. It shows two problems with that signal: a positional \"shift\" that biases selection toward lower image patches, and a \"dispersion\" that spreads attention too evenly to separate important tokens from redundant ones. The proposed alternative, VisPruner, keeps a small set of high-attention tokens from the image encoder and adds back a similarity-diverse set of the remaining tokens, all before the language model runs. The reported payoff is that LLaVA-1.5-7B retains 91.5% of its average score across ten benchmarks with only 32 of 576 visual tokens, and the method beats text-attention and token-merging baselines at every tested token budget. A reader should care because visual tokens dominate inference cost, and this suggests that a cheap visual-saliency signal plus diversity can outperform question-sensitive attention for pruning.","feed_headline":"Drop 94% of visual tokens, keep 91.5% of accuracy","feed_subtitle":"Image-encoder attention plus diversity beats text-attention pruning, training-free, across vision-language benchmarks.","key_machinery":"The machinery is a two-step, training-free selection performed before the language model. First, the [CLS] attention row of the visual encoder, defined as the attention from the global class token to each image patch, is averaged over heads and thresholded to pick a small set of important tokens that capture concentrated foreground saliency. Second, on the remaining tokens the method iteratively removes tokens most similar to others under cosine similarity, leaving a diverse set that covers background areas; the important and diverse sets are then re-sorted into original image order and projected into the language model. This design lets pruning happen before the LLM, which keeps the method compatible with attention optimization techniques and avoids the positional bias found in in-LLM attention.","core_discovery":"The paper's central discovery is that text-visual attention inside the language model is not a trustworthy basis for visual token pruning, and that a better signal comes from the visual encoder itself. It documents an attention shift caused by rotary position embeddings, where later text tokens preferentially attend to lower image patches from the first layer onward, and an attention dispersion that leaves text-visual attention too uniform to identify important tokens. VisPruner replaces that signal with the [CLS] attention of the visual encoder, which is concentrated on foreground content, and then adds a similarity-based diverse set of the remaining tokens so background information survives. On LLaVA-1.5-7B, retaining 32 of 576 visual tokens is reported to preserve 91.5% of average performance over ten benchmarks, and the method is reported to outperform FastV, SparseVLM, and VisionZip at 128, 64, and 32 retained tokens.","pith_inferences":["Because the benchmarks are mostly question-answering and hallucination tasks, the paper does not establish that the method survives detailed captioning or tasks where answers depend on small background objects; testing those would settle the generality.","The diversity term is question-blind, so a natural extension would condition the retained background tokens on the text prompt, a variant the paper does not explore.","If visual-encoder attention is indeed the better pruning signal, then improving or aligning that attention with the language model should directly improve pruning quality, pointing toward co-design of encoders and pruners."],"forward_implications":["At a 94.4% visual-token reduction, LLaVA-1.5-7B retains 91.5% of its average score across the ten evaluated benchmarks without any fine-tuning.","Pruning before the language model cuts FLOPs and latency more than in-LLM pruning and works with FlashAttention, giving LLaVA-1.5-7B a reported 91% FLOP reduction and 75% latency reduction.","The method transfers beyond LLaVA-1.5, preserving more relative performance than FastV on LLaVA-NeXT, Video-LLaVA, Qwen-VL, InternVL, and CogVLM at matching token budgets.","At high reduction ratios, question-independent visual saliency plus diversity outperforms text-visual attention and token merging, suggesting that merging similar tokens can inject noise relative to simply dropping them.","The same visual-cue approach extends to longer visual sequences, including high-resolution images and video inputs, where redundancy is higher and relative gains are larger."],"supporting_citations":[{"why":"Supplies the main text-visual attention pruning baseline and the \"last attention\" definition the analysis compares against.","marker":"[9]"},{"why":"Provides the CLIP visual encoder whose [CLS] attention is used to select the important visual tokens.","marker":"[43]"},{"why":"RoFormer's rotary position embedding is cited as the source of the long-term decay behind the attention shift.","marker":"[47]"},{"why":"Documents the layer-wise positional bias in higher RoPE attention dimensions that the shift analysis relies on.","marker":"[18]"},{"why":"SparseVLM is the strongest text-attention baseline that VisPruner is reported to outperform at each token budget.","marker":"[65]"},{"why":"VisionZip is the visual token merging baseline that VisPruner outperforms at equal retained token counts.","marker":"[59]"},{"why":"LLaVA-PruMerge is the adaptive merging baseline used as a comparison across reduction ratios.","marker":"[45]"}],"fun_headline_variants":["Visual cues beat text attention for VLM token pruning","VisPruner prunes 94% of visual tokens, keeps accuracy","Training-free: visual attention trims VLM compute by 91%","Drop 576 to 32 tokens, preserve 91.5% accuracy in VLMs","Use visual cues, not text, to slash VLM latency by 75%"],"cache_read_input_tokens":32640,"weakest_assumption_plain":"The load-bearing premise is that the image encoder's [CLS] attention marks the tokens the language model needs and that similarity-based diversity covers what it misses; if important information lives in small, background, or question-specific tokens, high-ratio pruning will collapse.","fun_headline_variants_meta":{"raw":{"variants":["Visual cues beat text attention for VLM token pruning","VisPruner prunes 94% of visual tokens, keeps accuracy","Training-free: visual attention trims VLM compute by 91%","Drop 576 to 32 tokens, preserve 91.5% accuracy in VLMs","Use visual cues, not text, to slash VLM latency by 75%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1288,"prompt_tokens":986,"completion_tokens":302,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":602,"tokens_out":302,"duration_ms":3408,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:53:04.370701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical 32-token pruning comparison on a benchmark built around small background details, such as a detailed captioning or spatial-relation benchmark: if VisPruner's accuracy falls toward random or drops below FastV at the same token budget, the claim that visual-encoder attention plus diversity preserves needed information is falsified.","supporting_citations":[],"review_version":1}