{"id":"04583dfa-913c-4652-bc11-04e689d2ce01","arxiv_id":"2411.15446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"freePruner selects 50 percent of visual tokens using attention-based importance and keeps accuracy close to the original model, enabling a training-free about 2x acceleration for LMMs.","lead":"This paper presents freePruner, a method that speeds up vision-language AI models by dropping half of the image tokens before they reach the language model, with no retraining needed. The appeal is a possible 'free lunch' speedup for open multimodal models whose training data is not available.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 2x speedup rests on roofline estimates; measured wall-clock latency could be materially lower, so the central acceleration claim is unverified.","rationale":"The paper's strongest empirical contribution is Table 1: at 50% tokens, freePruner stays within about one point of LLaVA-1.5 on most benchmarks and beats training-free PruMerge+, and the ablation in Fig. 8 supports the claim that complementary tokens add value beyond random tokens. That part is credible as reported. The weakness is the other half of the headline claim, '2x acceleration.' Section 4.4 and Table 3 are explicitly roofline estimates computed with LLM-Viewer, not measured latency. The table itself reveals that memory traffic falls only 20% (39.2 to 31.5 GB) when OP count falls about 48% (15.9 to 8.2 TB) for LLaVA-13B FP16, so a purely compute-proportional time model is questionable on a memory-limited GPU. The estimate also excludes the unpruned visual encoder forward pass and the cost of materializing attention maps for selection, both of which occur before the LLM. A user could therefore obtain materially less than 2x end-to-end. This is more load-bearing than the Eq. 3 semantic-validity question because the benchmark table already provides an empirical check of the selection metric's adequacy, whereas no corresponding measured check exists for the speedup. The reader flagged the roofline issue but selected Eq. 3 as the weakest assumption; I partially agree with the reader's overall conditional assessment but consider the unmeasured speedup the single decisive gap. A wall-clock experiment on the claimed hardware with the authors' exact hyperparameters would settle it; the missing hyperparameters make this test impossible until supplied. The novelty concern about FastV is real but does not bear directly on the correctness of the empirical speedup claim.","tokens_in":15944,"tokens_out":7541,"duration_ms":75983,"concrete_test":"Run LLaVA-1.5-13B (Vicuna-13B) on an A6000 with the paper's 50% visual-token configuration and the authors' exact Ls, k, and outlier threshold; measure wall-clock prefill and full-generation latency over at least 200 VQAv2 samples using torch.cuda.Event, and separately time the visual encoder plus selection overhead. Compare against unpruned LLaVA-1.5 under identical batching. If measured prefill speedup is below roughly 1.75x, or end-to-end speedup below roughly 1.5x, the '2x acceleration' headline is not supported. If the authors cannot supply the hyperparameters, the main result is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that freePruner achieves 2x acceleration is supported only by LLM-Viewer roofline estimates in Table 3, not by wall-clock measurements. This matters because Table 3 reports only estimated LLM prefill time; the full pipeline still runs the unpruned visual encoder and pays for computing or materializing per-layer attention maps used for selection. The table itself shows memory traffic falling only 20% (39.2 to 31.5 GB for LLaVA-13B FP16) while OP count falls about 48% (15.9 to 8.2 TB), so a purely compute-proportional speedup model is suspect on a memory-limited GPU. For VideoLLaVA, the per-frame encoder and selection overhead is not included in the 4x estimate at all. Thus the headline acceleration figure is not established by the evidence in the paper. A secondary reproducibility issue compounds this: Algorithm 1 requires hyperparameters Ls, k, and the adaptive outlier threshold, but none are specified, so the exact 50% token configuration cannot be reproduced to verify either accuracy or latency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces freePruner, a training-free token reduction method for large multimodal models. It selects a subset of visual tokens in two stages: pivotal tokens are identified by a contribution-degree metric based on total inbound attention (Eq. 3) over layers, and complementary tokens are chosen in the penultimate layer by their attention to pivotal tokens. The selected tokens are passed to the LLM without any retraining. Experiments on LLaVA-1.5 show accuracy comparable to, or slightly better than, the full model on VQAv2, TextVQA, POPE, SQA, MME, and MMBench at roughly 50% token count; the method is also applied to VideoLLaVA and combined with post-training quantization. The paper claims a 2x (and 4x for video) acceleration, but this claim is based on LLM-Viewer roofline estimates in Table 3 rather than on wall-clock latency measurements.","tokens_in":16207,"tokens_out":8941,"duration_ms":79821,"significance":"The training-free property is practically valuable, and the benchmark evidence is mostly consistent with the claim of maintained accuracy: the losses on VQAv2 and MME are small, while POPE, SQA, and TextVQA show small gains. The ablations in Fig. 8 support the complementary-token module, and the orthogonality to post-training quantization is a useful property. If the speedup were measured end-to-end, freePruner would be a simple and broadly applicable acceleration baseline. At present the significance is moderated by the absence of measured latency and by missing hyperparameter specifications, which prevent independent verification of the central acceleration claim.","major_comments":[{"comment":"The headline acceleration claim is not established because all 'Prefill Time' values in Table 3 are LLM-Viewer roofline estimates rather than wall-clock measurements. The table itself shows that for LLaVA-13B FP16 the accessing-memory cost drops only from 39.2 GB to 31.5 GB (about 20%) while OP count drops by 48%; on a memory-bound A6000 workload, speedup therefore cannot be assumed to scale with OP count. The reported prefill estimates also exclude the cost of computing the attention maps and contribution degrees used for token selection, and for VideoLLaVA they exclude per-frame encoder and selection overhead. Please report end-to-end wall-clock latency, including selection overhead, on the target hardware.","section":"Sec. 4.4 / Table 3"},{"comment":"The hyperparameters Ls, k, and the adaptive outlier threshold for complementary tokens are never specified, yet the 50%-token results in Table 1 and Fig. 7 depend on them. Because Algorithm 1 outputs an adaptive m, the mapping from '50% of tokens' to a concrete setting of Ls, k, and threshold is not reproducible. Please state the exact values used and include a sensitivity analysis over Ls and k.","section":"Algorithm 1 / Sec. 4.1"},{"comment":"The contribution-degree metric is the load-bearing selection signal, but the paper does not provide evidence that inbound attention excluding self tracks the information the LLM actually needs. The text itself acknowledges attention sinks (Sec. 3.2 and Fig. 5), which can produce high contribution scores for positional or background tokens. A minimal control would be random 50% token selection at the same budget, and a task-based importance ranking (e.g., gradient-based or leave-one-out on a small validation set). Without such a control, the accuracy results could reflect the robustness of LLaVA to arbitrary token subsampling rather than the specific selection metric.","section":"Eq. (3) / Sec. 3.2"},{"comment":"The claim that this is 'the first exploration of training-free token reduction specifically designed for LMMs' is not correct as written. FastV (Liang Chen et al., 'An Image is Worth 1/2 Tokens After Layer 2', 2024) proposes training-free token pruning for multimodal LLMs and is neither cited nor compared against. Please correct the novelty statement and add a direct experimental comparison at matched token budgets.","section":"Sec. 2 (last paragraph) / Sec. 4"}],"minor_comments":[{"comment":"'freePruner halves the visual tokens, which results in a twofold increase in prefill times' should read 'a twofold reduction in prefill time'.","section":"Sec. 4.4"},{"comment":"The phrase 'the iL-th layer' is ambiguous; it should be the l-th layer or explicitly the penultimate layer.","section":"Algorithm 1, line 8"},{"comment":"'Fig.6 illustrates the performance comparisons among three groups' is a wrong cross-reference; the comparison plot is Fig. 8.","section":"Sec. 4.5"},{"comment":"Please state the index convention in Eq. (1)-(3), i.e., whether A_l[i,j] is attention from token i to token j; this determines whether Eq. (3) is an inbound or outbound sum.","section":"Sec. 3.2 / Eq. (1)-(3)"},{"comment":"The abstract and conclusion claim applicability to 'any open-source LMM', but experiments cover only LLaVA-1.5, LLaVA-Next, and VideoLLaVA; please qualify the claim or add a third model family.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The main obstacles to acceptance are the missing wall-clock latency measurements and the missing hyperparameter specifications; both are fixable with additional experiments. The roofline-based speedup estimate should not remain the sole evidence for the central acceleration claim. I also recommend asking the authors to address the unmentioned FastV prior work before restating the novelty claim. I do not see grounds for rejection, since the accuracy results are credible and the method is simple."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The actual new thing here is the two-stage selection: pivotal tokens chosen by inbound attention minus self, then complementary tokens chosen by attention to pivotal tokens in the penultimate layer. That specific recipe doesn't appear in PruMerge+ or TokenPacker, and the ablation in Fig. 8 does show the complementary stage adds value over random selection. The accuracy results are the paper's real strength: at 50% tokens on LLaVA-1.5 7B/13B, the model stays within a couple points of the full model on VQAv2, MME, MMBench, and actually gains on POPE, SQA, and TextVQA. The video numbers are even more impressive (8x fewer tokens, no drop). Those results look real, and the paper is honest that the method is a heuristic.\n\nThe soft spots are mostly around the efficiency claim. The 2x acceleration is an LLM-Viewer roofline estimate, not wall-clock latency. That wouldn't be a problem by itself, but Table 3 shows the problem: for LLaVA-13B FP16, OPs drop from 15.9 to 8.2 TB (about half), but memory traffic only drops from 39.2 to 31.5 GB (~20%). On a memory-bound GPU, a compute-proportional speedup is not credible. The paper needs actual end-to-end latency measurements, including the cost of computing attention maps for selection. Without that, the headline \"2x\" is not established.\n\nThe other real issue is reproducibility. Algorithm 1 depends on the starting layer Ls, the pivotal count k, and an adaptive outlier threshold. None of these are given anywhere in the paper or appendix. So the exact 50%-token configuration that produces all the tables cannot be reproduced. That's a fixable but necessary change.\n\nThe related-work gap is also real: FastV and other training-free attention-based token pruning methods exist, and the paper's claim to be the \"first exploration of training-free token reduction for LMMs\" is simply wrong. That's an easy fix but it needs to be done.\n\nThe contribution-degree metric in Eq. 3 is a reasonable heuristic, and the paper even acknowledges the attention-sink phenomenon, so I don't see that as a fatal flaw. It's a heuristic, evaluated empirically, and the empirical results hold up.\n\nWho's this for? People working on LMM inference efficiency. It's a solid enough empirical study that it deserves a serious referee, but the referee should push for measured latency and full hyperparameter disclosure before acceptance.","headline":"Plausible training-free token pruning with solid accuracy-at-50%-tokens data, but the headline 2x speedup is roofline-only and key hyperparameters are undisclosed, so the acceleration claim is unverified.","tokens_in":16702,"tokens_out":2852,"would_cite":true,"duration_ms":23497,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free token selection scheme that keeps only the most-attended visual tokens can double a large multimodal model's prefill speed while holding VQA accuracy.","keywords":["large multimodal models","token reduction","training-free acceleration","visual question answering","attention contribution degree","token selection","prefill speedup","post-training quantization"],"falsifier":"Compare freePruner against random token selection at the same 50% budget on OCR-oriented benchmarks such as TextVQA, and inspect which patches are kept: if randomly kept tokens match freePruner's accuracy whenever the high-attention tokens are attention sinks rather than the text region, the contribution-degree metric is not carrying the claimed semantic information. A concrete decisive test is an image set where the answer depends on a small peripheral object that receives little inbound attention; if accuracy collapses while random selection at the same budget does not, the central premise fails.","tokens_in":15784,"feed_emoji":"⚡","tokens_out":5137,"duration_ms":44768,"temperature":0.7,"pith_summary":"This paper tries to establish that a large multimodal model can be sped up by simply dropping visual tokens before the language model, with no retraining and no access to training data. It claims a two-stage, attention-based selection—pivotal tokens chosen by cross-layer contribution degree, plus complementary tokens chosen by penultimate-layer attention to those pivots—preserves both high-level semantics and low-level visual details. On LLaVA-1.5, keeping half the visual tokens roughly doubles prefill speed and keeps benchmark scores within a point or two, sometimes exceeding the unpruned model. On VideoLLaVA, the same recipe cuts tokens to a quarter and yields about a fourfold prefill speedup. If the claim holds, any open-weight multimodal model, including those whose training data is private, can be accelerated as a plug-in and combined with post-training quantization.","feed_headline":"Training-free token pruning doubles vision-LM speed","feed_subtitle":"Half the visual tokens, nearly double the prefill speed, and no retraining or accuracy loss on VQA benchmarks.","key_machinery":"The central object is the token contribution degree, $r_i^l = \\sum_j A_l[j,i] - A_l[i,i]$, computed from the self-attention map at layer $l$: it measures how much other tokens direct their attention to token $i$, excluding the token's attention to itself. This metric is used to select pivotal tokens with the highest contribution across a range of layers, and those pivotal tokens then serve as anchors for selecting complementary tokens via penultimate-layer attention patterns. The two-stage selection is what lets the method avoid token merging, so the surviving tokens remain in the original distribution and require no retraining.","core_discovery":"The central claim is that pure token selection, without merging and without training, is enough to accelerate large multimodal models: dropping half the visual tokens yields about 2x prefill speedup while keeping visual question-answering performance comparable, because the surviving tokens carry both the high-level semantics and low-level details the language model needs. freePruner defines a token contribution degree as the column sum of the attention map minus the token's self-attention, $r_i^l = \\sum_j A_l[j,i] - A_l[i,i]$, and keeps the highest-contribution tokens across layers as pivotal tokens. It then selects complementary tokens by finding, in the penultimate layer, tokens that attend strongly to the pivotal tokens, thereby recovering low-level details the pivots miss. The paper argues that prior token-reduction methods fail in the training-free setting because merging changes the token distribution that the pretrained projector and LLM expect, whereas selection leaves the distribution intact. The reported experiments on LLaVA-1.5, LLaVA-Next, and VideoLLaVA, together with the quantization combination, support the practical claim of a training-free speedup.","pith_inferences":["Editorial inference: the paper does not isolate attention sinks from semantic pivotal tokens, so a version that removes all initial and positional tokens would reveal how much of the gain comes from preserving sinks rather than content-bearing regions.","Editorial inference: since the selection is computed entirely from the visual encoder's attention maps, the same recipe likely transfers to models with different language backbones or projectors, but the paper only demonstrates LLaVA-family and VideoLLaVA variants.","Editorial inference: the reported 2x figure is a roofline estimate for prefill; end-to-end wall-clock gains will be smaller when decoding dominates, so measuring token-generation latency is a natural next test.","Editorial inference: the sparsity of contribution degrees across layers could serve as a cheap diagnostic for how much token reduction a given multimodal model tolerates before running any benchmark."],"forward_implications":["Halving visual tokens gives roughly 2x prefill speedup on LLaVA-1.5 at both 7B and 13B scale while VQAv2, ScienceQA, TextVQA, POPE, MME, and MMBench scores stay close to the unpruned model, with POPE and ScienceQA sometimes higher.","Because no training data or fine-tuning is needed, the method extends to open-weight multimodal models whose training data is private and cannot be reproduced.","On VideoLLaVA, cutting visual tokens to a quarter yields about 4x prefill speedup with comparable or better video QA scores, plus large reductions in memory access and activation storage.","freePruner is compatible with post-training quantization methods such as Q-VLM, so token pruning and quantization can be stacked in the same deployment pipeline.","Performance improves as the selected token budget grows on most benchmarks, making the pruning ratio a controllable accuracy-speed knob."],"supporting_citations":[{"why":"Supplies the main token-reduction baseline (PruMerge+) that the paper compares against in both trained and training-free settings.","marker":"[46]"},{"why":"Provides the token-merging mechanism the paper identifies as the reason prior methods need retraining, motivating pure selection.","marker":"[6]"},{"why":"Defines the ViT attention structure used to derive the token contribution degree metric.","marker":"[15]"},{"why":"The LLaVA-1.5 model family on which the main 2x acceleration and benchmark results are established.","marker":"[34]"},{"why":"The LLaVA-Next AnyRes variant used to show the method extends to high-resolution inputs.","marker":"[35]"},{"why":"The LLM-Viewer roofline analysis used to estimate prefill time, memory access, and activation storage for the efficiency claims.","marker":"[63]"},{"why":"Attention-sink finding invoked to justify keeping initial tokens that have high contribution despite low semantic content.","marker":"[60]"},{"why":"VideoLLaVA, the video model used to demonstrate generalization beyond images.","marker":"[31]"},{"why":"MME benchmark used in the main results and ablations as a measure of multimodal understanding.","marker":"[18]"}],"fun_headline_variants":["freePruner: Training-free token pruning gives 2x LMM acceleration","No training, just token pruning: freePruner doubles LMM speed","Prune tokens, not weights: freePruner achieves 2x LMM speedup","Training-free token selection: 2x speedup for LMMs","freePruner: Drop half tokens, double speed, no retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a token's total incoming attention from other tokens, after removing its self-attention, is a faithful measure of how much visual information that token contributes to the model's final answer.","fun_headline_variants_meta":{"raw":{"variants":["freePruner: Training-free token pruning gives 2x LMM acceleration","No training, just token pruning: freePruner doubles LMM speed","Prune tokens, not weights: freePruner achieves 2x LMM speedup","Training-free token selection: 2x speedup for LMMs","freePruner: Drop half tokens, double speed, no retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1480,"prompt_tokens":971,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":587,"tokens_out":509,"duration_ms":4659,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:17:18.339903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare freePruner against random token selection at the same 50% budget on OCR-oriented benchmarks such as TextVQA, and inspect which patches are kept: if randomly kept tokens match freePruner's accuracy whenever the high-attention tokens are attention sinks rather than the text region, the contribution-degree metric is not carrying the claimed semantic information. A concrete decisive test is an image set where the answer depends on a small peripheral object that receives little inbound attention; if accuracy collapses while random selection at the same budget does not, the central premise fails.","supporting_citations":[{"cited_title":"Token merging: Your ViT but faster","cited_arxiv_id":null,"evidence_quote":"Provides the token-merging mechanism the paper identifies as the reason prior methods need retraining, motivating pure selection."},{"cited_title":"Llava-next: Im- proved reasoning, ocr, and world knowledge","cited_arxiv_id":null,"evidence_quote":"The LLaVA-Next AnyRes variant used to show the method extends to high-resolution inputs."},{"cited_title":"Video-llava: Learning united visual representation by alignment before projection","cited_arxiv_id":null,"evidence_quote":"VideoLLaVA, the video model used to demonstrate generalization beyond images."}],"review_version":1}