{"id":"7eaa4b43-224c-4aa8-8b2b-ebedf9073459","arxiv_id":"2501.13652","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A language-guided token pruning method cuts LLaVA-1.5 inference FLOPs by 62.1% with only about 0.45% average benchmark loss.","lead":"LVPruning removes up to 90% of image tokens from LLaVA-1.5 while keeping most of its accuracy, guided by what the text prompt asks about. It inserts small trainable scoring modules without touching the original model weights, cutting inference compute by about 62%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy may depend on answer tokens visible to the cross-attention scorer only during training; no experiment separates this from prompt-only inference, so the 0.45% loss claim is not yet pinned down.","rationale":"The central claim is that LVPruning keeps accuracy while pruning aggressively. For that to hold, importance scores at layer 1 must identify which visual details the eventual answer needs. The paper never verifies this under the actual inference condition: the scorer is trained with the full answer in the sequence and deployed with only the prompt/early prefix. This is not a stylistic gap; it is a concrete mechanism by which the reported numbers could be optimistically biased. If the proposed masked retraining matches the reported accuracy, then the scorer does not actually need future tokens and the claim is robust; if it does not match, the published 0.45% loss is an artifact of a privileged training signal. Other weaknesses, including missing MLLM-pruning baselines, omitted hyperparameters, and dummy-input TFLOPs, are real but secondary and were already listed by the reader. I therefore keep the reader's CONDITIONAL verdict rather than moving it; the condition is the proposed test.","tokens_in":11958,"tokens_out":14470,"duration_ms":144997,"concrete_test":"Retrain the decision modules with the same data and hyperparameters, but mask Eq. 2 so that vision queries can only attend to prompt tokens (exclude all answer tokens from K/V) during training. Evaluate at rho=0.5 on the same nine benchmarks and compare the average score to LLaVA-1.5. If the mean drop exceeds about 1 point while the reported full-attention variant stays near -0.45, the published loss depends on future-answer tokens and the prompt-only pruning claim is not established; if the masked variant matches, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The cross-attention decision module (Eqs. 1-3) computes pruning scores with vision tokens as queries and all language tokens as keys/values. During training, the language token set includes the ground-truth answer; although causal self-attention limits each answer token's context, its layer-1 representation still contains its own input embedding, so vision queries can attend to future answer-token identities. This is privileged information: the score is optimized with access to tokens that do not exist at the first autoregressive decoding step. At inference, under the natural KV-cache implementation, tokens pruned at layer 1 are physically removed and cannot be restored by later decoding steps; if instead every step re-encodes all vision tokens, the single-forward-pass TFLOPs in Figure 4 and the 62.1% reduction do not describe full generation cost. Section 3.3 specifies token removal but never states which protocol produced Tables 1-2, and Section 7 does not mention this gap. Since the 0.45% average-loss claim is precisely about what survives early, prompt-only pruning, this is the load-bearing assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LVPruning, a method to prune vision tokens in MLLMs by inserting lightweight cross-attention decision modules into specific layers of LLaVA-1.5. The modules use vision tokens as queries and language tokens as keys/values to predict keep/remove scores; during training the decisions are applied via attention masks with Gumbel-Softmax and an MSE ratio loss, while at inference the scores are sorted and the top-k tokens are physically removed. The authors report a 62.1% reduction in inference TFLOPs at a 50% token retention ratio with roughly 0.45% average performance loss across nine multimodal benchmarks, and compare favorably against Q-former–based models.","tokens_in":12210,"tokens_out":6694,"duration_ms":56309,"significance":"If the central claims hold, the method is attractive: it requires no modification of the base MLLM parameters, trains only the small inserted modules, and achieves a large compute reduction with modest accuracy change. The language-guided pruning mechanism is a sensible extension of DynamicViT to multimodal settings and could be useful for deployment. However, the load-bearing claims rest on experimental details that are currently under-specified, especially the inference protocol and the exact averaging used for the headline 0.45% loss figure.","major_comments":[{"comment":"The paper never states which language tokens are available to the cross-attention decision modules at inference time. In training, the language-token set includes the ground-truth answer tokens (Eqs. 1–3), so the scorer can attend to future-answer identities. At inference, when the KV cache is used, only prompt tokens are present at the first pruning layer and only prompt plus already generated tokens at later layers. Since the method's rationale is that language context identifies useful vision tokens, the reported 0.45% loss cannot be interpreted without knowing whether the inference protocol matches this setup. The authors should specify the exact generation protocol and, ideally, provide an ablation training the decision modules with only prompt tokens (or comparing prompt-only vs. answer-token training) to verify that the method does not rely on privileged information.","section":"§3.3, §7, Tables 1–2"},{"comment":"The claimed '0.45% average performance loss' at a 50% token retention ratio is not directly supported by the reported per-benchmark numbers. Computing the average signed delta over the twelve reported columns gives approximately −0.3 points, while averaging per-benchmark (collapsing the POPE and MMBench splits) gives approximately −0.3 to −0.9 points, and the relative percentage changes average to roughly +0.3%. None of these straightforward aggregations yields 0.45%. The authors must define precisely how the average is computed (e.g., mean relative drop over the nine benchmarks, or mean absolute drop) and add an average row to the results tables.","section":"Abstract, Tables 1–2, §6"},{"comment":"Section 3.2 contains a leftover description of a Huber loss with threshold β: the sentence 'where δ(Dls, ρs) is the Huber loss, β is a threshold that determines the loss function used. We set β = 0.5' appears immediately after an equation that is a plain MSE loss with no δ or β. In addition, the loss weights λ_causal and λ_ratio in Eq. (12) are never reported, even though they control the balance between task performance and the pruning-ratio constraint. These values are necessary for reproducibility and for understanding whether the ratio loss dominates the objective.","section":"§3.2, Eqs. (11)–(12)"},{"comment":"All TFLOPs are computed with a single dummy input of 1 image and 30 text tokens. This is not representative of the varying prompt lengths in the nine benchmarks, and the text states that the inserted decision modules add 0.71 TFLOPs without clarifying whether this overhead is included in the reported 3.18 TFLOPs for LVPruning at ρ=0.5. The authors should describe the FLOPs methodology in more detail, including whether a KV-cache is assumed and whether the pruning decisions are applied once during prefill or at every decoding step.","section":"§4.1, Figure 4, Appendix B"}],"minor_comments":[{"comment":"Equation (13) uses ρs for the inference kept ratio, but the paper defines inference ratios as \\\\hat P = [\\\\hat ρ_1, ..., \\\\hat ρ_S]. The notation should be made consistent.","section":"§3.3, Eq. (13)"},{"comment":"The rendered captions show 'LVPruning ( =0.5)' with a missing ρ symbol; this appears to be a LaTeX rendering issue and should be fixed.","section":"Figure 3 and Figure 4 captions"},{"comment":"The asterisk on LLaVA-1.5's SQA-IMG score (69.4*) is never explained in the text or caption.","section":"Table 1"},{"comment":"The IDEFICS model is cited to the OBELICS paper (Laurençon et al., 2023), but the model itself is described in a separate technical report; the citation should be corrected or supplemented.","section":"§2, References"},{"comment":"The paper states it removes all training entries without image inputs, but does not report the exact number of remaining samples; the approximate 620k number should be made precise, or a reference to the dataset version should be given.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the empirical results, if verified under a clean protocol, would be publishable. However, the train/inference language-token mismatch is a substantial correctness concern that is not addressed anywhere in the manuscript, and the headline average-loss number does not match the tables. Both are fixable, but they require additional experiments or careful clarification, so a major revision is appropriate. I would also ask the editor to verify that the authors can release the code, because the ambiguity in the inference protocol makes the current description hard to reproduce."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LVPruning is a clean, simple method for cutting vision tokens in LLaVA-style models, and the empirical story is plausible. But the headline numbers are not fully backed by the tables, the FLOPs accounting rests on a dummy input and a single forward pass, and the paper omits the obvious MLLM pruning baselines. The core idea deserves a referee, but the claims need tightening.\n\nWhat's new: the cross-attention scoring where vision tokens query language tokens is a sensible extension of DynamicViT to multimodal inputs, and freezing the base model while training only the inserted modules makes it easy to apply or remove. The adjustable inference ratios without retraining are a nice practical feature. The benchmark coverage is broad, and the results at ρ=0.5 mostly show small deltas, even some gains on VizWiz and LLaVA-Wild.\n\nSoft spots, in rough order of weight. First, the 0.45% average loss in the abstract and conclusion does not match any obvious aggregation of Tables 1-2. For ρ=0.5, the mean delta across the twelve reported metrics is about -0.3, and the mean absolute delta is about 0.68. For ρ=0.6, the deltas are mostly positive. The paper never states which configuration and which averaging produced 0.45%. That needs to be pinned down.\n\nSecond, the FLOPs numbers. Everything is computed on a dummy input of one image and 30 text tokens, with no generation length. During autoregressive decoding, the vision tokens appear only in the prompt, so the 62.1% reduction presumably applies to the prompt-processing phase, not the whole response. The paper should say so.\n\nThird, and this is the load-bearing assumption: the cross-attention scorer is trained with the ground-truth answer tokens in the language sequence. There is no causal mask inside the decision module, so vision queries can attend to answer-token identities that do not exist at inference. No experiment separates this privileged training signal from prompt-only scoring, so the claim that pruning based on the prompt alone preserves accuracy is not yet established. The paper does not mention this mismatch anywhere.\n\nFourth, the missing baselines. FastV and PyramidDrop already prune vision tokens in MLLMs; the paper cites only single-modal pruning. A comparison is needed for the novelty claim to hold.\n\nMinor: Eq. 11 defines an MSE loss but the surrounding text describes a Huber loss with β=0.5, which is a leftover edit. The Section 7 limitations are generic and do not address any of the above.\n\nWho this is for: people working on MLLM inference efficiency. It's a useful data point, but not a fundamental advance. Deserves a serious referee; I would send it out but require a major revision that addresses the scoring mismatch and the baseline comparisons.","headline":"A simple, plausible method for language-guided vision token pruning in MLLMs, but the headline accuracy and FLOPs claims are under-supported and the paper skips the obvious MLLM pruning baselines.","tokens_in":12731,"tokens_out":4239,"would_cite":true,"duration_ms":33017,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight language-guided pruning module can remove up to 90% of vision tokens from a multimodal LLM while keeping average benchmark loss near 0.45%.","keywords":["vision token pruning","multimodal large language models","efficient inference","cross-attention","LLaVA-1.5","token sparsification","instruction tuning","Gumbel-Softmax"],"falsifier":"Compare LVPruning at a kept ratio of 0.5 against random pruning of vision tokens at the same ratio on the nine benchmarks; if random pruning matches LVPruning's accuracy, the language-guided scoring is not what preserves performance. A complementary test: use a task whose correct answer depends on a tiny image region and see whether early pruning discards that region and causes a far larger loss than the reported ~0.45% average.","tokens_in":11786,"feed_emoji":"✂️","tokens_out":10678,"duration_ms":79634,"temperature":0.7,"pith_summary":"This paper introduces LVPruning, a method for cutting the inference cost of multimodal large language models by pruning vision tokens early and progressively. The central claim is that language tokens—the prompt and partially generated text—are sufficient guides for deciding which image patches matter, so that up to 90% of vision tokens can be dropped by the middle of LLaVA-1.5 without retraining the base model. The reported result is a 62.1% reduction in inference TFLOPs (8.38 to 3.18) with an average performance loss of roughly 0.45% across nine benchmarks. This matters because it suggests that a large share of visual computation in MLLMs is redundant for the task, and that efficiency gains do not have to come at the cost of accuracy or from building a new architecture.","feed_headline":"Drop 90% of image tokens, cut multimodal-model compute by 62%","feed_subtitle":"Language-guided pruning holds average benchmark loss to about 0.45%, so the model stays accurate at a third of the cost.","key_machinery":"The load-bearing component is the cross-attention decision module: a lightweight transformer block inserted at selected LLM layers where vision tokens act as queries and text tokens as keys and values. Its attention output goes through an FFN and a linear layer that scores each vision token as keep or remove. During training, Gumbel-Softmax turns the scores into differentiable binary decisions, an attention mask zeroes out pruned tokens without changing the token count, and an MSE loss drives the kept ratio to preset values per layer. At inference, scores are sorted and the top-k vision tokens are kept, with original positional embeddings preserved so the model sees the same distribution it saw during training.","core_discovery":"The paper's central claim is that vision tokens can be scored by how much they matter to the language context through small cross-attention modules, and that pruning the lowest-scoring tokens at a few fixed layers preserves accuracy almost completely. On LLaVA-1.5-7B, three such modules are inserted after layers 1, 8, and 16; by the middle layer as many as 90% of vision tokens are gone. Inference cost drops from 8.38 to 3.18 TFLOPs (62.1% reduction) while the average score across VQAv2, GQA, VizWiz, SciQA-IMG, TextVQA, POPE, MMBench, LLaVA-Wild, and MM-Vet falls by about 0.45% relative to the unpruned model. The authors further show that the retention ratio can be adjusted at inference time without retraining, and that the pruned model often beats Q-former-based baselines that already use many fewer tokens.","pith_inferences":["If the language-guided scoring is as effective as the paper reports, a large fraction of visual tokens in LLaVA-1.5 is redundant for text-conditioned tasks; a natural but untested extension would be to apply the same scoring to higher-resolution images or video frames, where redundancy is likely even larger.","The method relies on the text available at early layers being sufficient to know which visual details the entire response will need; one test not reported is whether the pruned tokens differ when the same image is paired with different prompts, or when longer generation changes what later text requires.","A comparison against random pruning at the same kept ratios would isolate whether the language guidance, rather than mere token removal, is what preserves accuracy; the paper does not include that baseline."],"forward_implications":["Deploying a frozen MLLM becomes much cheaper: at the main setting, inference drops from 8.38 to 3.18 TFLOPs, making resource-constrained environments far more feasible.","Because the kept ratio is an inference-time knob, one model can trade a few accuracy points for large speedups on demand, with no retraining.","The base model parameters are never modified, so LVPruning can be added to an existing MLLM and later removed cleanly.","Most vision tokens disappear before the middle of the network, so the compute savings apply across the majority of layers, not just near the output."],"supporting_citations":[{"why":"Supplies the token-pruning decision mechanism (Gumbel-Softmax, attention masking, ratio loss) that LVPruning adapts for the multimodal setting.","marker":"Rao et al., 2021"},{"why":"Defines LLaVA-1.5, the frozen base MLLM and vision-instruction dataset used for all training and evaluation.","marker":"Liu et al., 2023a"},{"why":"Provides the cross-attention formulation used inside the decision modules to compute vision-token importance from language tokens.","marker":"Vaswani et al., 2017"},{"why":"VQAv2 is one of the nine benchmarks carrying the claimed ~0.45% average performance loss.","marker":"Goyal et al., 2017"},{"why":"GQA is the benchmark used for the efficiency-performance trade-off plot (Figure 5) and for comparisons with Q-former models.","marker":"Hudson and Manning, 2019"},{"why":"BLIP-2 is the Q-former-based baseline that LVPruning outperforms or matches at similar or lower compute.","marker":"Li et al., 2023a"}],"fun_headline_variants":["Prune 90% of vision tokens, cut LLM compute 62%, lose 0.45% accuracy","Language-guided token pruning slashes multimodal compute by 62% with 0.45% loss","Cut vision tokens 90%: multimodal LLM keeps accuracy, drops TFLOPs 62%","Simple cross-attention prunes image tokens, saving 62% compute with minimal loss","Prune vision tokens by language score: 90% cut, 62% fewer TFLOPs, 0.45% loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the language tokens available at each pruning layer—the prompt and any generated tokens so far—are sufficient to identify every vision token the final answer will need, even when a small visual detail only becomes relevant deep into generation.","fun_headline_variants_meta":{"raw":{"variants":["Prune 90% of vision tokens, cut LLM compute 62%, lose 0.45% accuracy","Language-guided token pruning slashes multimodal compute by 62% with 0.45% loss","Cut vision tokens 90%: multimodal LLM keeps accuracy, drops TFLOPs 62%","Simple cross-attention prunes image tokens, saving 62% compute with minimal loss","Prune vision tokens by language score: 90% cut, 62% fewer TFLOPs, 0.45% loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3675,"prompt_tokens":952,"completion_tokens":2723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2589}},"tokens_in":568,"tokens_out":2723,"duration_ms":15137,"temperature":1.0,"reasoning_tokens":2589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:43:29.669631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare LVPruning at a kept ratio of 0.5 against random pruning of vision tokens at the same ratio on the nine benchmarks; if random pruning matches LVPruning's accuracy, the language-guided scoring is not what preserves performance. A complementary test: use a task whose correct answer depends on a tiny image region and see whether early pruning discards that region and causes a far larger loss than the reported ~0.45% average.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the token-pruning decision mechanism (Gumbel-Softmax, attention masking, ratio loss) that LVPruning adapts for the multimodal setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VQAv2 is one of the nine benchmarks carrying the claimed ~0.45% average performance loss."}],"review_version":1}