{"id":"47ab30fb-b675-4aca-af10-c94e16233c17","arxiv_id":"2505.12359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"STAR cuts visual tokens in LVLMs in two attention-guided stages, reducing FLOPs by up to 48.6 percent at 95 percent token pruning while keeping about 98 percent of baseline accuracy.","lead":"Large vision-language models read images as hundreds of small tokens, which makes inference slow. This paper introduces STAR, a two-stage pruning method that removes unimportant image tokens before and inside the model, claiming large speedups with little accuracy loss.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5)'s second-stage score is not computable in standard inference: causal masking blocks image-to-text attention and H_resp does not exist at prefill.","rationale":"Good-faith reading: STAR aims to be a training-free, plug-and-play accelerator. The two-stage idea is plausible, but the implementation of Stage II is the load-bearing piece. The reader's H_resp concern is valid; I add that the formula also inverts the attention direction relative to the model's causal mask. These are not stylistic issues: Eq. (5) is not evaluable in the standard forward pass. If the authors intended a draft-response second pass, the efficiency table is misleading because the draft pass costs the baseline. The concrete test would settle this. Because the central claim cannot be accepted as written and requires either a corrected mechanism or a reworked efficiency evaluation, I move from CONDITIONAL to REJECT.","tokens_in":12446,"tokens_out":9769,"duration_ms":105875,"concrete_test":"Run LLaVA-1.5-7B with the Table 1 input layout and, at layer K=14, directly compute Eq. (5) under the model's causal mask; if the image-to-text entries are masked to zero, the formula is not evaluable. Then re-run the 29-token rows of Table 1 (VQAv2, GQA) using text-to-image attention averaged over question tokens only and no H_resp, and compare accuracy and end-to-end latency with the reported STAR numbers. Material differences would confirm that the published results depend on information unavailable in standard autoregressive inference.","verdict_should_be":"REJECT","load_bearing_attack":"STAR's central claim requires that the Stage II score in Eq. (5) is a valid cross-modal signal during inference. In LLaVA-1.5 and LLaVA-NeXT, the decoder is a causal self-attention transformer (Section 3.1, Eq. (2)) with input order [image tokens; text prompt]. Image tokens therefore precede all text tokens, so the causal mask forces the image-to-text block of the attention matrix to -inf before softmax; Eq. (5), which uses H_v as queries and [H_q; H_resp] as keys, is not a quantity the forward pass computes. No unmasked attention computation is described. Independently, even if the mask were removed, H_resp is the model's generated response and is unavailable during prefill. If H_resp comes from a draft or gold-answer pass, that pass costs the full baseline FLOPs, so the 28.7% FLOPs reduction and 134.68 ms latency in Table 3 cannot hold for single-pass inference. The abstract's speedup claim therefore rests on an unspecified and likely non-causal computation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STAR, a training-free, plug-and-play method to reduce visual tokens in large vision-language models in two stages: Stage I prunes tokens after the vision encoder using averaged visual self-attention scores, and Stage II prunes a second batch of tokens at an intermediate decoder layer using cross-modal attention from visual tokens to the concatenated question and response tokens. The authors report experiments on LLaVA-1.5-7B/13B and LLaVA-NeXT-7B across seven VQA benchmarks, claiming up to 95% token pruning, a 28.7% FLOPs reduction, and better accuracy preservation than FastV, FasterVLM, and SparseVLM at high pruning ratios. A FLOPs analysis in Section 4.3 and a latency comparison in Table 3 are presented to support the efficiency claims.","tokens_in":12683,"tokens_out":7797,"duration_ms":81891,"significance":"If the method worked as described, the two-stage design would be a useful contribution: it is training-free, introduces no learned parameters, and the empirical tables show STAR maintaining near-baseline scores at much higher pruning ratios than the single-stage baselines. The paper is clearly written, and the experimental breadth across model scales and benchmarks is a strength. However, the feasibility of Stage II is unresolved: Eq. (5) uses response tokens that do not exist at prefill and an image-to-text attention block that the causal mask never computes. Until this is specified and costed, the significance of the accuracy and speedup numbers is conditional, and the FLOPs accounting in Section 4.3 is inconsistent with the stated pruning schedule.","major_comments":[{"comment":"The second-stage importance score is not computable in the inference protocol claimed by the paper. In LLaVA-1.5 and LLaVA-NeXT, the decoder is a causal transformer with input order [image tokens; text prompt], so the causal mask prevents image tokens from attending to text tokens; the map C_K = Softmax(H_v H_tilde_q^T / sqrt(d)) is not produced by the standard forward pass. Moreover, H_resp does not exist during prefill. If H_resp is obtained from a draft pass or a gold-answer pass, that pass costs roughly the full baseline FLOPs, and the reported 28.7% FLOPs reduction and the latencies in Table 3 cannot describe single-pass inference. The manuscript must specify exactly how the cross-modal score is computed, where H_resp comes from, and the end-to-end cost of that computation; otherwise the central speedup and accuracy claims are not verifiable.","section":"Section 4.2, Eq. (5)"},{"comment":"The FLOPs derivation assumes that N_i = R L0_v tokens are pruned at every decoder layer i=1..K and N_i = P L0_v tokens at every layer i=K+1..Omega. This does not match the described schedule of one prune immediately after the projector and one prune at layer K. Under the actual schedule, Stage I reduces the sequence length before the decoder, so Stage II operates on (1-R)L0_v remaining tokens, and pruning at one layer affects that layer and all subsequent layers rather than being charged independently at every layer. The formula therefore overcounts the FLOPs savings, and the abstract's claim of a 28.7% reduction is not supported by the method as described.","section":"Section 4.3, Eqs. (7)-(9)"},{"comment":"The latency and throughput numbers do not support the claimed acceleration. At 288 remaining tokens, STAR reduces FLOPs by 28.7% relative to baseline but increases latency by 0.7% (134.68 ms vs. 135.58 ms); even at 29 tokens, the latency is 121.80 ms, which is slower than FasterVLM at 288 tokens (103.35 ms). The authors should report latency for the exact pipeline including the cross-modal scoring computation, and explain why the FLOPs savings do not translate to wall-clock speedups.","section":"Table 3 and Section 5.3"}],"minor_comments":[{"comment":"The threshold definition can retain fewer than the intended (1-R)L_v tokens when multiple tokens share the same score at the boundary; the tie-handling rule should be stated.","section":"Section 4.1, Eq. (3)"},{"comment":"The text lists eight benchmarks including MM-Vet, but the experimental tables report only seven; the abstract also says seven tasks, so either MM-Vet results should be added or the list corrected.","section":"Section 5.1"},{"comment":"The two panels lack axis labels and a description of the underlying evaluation setup, which makes the motivating claims about pruning ratio and pruning layer difficult to verify.","section":"Figure 2"},{"comment":"There are several naming inconsistencies, such as 'LLaVA-NeXT' versus 'LLaV A-NeXT', and the affiliation 'De Artificial Intelligence Lab' appears malformed; these should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the feasibility of Eq. (5). The authors should be asked to provide code or a precise description of how H_resp is obtained and how the cross-modal attention score is computed under causal masking. If H_resp comes from a gold-answer pass, the reported accuracy numbers would constitute a form of test-set leakage; if it comes from an extra draft pass, the FLOPs and latency numbers need to include that pass. I do not recommend sending the manuscript back for minor revision; the required changes are substantive and may invalidate the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful piece: STAR combines two known pruning signals — visual self-attention (FasterVLM) and cross-modal attention (FastV/SparseVLM) — into a two-stage schedule. That combination is new, and the empirical tables show it retaining accuracy at high pruning ratios better than any single-stage baseline across LLaVA-1.5 (7B/13B) and LLaVA-NeXT. If the method works as described, it would be a useful plug-in.\n\nBut I don't think the method as described is executable. Eq. (5) computes C_K = Softmax(H_v [H_q; H_resp]^T), i.e., image-to-text attention. In LLaVA's causal decoder, image tokens precede all text tokens, so that block is masked to -inf. The paper never says they unmask it. And H_resp, the embeddings of the generated response, are not available during prefill. The paper doesn't mention a draft pass or a second generation pass. So either the score is computed in a way that the paper doesn't explain, or the results rely on an oracle that sees the answer. Neither is acceptable for a training-free accelerator.\n\nThe FLOPs analysis in Section 4.3 matches neither the description nor the method. It assumes pruning R*L0 tokens before every layer in stage 1 and P*L0 before every layer in stage 2. The actual method prunes once at the projector and once at layer K. That means the efficiency numbers in Table 3 are overestimates; the correct reduction would be smaller, especially for the later layers. The abstract's \"prune up to 95%... reducing FLOPs by more than 28.7%\" is also a conflation: the 28.7% is for the 288-token setting, not the 95% one.\n\nOther soft spots: no code, no error bars, no ablations on K and P, and the 97.95% performance retention is not defined. The writing is clear and the related work is cited properly, but these omissions matter.\n\nMy recommendation: send it to reviewers. The idea is timely and the empirical pattern would be worth knowing if it's real. But a serious referee should ask for code, an actual implementation description that respects the causal mask (likely text-to-image attention with only H_q, no H_resp), and a corrected FLOPs calculation. Without those, the results are unverifiable. I'd want to see the authors' response before trusting any of the accuracy claims.","headline":"A promising two-stage pruning idea, but the method as written cannot be computed during inference: Eq. (5) uses masked image-to-text attention and response tokens that don't exist at prefill, and the FLOPs math assumes per-layer pruning rather than the one-time pruning described.","tokens_in":13189,"tokens_out":6862,"would_cite":false,"duration_ms":65082,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STAR claims that two-stage attention-guided token pruning removes up to 95% of visual tokens in large vision–language models while retaining at least 97.95% of baseline performance, and that this beats one-stage methods at high pruning…","keywords":["visual token pruning","large vision-language models","training-free acceleration","cross-modal attention","self-attention importance scores","token reduction","LLaVA","FLOPs reduction"],"falsifier":"Run the exact STAR pipeline on LLaVA-1.5-7B with $H_{\\mathrm{resp}}$ replaced by a first-pass draft response (no ground-truth answers), measure VQAv2, GQA, and MME accuracy and wall-clock time, and compare against the paper's tables. If accuracy drops materially or per-image latency rises by the extra generation pass, the central claim as stated fails.","tokens_in":12259,"feed_emoji":"✂️","tokens_out":7120,"duration_ms":62556,"temperature":0.7,"pith_summary":"STAR claims that visual token pruning in large vision–language models fails when it happens in one place, and that a two-stage schedule can cut most visual tokens without losing task performance. The first stage prunes conservatively using visual self-attention right after the vision encoder; the second stage prunes aggressively in an intermediate decoder layer using cross-modal attention between the remaining visual tokens and the question-and-response text. If this is right, training-free acceleration becomes practical at the high pruning ratios where existing single-stage methods degrade sharply. The paper reports up to 95% token reduction, more than 28.7% FLOPs savings, and at least 97.95% of baseline accuracy across seven vision–language benchmarks on LLaVA-1.5 and LLaVA-NeXT.","feed_headline":"Two-stage pruning cuts 95% of vision tokens, keeps 98% accuracy","feed_subtitle":"STAR prunes early by visual attention, late by question–response attention, holding near baseline at 29 tokens.","key_machinery":"The load-bearing object is the two-stage importance score. Stage 1 uses the visual encoder's self-attention matrix $A\\in\\mathbb{R}^{L_v\\times L_v}$ and scores token $i$ by $r_i=\\frac{1}{L_v}\\sum_{j}A_{ij}$; a dynamic threshold $\\tau$ keeps the top $(1-R)L_v$ tokens. Stage 2 uses the cross-modal attention map at decoder layer $K$, $C_K=\\mathrm{Softmax}(H_v\\tilde{H}_q^{\\top}/\\sqrt{d})$, and scores each visual token by its mean attention to all text positions; the $P\\cdot L_v$ lowest-scoring tokens are dropped. The two stages are complementary: the first reduces input length before the projector, the second removes task-irrelevant tokens after vision–language fusion has matured. The theoretical FLOPs account treats each pruned token as saving $6D^2$ plus $2N_iD$ per layer, giving the paper's claimed $\\Delta_{\\mathrm{total}}$ sum over stages.","core_discovery":"The paper's central discovery is that pruning visual tokens twice, with different attention signals at different depths, preserves more task-critical visual information than either signal alone. In stage one, tokens are scored by mean self-attention weight inside the vision encoder and the lowest-scoring fraction is dropped before the LLM. In stage two, at an intermediate decoder layer $K$, surviving tokens are scored by their average cross-modal attention to the concatenated query and response stream $\\tilde{H}_q=[H_q;H_{\\mathrm{resp}}]$, and the least-attended fraction is discarded. The authors report that this schedule keeps near-baseline accuracy even with 29 of 576 tokens remaining, that it consistently beats single-stage baselines at the same token budgets, and that on MME it can exceed the unpruned baseline under heavy pruning. The proposed explanation is that conservative self-attention pruning removes low-level noise while the later cross-modal stage keeps only tokens aligned with the actual question-and-answer context.","pith_inferences":["The second-stage score needs $H_{\\mathrm{resp}}$, the generated response, which does not exist when the prompt is first processed; the paper never says whether it uses a draft pass or the benchmark's answer. A testable repair is to generate a short draft with the unchanged model, then prune using that draft, and compare accuracy and latency end-to-end.","If response-aware scores are replaced by the model's own draft, STAR becomes a two-pass method, and the paper's per-image latency and throughput numbers likely undercount the real cost.","The depth $K$ and the stage-wise split may be task-dependent: fine-grained reading tasks like TextVQA might need later cross-modal pruning than holistic hallucination checks like POPE.","The success of a conservative-then-aggressive schedule suggests that pruning guidance should match the maturity of vision–language fusion at each layer, a principle that could apply to other modalities such as audio or depth tokens in future multimodal models."],"forward_implications":["At equal remaining-token budgets, STAR reports near-baseline accuracy on VQAv2, GQA, and MME where FastV, SparseVLM, and FasterVLM degrade, and at 29 tokens it can surpass the unpruned baseline on MME.","Because STAR is training-free and plug-and-play, the same procedure transfers across model scales (7B, 13B) and input resolutions (LLaVA-NeXT) without retuning.","The enforced schedule $R