{"id":"408ce644-6c76-4021-b938-40929476f9a0","arxiv_id":"2507.08806","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free test-time method prunes low-attention reasoning tokens from the KV cache, guided by an injected end-of-thinking token, and reports accuracy gains on math competition benchmarks.","lead":"This paper proposes pruning redundant tokens from a reasoning model's KV cache at test time, using attention from an injected 'end of thinking' token to decide what to remove. The method reports accuracy gains on math competition benchmarks such as AIME and AMC while using less memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sign of the pruning gain vs. token-eviction effect is confounded by the injected summarization prompt; the prompt-only baseline in Table 3 shows a 7.5-point AMC gain without real pruning.","rationale":"I agree with the reader that attention-as-importance (Eq. 1) is a core premise, but the more decisive, currently underaddressed confound is visible inside the paper's own ablation: Table 3 shows that injecting the summarization prompt alone (Summ=✓, Step=✗) raises AMC2023 from 70.0 to 77.5 while evicting zero tokens—almost the same magnitude as the paper's marquee AMC improvement. The main Table 1 comparison (FullKV vs. Ours) has no such no-eviction control for the final configuration, so the central causal claim is not yet isolated. This is a testable experimental confound rather than a theoretical objection to Eq. (1), and it should be resolved before the claim 'removing redundancy improves accuracy' is accepted. The paper is otherwise clearly written, the method is reproducible in principle, and the efficiency results in Table 2 are consistent with a pruning method that also benefits from prompt-induced summarization. My recommended verdict remains CONDITIONAL, consistent with the reader's, because the requested control experiments and multi-seed results are the natural acceptance conditions. I do not think the evidence supports REJECT: Tables 3 and 4 show that deliberate step-aware pruning adds a real increment over both random and attention-accumulation baselines within the same framework, and GPQA (Table 5) gives an out-of-domain positive signal. The concern is about attribution of the headline magnitude, not about fabrication or an obviously broken method. The reader's weakest_assumption (attention as importance proxy) is plausible and supported by Figure 1 and by the Table 4 comparison against H2O/Random, so I mark agreement as partial: the reader identified a real risk, but the prompt-only confound is more directly load-bearing for the stated central claim because the paper's own Table 3 already suggests large gains before any pruning occurs.","tokens_in":12884,"tokens_out":2109,"duration_ms":18634,"concrete_test":"Run the full Table 1 protocol (Qwen2.5-7B, six benchmarks, same seed/temperature) under four conditions: (1) FullKV with no injected prompts; (2) FullKV with the periodic summarization prompt injected every p=200 tokens but with eviction budget k=0 (i.e., no tokens actually removed); (3) the paper's full method; (4) the paper's full method with the summarization prompt removed from the computation of s_t (e.g., using attention from a normally generated </think>). If condition (2) alone reproduces most of the FullKV→Ours accuracy gain, the paper's pruning mechanism is not the cause of the reported improvement. Additionally, rerun AMC2023 and AIME2024/25 with 3–5 seeds and report per-seed accuracy and variance to confirm the 75.0→82.5 and 23.3→36.7 gaps exceed seed noise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim is that evicting redundant reasoning tokens improves accuracy (Qwen2.5-7B AMC2023: 75.0→82.5, average 57.9→63.4). However, the ablation in Table 3 contains a prompt-only condition that already moves AMC2023 from 70.0 (baseline) to 77.5 with self-summarization but without step-aware eviction, i.e., a 7.5-point gain relative to the shown baseline. The final method adds only a further 5.0 points (77.5→82.5) on AMC and 10.0 points (36.7→46.7) on AIME2024. A separate 'Summarization only, no eviction' baseline is never reported for the main Table 1 configuration, so the central comparison cannot separate (a) the effect of injecting periodic 'time is up' summarization prompts (which changes the sampled distribution, may trigger self-correction, and effectively increases test-time compute) from (b) the effect of KV pruning itself. Table 4 partially addresses this by comparing Random/H2O eviction under the framework, but it does not include the no-eviction-with-summarization condition. If the full gain survives the prompt-only control, the pruning mechanism is doing causal work; if not, the central claim reduces to a known 'summarization improves CoT' effect and the KV-pruning story is unsupported. The single-seed evaluation on small AIME/AMC sets (30/40 items) further amplifies this: a 2-question shift on AIME2024 is 6.7 points.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-free, test-time KV-cache pruning method for reasoning LLMs. The method periodically injects a 'Time is up...' summarization prompt ending with </think>, uses the attention from that </think> token to earlier tokens as a per-token importance score (Eq. 1), segments the reasoning trace into steps by trigger words, and evicts tokens from low-scoring steps under a fixed budget (Eqs. 2-3). The main experiments compare this method against full-KV decoding on six math benchmarks with Qwen2.5-7B, Llama3.1-8B, and Qwen2.5-1.5B, plus GPQA Diamond, reporting average accuracy gains (e.g., Qwen2.5-7B average 57.9 to 63.4, Table 1) and reduced average KV cache length in some settings. The paper also reports an efficiency comparison on MATH-500 (Table 2), ablations on AIME2024 and AMC2023 (Tables 3-4), and a small-model result in the appendix (Table 6).","tokens_in":13210,"tokens_out":6875,"duration_ms":69586,"significance":"If the causal claim is established, the method would be a noteworthy plug-and-play inference-time intervention: it improves accuracy without training, is model-agnostic across three R1-distill backbones, and shows consistent average gains on six math benchmarks plus GPQA. The paper also includes a useful comparison against H2O and random eviction, a clear algorithmic recipe, and an explicit limitation section. However, the central attribution of the gains to token pruning is not yet supported because the main comparison conflates pruning with prompt-injected summarization, the baselines in the ablation are inconsistent with Table 1, and the competition benchmarks are small single-seed evaluations. These issues need to be resolved before the headline claim is convincing.","major_comments":[{"comment":"The ablation in Table 3 shows that self-summarization alone, without step-aware eviction, improves AMC2023 from 70.0 to 77.5, that is, 7.5 points of the total 12.5-point gain over that baseline is obtained with no eviction at all. The main Table 1 comparison ('Ours vs FullKV') does not include a 'summarization only, no eviction' condition for the six-benchmark suite, so the average gain from 57.9 to 63.4 cannot be attributed to the pruning mechanism; prompt injection itself changes the sampled distribution and may act as a self-correction or test-time-compute intervention. Please report the no-eviction-with-summarization control on all six benchmarks and quantify the marginal effect of eviction beyond the prompt effect.","section":"Section 5.4, Tables 1 and 3"},{"comment":"For Qwen2.5-7B, Table 1 reports FullKV accuracy of 36.7 on AIME2024 and 75.0 on AMC2023, while Table 3's 'no summarization, no step' rows report 40.0 and 70.0 for the same datasets and model. Both tables state the same decoding setting (temperature 0.6, fixed seed), so this is an unexplained contradiction. It changes the decomposition: from the Table 1 baseline, the AMC gain of the full method is 7.5 points, not 12.5. Please reconcile the numbers or state the exact evaluation subset, seed, or sampling used in each table.","section":"Tables 1 and 3"},{"comment":"AIME2024 and AIME2025 contain 30 questions each, and AMC2023 contains 40; the evaluation uses a single fixed seed with no error bars or confidence intervals. A two-question change on AIME is 6.7 points, and several headline differences are of that magnitude (for example, Qwen2.5-7B AIME2024 improves from 36.7 to 46.7, a three-question shift). The claims of 'significant' and 'consistent' gains therefore require bootstrap confidence intervals, multiple seeds, or a larger evaluation set before they are statistically credible.","section":"Section 5.1 and Tables 1, 3, 4"},{"comment":"The importance score is defined solely by the attention from the injected </think> token to each previous token. The paper provides only qualitative attention maps (Fig. 1) and no quantitative evidence that low-attention tokens are causally non-load-bearing. Since the summarization prompt alone changes accuracy (Table 3), the extra gain of Ours over H2O (Table 4) is suggestive but does not establish that the </think>-attention ranking is measuring redundancy rather than merely being a byproduct of the prompt. Please add a validation where the importance score is compared with an oracle measure (for example, leave-one-chunk-out accuracy) or where pruning by the proposed score is compared with pruning by other scores under identical prompt injection.","section":"Section 4.1, Eq. (1)"},{"comment":"The memory-reduction claim is not universally supported. Table 6 shows that on Qwen2.5-1.5B, Ours uses a larger average KV cache than FullKV on MATH-500 (6166 vs 6120) and GaoKao (6071 vs 5825), and Table 1 shows that on Llama3.1-8B, Ours exceeds FullKV on AIME2024 (7210 vs 7067) and AIME2025 (7375 vs 7088). Because the injected summarization prompt adds tokens, pruning does not always yield a net memory reduction. Please qualify the abstract and Section 5.3 claims by reporting net KV length and the number of tokens added by the summarization prompt for every benchmark.","section":"Tables 1, 6 and Section 5.3"}],"minor_comments":[{"comment":"The caption contains a typo: 'Avgerage' should be 'Average'.","section":"Table 6 caption"},{"comment":"In the related-work paragraph on token pruning, 'eviting redundant reasoning tokens' should be 'evicting redundant reasoning tokens'.","section":"Section 2"},{"comment":"The algorithm formatting is unclear at line 12 ('of c'), and the sorting and budget-allocation steps could be stated more explicitly.","section":"Algorithm 1"},{"comment":"The summation upper limit in Eq. (3) is missing; it should be the sum over j from 1 to i-1 with explicit indices for the eviction budgets.","section":"Eq. (3)"},{"comment":"The paper cites Hammoud et al. 2025a and 2025b with identical titles ('Beyond the last answer: Your reasoning trace uncovers more than you think'); please disambiguate or merge these references.","section":"References"},{"comment":"The evaluation protocol says 'fixed seed' but does not report the actual seed value; please include it for reproducibility.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a CS/AI venue and the core idea is interesting. The main concern is that the central comparison conflates the pruning mechanism with the injected summarization prompt, and the Table 1 vs Table 3 baseline inconsistency calls for a careful check of the experimental pipeline. I do not see grounds for rejection, as the method may well work, but the evidence needs to be clean: a no-eviction-with-summarization control on the full benchmark suite, consistent baselines, and uncertainty estimates on the small competition sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the injected </think>-attention scoring plus step-aware eviction is a new combination, and the gains are consistent enough across models and benchmarks that I don't think this is noise. But the paper never cleanly separates \"the prompt helps\" from \"pruning helps,\" and several key numbers rest on 30-question sets at one seed. I'd send it out, with the expectation of revision.\n\nWhat's actually new: using a forced summarization prompt with an end-of-thinking token to score token importance by attention, then allocating the eviction budget hierarchically over reasoning steps, is not in H2O/StreamingLLM/PyramidInfer. The analysis in Fig 1 is suggestive, and the method is described clearly enough to reimplement. Table 1 shows average gains on all three model sizes; the GPQA result is a nice breadth check. The memory numbers in Table 2 are also reasonable.\n\nSoft spots, in order:\n\n1. The main comparison inherits the injected prompt. Table 3's prompt-only row on AMC2023 moves 70.0 to 77.5 before any eviction; the full method adds 5 more points. On AIME2024, prompt-only is if anything harmful (40.0→36.7) while full method reaches 46.7, which argues pruning does real work there. But Table 3 is only two datasets, and Table 1 has no prompt-only control. The central claim \"eviction improves accuracy\" needs that control in the main benchmark table.\n\n2. AIME2024/2025 and AMC2023 are small sets, single fixed seed, no error bars. A two-question shift on AIME2024 is 6.7 points. The paper's biggest headline numbers are exactly the ones with the least statistical weight.\n\n3. The eviction budget k is never stated in the main text or appendix. The algorithm depends on it; the reader cannot reproduce the exact runs without guessing. That is a fixable but real reporting gap.\n\n4. The importance score is a self-referential heuristic. That is not a flaw by itself, but it means the mechanism story rests on attention-as-proxy, and the paper would be stronger with a sensitivity check (e.g., does the score correlate with answer correctness on a per-token basis?).\n\nThe citation pattern looks fine; related work is standard. No sign of fitting-to-benchmarks beyond the usual. The limitations section is honest.\n\nBottom line: this is a legitimate inference-time intervention worth taking seriously. Send it to review, but the revision must add the prompt-only baseline in the main table, multi-seed runs with variance on the small sets, and the eviction budget values.","headline":"A plausible and genuinely new KV-pruning trick that likely does something, but the headline causal claim is undercut by a prompt-only confound and thin evaluation.","tokens_in":13795,"tokens_out":2648,"would_cite":false,"duration_ms":26213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruning low-attention tokens from a reasoning model's own output improves accuracy without retraining.","keywords":["token pruning","KV cache eviction","chain-of-thought reasoning","test-time inference","large language models","mathematical reasoning","attention scoring","redundancy removal"],"falsifier":"Run the same eviction policy but score each token by a causal intervention, such as the change in final-answer probability when the token's embedding is zeroed; if that causal ordering disagrees with the </think>-attention ordering, or if pruning the causal low-importance tokens does not reproduce the reported gains, the attention-proxy premise fails.","tokens_in":12680,"feed_emoji":"🧹","tokens_out":9335,"duration_ms":84840,"temperature":0.7,"pith_summary":"The paper claims that long reasoning traces produced by reasoning models contain substantial redundancy, and that deleting the least useful tokens on the fly improves accuracy and saves memory with no retraining. Redundancy is detected by periodically injecting a short 'time is up, summarize' prompt that ends in the model's end-of-thinking token, then reading how much attention that token pays to each earlier token. Low-attention tokens, especially those inside low-scoring reasoning chunks, are evicted from the key-value cache, the stored memory of tokens generated so far, and generation resumes without the injected instruction. If the claim holds, token eviction acts as a test-time intervention that removes distraction, with the largest gains on competition math where reasoning traces are most redundant.","feed_headline":"Pruning redundant thoughts boosts math scores at test time","feed_subtitle":"Deleting low-attention tokens from the model's own reasoning cache raised AMC2023 accuracy from 75.0% to 82.5%.","key_machinery":"The load-bearing object is the token importance score $s_t^{(\\ell,h)} = \\alpha^{(\\ell,h)}_{\\texttt{</think>}\\rightarrow t}$ from Eq. (1): the attention weight that an injected end-of-thinking token pays to an earlier token $t$, read separately at each layer and head. The second component is a hierarchical eviction policy: the reasoning trace is segmented into coherent steps using surface markers such as 'Wait' or 'Alternatively', each step is scored by the mean importance of its tokens, and the eviction budget is spent first on the lowest-scoring steps and then on the lowest-scoring tokens inside those steps. This makes pruning structure-aware rather than uniform.","core_discovery":"The central discovery is that a reasoning model's attention can be used against its own redundancy: when the model is prompted to close a reasoning step with its end-of-thinking token, that token's attention to earlier tokens acts as an importance score, and evicting the lowest-scored tokens from the key-value cache improves final accuracy. On Qwen2.5-7B distilled from DeepSeek-R1, accuracy averaged over six math benchmarks rises from 57.9% to 63.4%, with AMC2023 rising from 75.0% to 82.5% and AIME2024 from 36.7% to 46.7%, while the cached token count falls. On GPQA Diamond, the same procedure raises accuracy from 32.0% to 36.4%. The authors interpret this as removing distraction: incorrect answers show more scattered attention and more abandoned reasoning chunks, so pruning re-focuses the model rather than merely compressing memory.","pith_inferences":["Editorial inference: the injected </think>-attention score could serve as an online confidence signal, since low average scores coincide with scattered attention and wrong answers; a testable extension is to stop or restart generation when the score drops below a threshold.","Editorial inference: because the method is test-time only, it composes with sampling-based decoding, so pruning each sampled trace before majority voting could cut the cost of self-consistency while possibly improving its reliability.","Editorial inference: the forced-summarization prompt and segmentation marker list are hand-tuned; automatically searching them per model family is a natural extension the paper leaves open."],"forward_implications":["Average math accuracy rises from 57.9% to 63.4% on Qwen2.5-7B and from 52.6% to 55.9% on Llama3.1-8B across the six tested benchmarks, with no training.","The accuracy gains are largest where reasoning is most redundant: AMC2023 goes from 75.0% to 82.5% and AIME2024 from 36.7% to 46.7% on Qwen2.5-7B.","The pruning also saves memory, e.g., a 10.3% reduction in KV cache tokens on DeepSeek-R1-Distill-Qwen-7B while accuracy improves.","At 50% cache budget the method retains over 94% of full-cache accuracy and beats StreamingLLM, H2O, and Pyramid-Infer at the same compression ratios.","The effect extends past math: GPQA Diamond accuracy rises from 32.0% to 36.4%."],"supporting_citations":[{"why":"Supplies the motivating premise that LLMs get distracted by irrelevant context, which the pruning method is designed to counteract.","marker":"Shi et al., 2023"},{"why":"Supplies the reasoning-step segmentation method (surface-marker splitting) that the step-aware eviction policy relies on.","marker":"Hammoud et al., 2025b"},{"why":"Provides the H2O accumulated-attention eviction baseline and the attention-as-importance idea that this paper extends with a summarization trigger.","marker":"Zhang et al., 2023"},{"why":"Provides the StreamingLLM baseline that keeps only initial and recent tokens, a key memory-efficiency comparison.","marker":"Xiao et al., 2024"},{"why":"Defines the DeepSeek-R1 models whose distilled checkpoints are the main evaluation testbed.","marker":"Guo et al., 2025"},{"why":"Supplies the MATH-500 benchmark used in the main effectiveness and efficiency tables.","marker":"Hendrycks et al., 2021"},{"why":"Documents the Qwen2.5 backbone models used for the distilled reasoning models.","marker":"Yang et al., 2024b"},{"why":"Supplies the AMC2023 competition benchmark where the largest accuracy gain is reported.","marker":"AI-MO, 2023"}],"fun_headline_variants":["Cutting redundant reasoning tokens boosts math accuracy","Prune token cache to sharpen model reasoning","Attention-based token eviction improves math scores","Removing distraction from thoughts lifts reasoning","Cache pruning focuses model thinking, no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the attention an injected end-of-thinking token pays to an earlier token measures how much that token contributes to the final answer; if attention is a poor proxy for causal contribution, the pruning targets are wrong and the accuracy gains could reverse.","fun_headline_variants_meta":{"raw":{"variants":["Cutting redundant reasoning tokens boosts math accuracy","Prune token cache to sharpen model reasoning","Attention-based token eviction improves math scores","Removing distraction from thoughts lifts reasoning","Cache pruning focuses model thinking, no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1229,"prompt_tokens":932,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":232}},"tokens_in":548,"tokens_out":297,"duration_ms":3813,"temperature":1.0,"reasoning_tokens":232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:33.599785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eviction policy but score each token by a causal intervention, such as the change in final-answer probability when the token's embedding is zeroed; if that causal ordering disagrees with the </think>-attention ordering, or if pruning the causal low-importance tokens does not reproduce the reported gains, the attention-proxy premise fails.","supporting_citations":[],"review_version":1}