{"id":"c80f89f8-0b26-49b9-a802-763cb57ad70a","arxiv_id":"2607.05061","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"KVpop supervises fixed-budget KV eviction with an efficiently computed future-attention target and delayed mLSTM scoring, retaining ~97–100% of full-attention math performance at 75–88% compression.","lead":"KVpop learns which past tokens an LLM should keep in a fixed-size key-value cache by training scorers on future attention mass, optionally delaying the decision with a memory model. On Qwen3 math-reasoning benchmarks it keeps nearly full quality at 75–88% cache compression and beats prior eviction baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged proxy and evaluation caveats.","rationale":"The reader's CONDITIONAL verdict already correctly identifies the future-attention proxy (sparse LSE, domain-matched math train/eval, missing uncertainty, abstract/body drift) as the main soft spot without overstating it into a rejection. Stress-testing does not surface a stronger load-bearing flaw: the method is a coherent retrofit (fixed per-head budget, boundary pairwise loss, training-only transposed target, optional delayed stateful scorer), baselines are matched on budget and scorer capacity, and multiple ablations/OOD checks support that the learned ranking—not merely sparse training—drives the gains. Homogeneous caches also give a plausible efficiency edge over DMS's ragged gates (Fig. 4, Fig. 8). Therefore no verdict adjustment is warranted; the existing CONDITIONAL (ship code, report error bars, broaden eval) remains the right posture.","tokens_in":23464,"tokens_out":600,"duration_ms":5284,"concrete_test":"Re-run the Qwen3-4B CR=88% AIME/HMMT evaluation of KVpop vs DMS and StreamingLLM+ with 16 rollouts, report mean±std over seeds, and recompute relative retention using the body numbers (Table 1 Abs/Rel). If KVpop remains ≥ DMS by a clear margin and ≥90% of teacher after uncertainty, the claim stands; if the gap collapses inside error bars, the proxy/eval concern lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that boundary-supervised future-attention ranking (Eqs. 2–3, 5–7), optionally with delayed mLSTM scoring, yields near-teacher AIME/HMMT accuracy under fixed budgets B=s+w+k. The reader's weakest assumption—that sparse-LSE future mass (Eqs. 8–9, App. D.3) is a faithful enough proxy for generation utility—is the right soft spot, but the paper already supplies independent support that keeps it from being load-bearing against the claim as stated: (i) StreamingLLM+ isolates pattern adaptation from learned ranking and still trails KVpop; (ii) DMS under matched budget/params trails by 8–16 relative points, so the supervision target matters; (iii) delayed vs non-delayed mLSTM improves token accuracy (Fig. 5); (iv) OOD GPQA-D/LCB stay near teacher (Table 2); (v) top-k teacher recall averages 81% (Fig. 7). Abstract/body relative-score drift (98%/97% vs 95%/94%) is present but does not reverse the ordering. No internal inconsistency or hidden assumption that would overturn the matched-budget superiority is evident from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes KVpop, a post-training retrofit for fixed-budget KV-cache eviction in pretrained Transformers. Each head keeps sink tokens, a protected recent window, and a learned long-range top-k set. Scorers are supervised at the eviction boundary by a future-attention mass target (Eqs. 2–3, 5–7), computed via a training-only transposed-attention pass that reuses sparse LSE normalizers and avoids materializing dense S×S maps (Eqs. 8–9, App. D). A delayed mLSTM scorer can defer the keep/drop decision until the protected-window exit, using near-future context. On AIME/HMMT with Qwen3-4B/8B, KVpop and a stateless MLP variant outperform StreamingLLM, TOVA, a trained StreamingLLM+, and parameter-matched DMS under matched budgets (Table 1), with near-teacher OOD results on GPQA-D and LiveCodeBench (Table 2) and favorable latency/VRAM at long generation lengths (Fig. 4).","tokens_in":23828,"tokens_out":1401,"duration_ms":27700,"significance":"KV-cache growth is a central systems bottleneck for long-context decoding; a learned fixed-budget eviction policy that preserves hard reasoning quality would be practically valuable. The work’s main technical contributions—boundary-aware supervision with an efficiently computed future-attention target, and delayed stateful scoring that uses protected-window context before the decision—are concrete and well-motivated relative to heuristic eviction and to DMS-style relaxed gates. Empirically, the matched-budget comparisons (especially StreamingLLM+ isolating pattern adaptation, and DMS isolating the retention objective) and the homogeneous per-head budget that yields cleaner long-generation throughput are strengths. If the reported near-teacher AIME/HMMT retention under 75–88% compression holds under broader evaluation, the method is a meaningful advance for learned eviction retrofits.","major_comments":[{"comment":"Relative-performance numbers are inconsistent across the manuscript and do not always match Table 1. The abstract reports 98%/97% of teacher on Qwen3-4B at 75%/88% compression; §1 Contributions and §5 Conclusion report 95%/94% (and 95%/99% for Qwen3-8B). Table 1 gives Rel. 0.98/0.97 for Qwen3-4B and 1.00/1.00 for Qwen3-8B. These are load-bearing headline claims; please recompute from the Abs. columns, reconcile abstract/intro/conclusion with Table 1, and state the exact aggregation (mean of four pass@1s vs. other).","section":"Abstract; §1 Contributions; Table 1; §5 Conclusion"},{"comment":"The delayed-scoring claim is central (“uniquely among learned eviction methods”) but the only direct ablation (Fig. 5) reports token accuracy (+0.2 points after 2k steps), not AIME/HMMT pass@1 or other task metrics under the same budget. Without a task-level delayed vs. non-delayed comparison for the stateful scorer, it is hard to judge how much of Table 1’s KVpop vs. KVpopmlp gap is due to delay rather than recurrence capacity or the reduced top-k used to match memory. Please add (or clearly state the absence of) a task-level ablation.","section":"§4.2; Figure 5; Table 1"},{"comment":"The future-attention target uses sparse-query LSE as a proxy for dense causal LSE (Eqs. 8–9; App. D.3), and the paper asserts this “matches the dense-LSE target in downstream performance.” That approximation is an axiom of the training signal, yet no main-text or appendix table quantifies target agreement or task delta under dense vs. sparse LSE. A short ablation (or correlation of targets) would make the supervision claim more solid; if already run, report it explicitly.","section":"§3.2; Eqs. (8)–(9); Appendix D.3"}],"minor_comments":[{"comment":"Pass@1 is estimated with 16 rollouts; please report uncertainty (e.g., bootstrap SE or std over seeds) for Table 1 averages, especially where gaps to DMS are a few points.","section":"§4.1; Table 1"},{"comment":"Figure 6 / Appendix Fig. 9 eviction visualizations are informative but qualitative; a simple quantitative breakdown (e.g., retention rates for numeric vs. discourse/operator tokens) would strengthen §4.3.","section":"§4.3; Figure 6; Appendix Figure 9"},{"comment":"Clarify whether StreamingLLM+ and DMS use identical sink/window/top-k (or equivalent B) and the same Nemotron-Math training mixture and KL setup as KVpop; §4 states matching budget and scorer parameter count for DMS, but a one-row hyperparameter table would help reproducibility.","section":"§4 Baselines; Appendix F"},{"comment":"Notation: r̂ vs. r_tgt and effective scores with decay (Eq. 5) are clear, but Agg_g is only specified as max in the main text with alternatives in App. B; a brief main-text sentence on why max was chosen would help.","section":"§3.1; Eq. (3); Appendix B.1"},{"comment":"Typos / polish: “KVcache” spacing is inconsistent; “themLSTM” often lacks a space after “the”; Algorithm 1 line breaks and the “˜LSE” notation in the PDF are hard to read.","section":"Throughout; Algorithm 1"}],"recommendation":"minor_revision","confidential_remarks":"Solid systems/ML methods paper with careful baselines; the number drift between abstract and conclusion looks like an unreconciled draft artifact rather than result instability. Fit is appropriate for a strong ML systems / efficient-inference venue. I would not block on the sparse-LSE point if the authors either report the promised dense-vs-sparse check or clearly demote the claim to “empirically adequate under our budgets.”"},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: if you care about fixed-budget KV eviction that still does long-horizon reasoning, this is one of the cleaner recent attempts. They supervise the keep/drop decision at the protected-window boundary with a future-attention mass target, compute that target via a transposed attention pass that reuses sparse LSE normalizers, and optionally delay an mLSTM scorer so the decision sees near-future context. That combination is the real novelty relative to StreamingLLM/TOVA heuristics, DMC merges, and DMS-style insertion-time gates.\n\nWhat they do well is the experimental hygiene. Table 1 matches budgets and, for DMS, roughly matches scorer capacity; StreamingLLM+ isolates “just train under the sparse pattern” from “learn the ranking.” KVpop and the MLP variant sit clearly above those baselines on AIME/HMMT for Qwen3-4B/8B at 75% and 88% compression, with OOD GPQA-D and LiveCodeBench staying near the teacher. The efficiency story is also concrete: homogeneous per-head budgets give more stable long-generation latency than DMS’s ragged caches, and peak VRAM stays flat. The delayed-scoring ablation is small but points the right way, and the 81% teacher top-k recall is honest supporting evidence rather than hand-waving.\n\nSoft spots, in proportion: the sparse-LSE approximation to dense future mass is the load-bearing proxy, and they under-document how close it is. Train and main eval are both math-heavy, so the relative-retention numbers should not be treated as universal yet. Abstract vs body relative scores drift a few points (98/97 vs 95/94), which is sloppy but does not reverse the ordering. No error bars, free parameters around k/window/decay/loss weights, and methods-level rather than one-command reproducibility. None of that overturns the matched-budget superiority.\n\nThis is for people who ship long-context inference or work on learned sparse attention. It deserves a serious referee. I would engage: cite the objective and the delayed-scorer idea, and push for code plus broader eval before treating the percentages as settled.","headline":"Solid systems paper: future-attention boundary supervision plus delayed scoring actually moves fixed-budget KV eviction on hard math, with matched baselines that make the gains believable.","tokens_in":24462,"tokens_out":538,"would_cite":true,"duration_ms":5417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Supervising keep-or-drop decisions with future attention lets language models discard most of the KV cache while keeping nearly full reasoning accuracy.","keywords":["KV cache compression","predictive online pruning","future-attention target","learned eviction","delayed scoring","sparse attention","fixed-budget cache","autoregressive decoding"],"falsifier":"Train the same scorers and budgets but replace the future-attention target with a pure recency or random ranking (or with dense-LSE targets that systematically disagree with the sparse-LSE approximation) and check whether pass@1 on AIME/HMMT collapses relative to the reported KVpop numbers under identical compression ratios.","tokens_in":24358,"feed_emoji":"🗜️","tokens_out":987,"duration_ms":8607,"temperature":0.7,"pith_summary":"Autoregressive transformers store a growing key-value cache so later tokens can attend to earlier ones, but that cache quickly becomes a memory bottleneck. Most eviction methods drop tokens using fixed heuristics or local scores that do not track whether a token will still matter later. KVpop instead trains a lightweight scorer to decide, at a fixed budget, which older tokens to keep by supervising the keep-or-drop decision with the future attention mass each token receives after it leaves a short protected window. That target is computed during training with a transposed attention pass that never materializes a dense map, and the scorer can optionally wait until the eviction boundary so near-future context can inform the decision. On hard math reasoning, the method keeps roughly 97–98 percent of full-cache performance on Qwen3-4B at 75 percent and 88 percent compression, and approaches full teacher performance on Qwen3-8B, while also generalizing to code and science tasks and keeping peak memory nearly flat for long generations.","feed_headline":"Keep 12% of the KV cache, keep 97% of math accuracy","feed_subtitle":"Future-attention supervision decides which tokens stay under a fixed budget and still matches full-cache reasoning.","key_machinery":"Future-attention target plus boundary-aware retention loss: each token’s keep/drop label is defined by the mean attention mass it receives after leaving the protected window (aggregated across query heads sharing a KV head), recovered efficiently by a transposed-attention pass that reuses sparse log-sum-exp normalizers; the scorer is trained with a pairwise logistic loss only against the teacher’s top-k cutoff token, and may use a delayed memory-based (mLSTM) readout that scores the token only when it becomes eligible.","core_discovery":"Fixed-budget KV eviction can be cast as a supervised prediction problem at the eviction boundary: a scorer trained against each token’s post-window future-attention mass, optionally delayed until near-future context is available, recovers nearly full dense-attention accuracy under aggressive compression and outperforms both heuristic and prior learned eviction baselines on mathematical reasoning.","pith_inferences":["If future-attention mass is a transferable utility signal, the same boundary-supervised scorer could be distilled once and reused across related model families without re-collecting dense teacher maps.","The method suggests that irreversible eviction can approach sparse-retrieval quality on reasoning workloads provided the keep decision is postponed until a short future window is observed.","Hybrid designs that give a few critical layers denser budgets while keeping most heads under KVpop-style fixed top-k may further close residual quality gaps without returning to linear memory growth."],"forward_implications":["Aggressive fixed KV budgets (75–88 percent compression) can preserve near-teacher math-reasoning accuracy when eviction is supervised by future attention rather than local proxies.","Delaying the scoring decision until a token leaves a short protected window lets memory-based scorers use near-future context and improves token accuracy over immediate scoring.","The same learned policy, trained only on math traces, remains competitive on out-of-domain code generation and STEM reasoning without retraining the eviction rule.","Homogeneous per-head fixed budgets yield more regular sparse patterns and lower long-generation latency than dynamic gates that concentrate capacity in a few heads."],"fun_headline_variants":["KVpop keeps 97% math accuracy with 12% KV cache","Future-attention scorer prunes KV cache to 12% and holds 97%","Supervised keep-or-drop hits 97% accuracy at 88% KV compression","Delayed scorer matches full attention after pruning most of the KV cache","Fixed-budget eviction trained on future attention retains 97% on AIME"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The paper assumes that how much future attention a token receives after leaving the protected window is a good enough proxy for whether keeping that token actually helps the final answer, even when the normalizers used to measure that attention come from the sparse student rather than full dense attention.","fun_headline_variants_meta":{"raw":{"variants":["KVpop keeps 97% math accuracy with 12% KV cache","Future-attention scorer prunes KV cache to 12% and holds 97%","Supervised keep-or-drop hits 97% accuracy at 88% KV compression","Delayed scorer matches full attention after pruning most of the KV cache","Fixed-budget eviction trained on future attention retains 97% on AIME"]},"model":"grok-4.5","effort":"low","cost_usd":0.005292,"raw_usage":{"total_tokens":1412,"prompt_tokens":747,"num_sources_used":0,"completion_tokens":106,"cost_in_usd_ticks":52920000,"prompt_tokens_details":{"text_tokens":747,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":559,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":747,"tokens_out":106,"duration_ms":4409,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T09:23:42.792638+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same scorers and budgets but replace the future-attention target with a pure recency or random ranking (or with dense-LSE targets that systematically disagree with the sparse-LSE approximation) and check whether pass@1 on AIME/HMMT collapses relative to the reported KVpop numbers under identical compression ratios.","supporting_citations":[],"review_version":1}