{"id":"cc0a4a8a-a60e-4d43-9a8f-cd258f051405","arxiv_id":"2607.16213","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SelKV compresses the KV cache to 25% with per-token cosine-gated merging and an attention-ratio logit bias, staying within ~0.7 points of full-cache LongBench accuracy on GQA models.","lead":"This paper proposes SelKV, a training-free way to compress the key-value cache that large language models use during text generation, keeping only a quarter of the tokens and merging the rest with a similarity-based 'soft' rule plus an attention-correction bias. On the 16-dataset LongBench benchmark it comes within about 0.7 points of the uncompressed model on grouped-query-attention models and reports a 3.3x decoding speedup at 100k tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core merge-target routing (Eq. 3) is not computable from the stated observation-window attention tensor; the central algorithm as written is unreproducible.","rationale":"The reader's weakest assumption was that prefill observation-window attention remains representative during decode. That is a plausible empirical risk but not the most immediate gap. The strongest, most specific problem is internal: Section 3.4's Eq. 3 indexes A_{h,j,i} with query j, yet the paper only materializes attention for the last W query positions (Section 3.1). For an evicted token j outside that window, the tensor entry does not exist. The prose explains that routing should compare how observation-window queries attended to j versus to candidates, which is a vector-column comparison, not a scalar argmax. The equation and the text therefore describe different algorithms. Since merge-target routing is the core of the merge mechanism, and no code is provided, the central claim cannot be verified or refuted from the written description. This concern is more fundamental than statistical noise or attention drift because it invalidates the method's specification, not just the strength of its empirical support. The paper's self-acknowledged limitations (three models, no code, single runs) further support an UNVERDICTED status: conditional acceptance would require a fixable minor issue, whereas here a load-bearing step is undefined. A concrete test—requesting the exact routing implementation and rerunning the main table—would settle whether the concern lands.","tokens_in":13845,"tokens_out":9184,"duration_ms":99785,"concrete_test":"Obtain the authors' code (or a precise statement) for the routing step. For a randomly chosen evicted token j not in the last 32 positions, write down exactly how A_{h,j,i} in Eq. 3 is computed from the materialized H×W×n tensor. If it uses full attention, the complexity claim in §A fails; if it compares column vectors (e.g., cosine or argmax of sum over W of A_{h,q,j}·A_{h,q,i}), the equation must be corrected. Then re-run the LLaMA-3.1 16-dataset 25%-budget evaluation with the corrected, stated routing and confirm the reported 39.81 average.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 3.4 defines merge-target routing as π_h(j) = argmax_{i∈S∩B(j)} A_{h,j,i}, indexing attention from query j to key i. But Section 3.1 explicitly materializes only an H×W×n attention matrix for the last W=32 query positions, and evicted token j is generally not in W. Thus A_{h,j,i} is undefined for most evicted tokens. The accompanying text says routing uses the attention the observation-window queries paid to j and to candidate targets — a vector comparison between columns, not the scalar argmax written. So either the equation is a typo or the method requires the full n×n attention matrix, contradicting the claimed O(W·n) two-pass prefill. Because routing determines which kept token absorbs each evicted token's value (Eq. 5), the entire merge construction and subsequent compensation (Eq. 6) depend on this unspecified step. Without code, the reported LongBench results cannot be reproduced from the paper.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SelKV, a training-free KV-cache compression method that operates one-shot after prefill. It combines (i) a soft cosine gate that continuously modulates the merge intensity of each evicted token based on value-vector similarity, and (ii) an attention-ratio compensation that adds a logit bias α·log(R) during decoding to correct the attention sag of merged positions. The pipeline is described as six stages: importance scoring via attention×value-norm over a W=32 observation window, top-m selection, bucket-attention merge-target routing, gated attention-weighted merging, compensation-ratio computation, and optional RoPE repositioning. Evaluations on 16 English LongBench datasets at a 25% KV retention budget across Gemma-2-9B-IT, LLaMA-3.1-8B-Inst, and LongChat-7B show the best compressed average on the two GQA models, with reported gaps of -0.67 and -0.73 to the full cache, and a 3.3x decode speedup at 100k tokens. Ablations show blind merging hurts, the gate recovers most loss, and compensation adds a small further gain.","tokens_in":14023,"tokens_out":7806,"duration_ms":80186,"significance":"If the reported results are reproducible, SelKV offers a useful plug-in for KV compression: it is training-free, introduces no learned parameters, and its ablation design is honest (it shows blind merging can hurt, and that the gate/compensation contribute incrementally). The method is fully specified apart from the routing equation (see major comment), and the authors report hyperparameter sensitivity for W and K. The central claims—near-lossless GQA compression and beat-the-full-cache on multi-doc QA—are, however, based on small average gaps and per-dataset differences that lack error bars or significance tests, so the quantitative strength of the contribution is currently not established at the level the abstract implies.","major_comments":[{"comment":"Eq. (3) defines the merge target as π_h(j)=argmax_{i∈S∩B(j)} A_{h,j,i}, but §3.1 explicitly materializes only an H×W×n attention tensor for the last W=32 query positions. Since evicted token j is typically outside W (and recent-window tokens are retained), A_{h,j,i} is not available for the routing step. The text following Eq. (3) describes a different operation—looking up attention from window queries to j and to candidates—which is not the scalar argmax written. Because the merge target feeds Eq. (5) and Eq. (6), the algorithm as written cannot be reproduced without either the full n×n attention matrix (contradicting the O(W·n) prefill claim) or a precise restatement of the routing statistic computed from the H×W×n tensor. This needs to be resolved (ideally with code).","section":"§3.4, Eq. (3)"},{"comment":"The compensation mechanism introduces a scale α, set to 0.5 with no reported sensitivity sweep. The ablation in Table 2 shows that compensation adds +0.24 on LLaMA-3.1 at this single α, and the Gemma-2 6-dataset subset reports -0.02 for the gate alone. The claim that attention-ratio compensation is 'naturally calibrated' therefore rests on a hand-picked α. A sweep over α (e.g., 0.0–1.0) on at least one model is needed to show that the benefit is not a result of tuning on the test set.","section":"§3.6/§4.3, Table 2"},{"comment":"The claim that SelKV 'outperforms the full-cache baseline on complex multi-document QA tasks' is based on differences of 0.06–0.71 points (e.g., MuSiQue 22.90 vs 22.84, HotpotQA 39.44 vs 39.04). With 200 samples per dataset, these gaps are within typical LongBench noise, and no error bars or significance tests are reported. The abstract and §5 should either be softened to 'matches or slightly exceeds in some settings' or supported with variance estimates/statistical tests.","section":"§4.2, Table 1 and §5"}],"minor_comments":[{"comment":"When the union set exceeds the layer budget, trimming by mean aggregate score can discard a token that is critical to a single KV head because another head's selections inflated the union. Clarify the trimming rule and discuss whether per-head guarantees are preserved.","section":"§3.3 (GQA union selection)"},{"comment":"RoPE repositioning is described as part of the pipeline (Eq. 7) but disabled by default. The main text should state clearly that all reported results use the disabled setting; the current wording in §4.1 does so, but the algorithm summary in §3.8 still lists repositioning as a default step, which is confusing.","section":"§3.7"},{"comment":"The comparison omits recent merging methods D2O, WeightedKV, and ChunkKV, which are discussed in related work. The claim of 'representative one-shot baselines' is narrower than the related-work survey suggests; adding at least one of these would strengthen the comparison.","section":"§4.1 / Baselines"},{"comment":"The sensitivity of W and K is reported only on a 6-dataset subset. A statement about the full 16-dataset behavior would be more convincing, especially since the default choices are used in all main results.","section":"Appendix B, Table 3"},{"comment":"No code or link to an implementation is provided. Given the ambiguity in Eq. (3) and the complexity of the pipeline, a code release would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent empirical study with an honest ablation, but the Eq. (3) routing issue is a serious reproducibility gap. The claimed 'outperforms full cache' results are also overstated without significance testing. Both are fixable within the scope of a revision; I would not reject on the basis of the core idea, but the current manuscript is not acceptable as is. If the authors release code or precisely specify the routing step, add an α sweep, and provide error bars, the contribution could meet the bar for a journal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely reasonable KV-cache merging paper. The soft cosine gate and attention-ratio compensation are new relative to the cited literature, the ablations are honest, and the GQA results at 25% budget are competitive. But the paper as written is not reproducible: the routing equation (Eq. 3) indexes an attention matrix the method never materializes, and no code is shipped. The reader's conditional verdict is right; I would push it to major revision with code rather than accept.\n\nWhat is new and what works. The gate g = max(cos_sim, 0) turns the merge-versus-evict trade-off into a continuous per-token decision, something the merging literature has mostly treated in binary terms. The attention-ratio compensation (Eq. 6) is a sensible alternative to KeepKV's raw merge-count bias, and the staged ablation earns its keep: blind merging hurts on LLaMA-3.1 (39.01 vs 39.69 eviction-only), the gate recovers most of the gap, compensation adds a bit more, and they even report where the gate hurts (Gemma-2 subset, -0.02). That is honest reporting. The headline results are plausible: best compressed method on both GQA models, within about 0.7 points of the full cache, and the 10-90% retention sweep shows the ranking is stable from 25% up.\n\nThe soft spots, in order of size. First, routing. Eq. 3 routes evicted token j to argmax_i A_{h,j,i}, but the method only materializes attention for the last W=32 query positions, and evicted tokens are generally not in that window. The accompanying prose says they use the attention that observation-window queries paid to j and to its candidate targets, which is a column comparison, not the scalar argmax written. So the equation is either a typo or the prefill is O(n^2). Either way, the merge construction and the compensation both depend on this step, and without code the reported numbers cannot be reproduced from the paper. That is the main defect, and the stress-test note is right about it. Second, the 'beats full cache' moments are sub-point single-run differences (MuSiQue 22.90 vs 22.84, HotpotQA 39.44 vs 39.04) with no error bars, landing in exactly the multi-document tasks where such artifacts are common. Do not lean on those. Third, alpha=0.5 is hand-set; they sweep W and K but not the compensation scale, and compensation's contribution is small enough (0.24 on LLaMA-3.1) that a sensitivity check matters. Fourth, KeepKV, the paper they build on for compensation, is never evaluated as a baseline. That is a real omission. The decode-time drift worry (frozen prefill statistics) is plausible but speculative; the RoPE repositioning collapse on PassageRetrieval shows the family is sensitive to positional mismatch, but it does not by itself refute the method.\n\nWho this is for: people working on training-free KV compression. It is a useful, incremental, well-ablated data point. It deserves a serious referee, but the authors should be required to fix the routing spec and ship code.","headline":"Solid training-free KV-cache merging paper with two genuinely new mechanisms and honest ablations, but Eq. 3 routes evicted tokens via an attention tensor the method never materializes and no code is shipped, so it is not reproducible as written.","tokens_in":14717,"tokens_out":5787,"would_cite":true,"duration_ms":59104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"At a 25% KV cache budget, a per-token merge-or-drop gate with attention compensation keeps GQA model quality near-lossless and can beat the full cache on multi-document QA.","keywords":["KV cache compression","token merging","soft cosine gate","attention compensation","grouped-query attention","attention sag","training-free compression","long-context inference"],"falsifier":"Run a long-context retrieval task where the relevant evidence sits far from the last 32 query positions and must be re-attended later during generation, such as a needle-in-a-haystack test at varied depths. If SelKV's retrieval accuracy drops far below full cache while a variant that refreshes attention statistics every N decode steps stays close, the frozen-prefill-snapshot assumption fails.","tokens_in":13566,"feed_emoji":"⚡","tokens_out":5548,"duration_ms":53915,"temperature":0.7,"pith_summary":"The paper tries to show that KV cache compression does not have to choose between eviction and merging. A training-free, per-token gate decides for each evicted token whether to merge it into a retained entry, drop it, or merge it partially, based on the similarity of value vectors. A decoding-time logit bias, derived from prefill attention, then corrects the softmax imbalance that merged positions suffer. On 16 long-context English datasets at 25% KV retention, the method stays within about 0.7 points of the full cache on two grouped-query attention (GQA) models, slightly exceeds full-cache scores on several multi-document QA tasks, and decodes 3.3 times faster at 100k tokens.","feed_headline":"Per-token merge-or-drop keeps LLM quality near-lossless at 25% cache","feed_subtitle":"Training-free cosine gate plus attention compensation: within 0.7 points of full cache and 3.3x faster decode.","key_machinery":"The load-bearing object is the per-token soft cosine gate g=max(cos_sim,0), computed between the value vector of each evicted token and its merge target; it interpolates continuously between pure drop and full merge without learned parameters. Around it sits attention-ratio compensation R=(a_kept + sum(g*a_discarded))/a_kept, applied as alpha*log(R) to attention logits during decoding, which corrects attention sag without relying on raw merge counts. All quantities come from one prefill pass restricted to an observation window of W=32 query positions, so compression is one-shot and the decode phase only adds a scalar bias per position.","core_discovery":"The central claim is that merge quality, not just token selection, is a decisive axis for KV cache compression. SelKV derives, from a single prefill attention snapshot over the last 32 query positions, a per-token importance score (attention multiplied by value norm), a merge target for each evicted token via bucket attention, and a soft gate g=max(cos_sim,0) between value vectors. The gate makes merging selective: similar tokens merge fully, dissimilar tokens are dropped, and intermediate cases merge partially. Attention compensation then adds alpha*log(R) to decode logits, where R is the ratio of attention mass absorbed into each kept position, so merged entries are not under-attended. The","pith_inferences":["The frozen prefill snapshot is the natural stress point: if decode-time attention drifts from the last-32-query statistics, the compensation bias will be miscalibrated. A testable extension is to recompute or interpolate the attention ratios periodically during long generations.","The attention-ratio compensation may generalize beyond cosine gating, since it is calibrated by attention mass rather than merge counts; it could be plugged into learned or task-adaptive merge policies.","The gap between MHA and GQA results suggests head-specific compensation factors or per-head gating could close the MHA gap; the paper does not explore per-head alpha.","The method's weakness on code tasks, where exact token identity matters, points to a hybrid policy: use eviction for identity-sensitive tasks and selective merging for semantic tasks."],"forward_implications":["If true, a 25% KV retention budget is enough for near-lossless generation on GQA models, cutting KV memory by 75% with a quality gap under one point.","Decode throughput gains grow with context length: 3.3x at 100k tokens, so long-context inference benefits most.","Selective merging can act as an implicit attention filter: on some multi-document QA tasks the compressed cache scores above the full cache, suggesting that dropping or merging distracting tokens helps.","The gate is portable: adding it to an existing merging pipeline yields small quality gains on the GQA models tested.","At moderate budgets (around 80% retention on one GQA model), the method slightly exceeds full-cache quality, implying compression itself can regularize attention."],"fun_headline_variants":["SelKV: training-free KV cache merge, near-lossless at 25% cache, 3.3x faster","Selective KV merging with attention compensation: 25% cache, near-lossless","Merge-or-drop plus attention fix: LLM quality at 25% KV cache","25% KV cache via smart merging: near-lossless quality, 3.3x decode","SelKV: per-token merge-or-drop keeps LLM quality with 75% less cache"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on the assumption that the attention pattern captured by the last 32 query positions before generation stays representative of attention throughout decoding; if attention drifts, the frozen merge decisions and compensation bias are miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["SelKV: training-free KV cache merge, near-lossless at 25% cache, 3.3x faster","Selective KV merging with attention compensation: 25% cache, near-lossless","Merge-or-drop plus attention fix: LLM quality at 25% KV cache","25% KV cache via smart merging: near-lossless quality, 3.3x decode","SelKV: per-token merge-or-drop keeps LLM quality with 75% less cache"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2713,"prompt_tokens":778,"completion_tokens":1935,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":1815}},"tokens_in":522,"tokens_out":1935,"duration_ms":15134,"temperature":1.0,"reasoning_tokens":1815,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:35:08.454759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a long-context retrieval task where the relevant evidence sits far from the last 32 query positions and must be re-attended later during generation, such as a needle-in-a-haystack test at varied depths. If SelKV's retrieval accuracy drops far below full cache while a variant that refreshes attention statistics every N decode steps stays close, the frozen-prefill-snapshot assumption fails.","supporting_citations":[],"review_version":1}