{"id":"e0222bf5-0383-4cca-a0ff-ee77a23d8a1d","arxiv_id":"2606.24467","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CompressKV uses Semantic Retrieval Heads to guide KV-cache token selection and layer-wise budget allocation, retaining over 97% performance with 3% cache on LongBench QA tasks.","lead":"CompressKV identifies Semantic Retrieval Heads in grouped-query attention LLMs to decide which tokens to keep in the KV cache during long-context inference. A smart generalist might read it because shrinking the memory and compute cost of long contexts could let advanced models run on phones and edge hardware instead of only large servers.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Stability of identified Semantic Retrieval Heads and offline layer-wise error estimates across models/tasks not demonstrated","rationale":"The reader's weakest_assumption directly names the two transfer conditions required for the central claim. Because the provided abstract supplies no counter-evidence (cross-model results, online/offline correlation, or distribution-shift tests), the concern remains load-bearing; the concrete_test above would falsify it in one controlled experiment.","tokens_in":1782,"tokens_out":348,"duration_ms":10691,"concrete_test":"Re-run the full LongBench QA and NIAH suites on a second GQA model (e.g., Mistral-7B or Qwen2-7B) using the exact SRH identification procedure and offline budget allocator from the paper; if average performance drop exceeds 5 points relative to the Llama-3 numbers at the same 3%/0.7% budgets, the generalization claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline performance numbers (97% retention at 3% cache on LongBench QA; 90% NIAH at 0.7%) rest on two untested transfer assumptions: (1) that heads whose attention concentrates on prompt start/end plus a few mid-context evidence tokens remain the same SRHs when the model, task distribution, or prompt length changes, and (2) that the static per-layer eviction-error ranking computed offline continues to allocate budgets correctly during online autoregressive decoding. The abstract gives no indication of cross-model ablation, sensitivity to prompt distribution shift, or online vs. offline error correlation; if either assumption fails, the token-selection rule and budget allocator both degrade.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces CompressKV, a KV-cache compression framework for GQA-based LLMs. It identifies Semantic Retrieval Heads (SRHs) whose attention focuses on prompt start/end tokens and semantically important mid-context evidence to guide token retention, and allocates per-layer cache budgets according to offline-computed layer-wise eviction errors. On LongBench question-answering tasks the method is reported to retain over 97% of full-cache performance with 3% of the KV cache; on Needle-in-a-Haystack it reaches 90% accuracy with 0.7% storage, outperforming prior eviction baselines.","tokens_in":1942,"tokens_out":601,"duration_ms":18915,"significance":"If the central claims hold after the required clarifications, the work would demonstrate a practical improvement in the performance-memory trade-off for long-context inference by exploiting head specialization instead of uniform heuristics across all heads. The public release of code at the cited GitHub repository is a clear strength that supports reproducibility and further experimentation.","major_comments":[{"comment":"Abstract and §3 (Method): the procedure used to identify Semantic Retrieval Heads—specifically how attention concentration on prompt start/end tokens and mid-context evidence is quantified, thresholded, or selected—is not described. Because SRH selection directly determines which tokens are retained, this omission prevents assessment of whether the reported 97% retention is reproducible or generalizes.","section":"Abstract / §3"},{"comment":"§4 (Experiments): no cross-model, cross-task, or prompt-distribution-shift ablations are presented to test whether the SRHs identified on the training distribution remain stable. The headline claims (97% retention at 3% cache on LongBench QA; 90% NIAH accuracy at 0.7% storage) rest on the untested assumption that these heads transfer; without such evidence the generalization argument is unsupported.","section":"§4"},{"comment":"§3 and §4: the offline layer-wise eviction-error estimates used for budget allocation are not validated against online autoregressive decoding error. If the static ranking diverges from runtime behavior, the budget allocator itself becomes unreliable; a correlation plot or online-vs-offline ablation is required to support this component of the method.","section":"§3 / §4"}],"minor_comments":[{"comment":"The abstract states quantitative gains but supplies no standard errors, number of runs, or statistical significance tests for the reported percentages; adding these would strengthen the experimental claims.","section":"Abstract"},{"comment":"Notation for the eviction-error metric and the precise definition of “mid-context evidence” should be formalized with an equation or pseudocode in §3 to improve clarity.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will incorporate clarifications and additional experiments in the revised manuscript to strengthen the presentation of SRH identification, generalization evidence, and validation of the budget allocator.","responses":[{"response":"We agree that §3 requires an explicit description of the SRH identification procedure. In the revision we will add a dedicated subsection with the exact quantification (attention aggregation over start/end and evidence positions), thresholding rule, and selection algorithm, including pseudocode. The public code repository already implements this logic; the expanded text will make the method fully reproducible from the paper alone.","revision_made":"yes","referee_comment":"[Abstract / §3] Abstract and §3 (Method): the procedure used to identify Semantic Retrieval Heads—specifically how attention concentration on prompt start/end tokens and mid-context evidence is quantified, thresholded, or selected—is not described. Because SRH selection directly determines which tokens are retained, this omission prevents assessment of whether the reported 97% retention is reproducible or generalizes."},{"response":"We acknowledge that additional ablations would better support the transferability claim. The revision will include new experiments evaluating SRH stability on at least one additional GQA model, an extra task category, and a prompt-distribution shift (e.g., different LongBench subsets or synthetic variations). Results will be reported in an expanded §4 with the same metrics used in the original evaluation.","revision_made":"yes","referee_comment":"[§4] §4 (Experiments): no cross-model, cross-task, or prompt-distribution-shift ablations are presented to test whether the SRHs identified on the training distribution remain stable. The headline claims (97% retention at 3% cache on LongBench QA; 90% NIAH accuracy at 0.7% storage) rest on the untested assumption that these heads transfer; without such evidence the generalization argument is unsupported."},{"response":"We will add the requested validation. The revised §4 will contain (i) a scatter plot and Pearson correlation between offline eviction-error ranks and online per-layer error measured during autoregressive decoding, and (ii) an ablation comparing end-to-end performance when budgets are assigned via the offline estimator versus an online oracle. These results will directly address the reliability of the static allocator.","revision_made":"yes","referee_comment":"[§3 / §4] §3 and §4: the offline layer-wise eviction-error estimates used for budget allocation are not validated against online autoregressive decoding error. If the static ranking diverges from runtime behavior, the budget allocator itself becomes unreliable; a correlation plot or online-vs-offline ablation is required to support this component of the method."}],"tokens_in":1537,"tokens_out":596,"duration_ms":21854,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"CompressKV's main idea is to pick out a subset of attention heads whose patterns focus on prompt start and end tokens plus key mid-context evidence, then use only those heads to score which KV pairs to keep. They also set per-layer cache budgets from offline eviction-error measurements instead of a uniform rule. This is distinct from prior work that aggregates across all heads.\n\nThe numbers on the two benchmarks are the strongest part: over 97% of full-cache performance retained at 3% KV size on LongBench QA, and 90% needle-in-a-haystack accuracy at 0.7% storage. Public code is available, which is useful for anyone who wants to reproduce or extend it.\n\nThe soft spots are exactly where the stress-test note points. Nothing in the abstract shows that the same Semantic Retrieval Heads remain stable when the model, task, or prompt distribution changes, and the offline layer-wise error rankings have no reported correlation check against online decoding error. If either assumption slips, the token selection and budget allocation lose their grounding. Methods details on head identification and the exact eviction metric are also missing, so it is hard to judge how much of the gain is robust versus setup-specific.\n\nThe work is aimed at people building or deploying long-context inference on memory-limited hardware. A reader who needs concrete compression ratios on standard benchmarks will find the results worth looking at. It has enough empirical signal and a clear, testable claim to justify sending it to peer review rather than desk rejection, provided the full paper supplies the missing ablations on stability and online behavior.","headline":"SRH selection for KV eviction is a reasonable tweak on GQA methods but the transfer assumptions are untested.","tokens_in":2389,"tokens_out":386,"would_cite":false,"duration_ms":13706,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"CompressKV identifies Semantic Retrieval Heads to compress KV cache to 3% size while retaining over 97% of full performance on long-context tasks.","keywords":["KV cache compression","long-context LLM inference","attention heads","semantic retrieval","GQA","eviction methods","resource efficiency","long-context performance"],"falsifier":"Run the method on a held-out model and task at the 3% cache budget; if accuracy falls below the best uniform-eviction baseline at the same budget, the SRH-guided selection is not providing the claimed advantage.","tokens_in":2706,"feed_emoji":"🗜️","tokens_out":663,"duration_ms":21021,"temperature":0.7,"pith_summary":"The paper establishes that standard KV-cache eviction, which aggregates scores across all attention heads, discards critical tokens because heads serve different roles. CompressKV instead locates Semantic Retrieval Heads that attend to prompt start and end tokens plus key mid-context evidence, then retains only the tokens those heads highlight. It further assigns different cache budgets per layer using precomputed eviction-error estimates. If the approach holds, long-context inference becomes feasible on hardware with far less memory while accuracy stays close to the full-cache baseline.","feed_headline":"KV cache cut to 3% keeps 97% accuracy on long tasks","feed_subtitle":"Semantic Retrieval Heads guide token retention and layer budgets, beating uniform eviction on LongBench and Needle-in-a-Haystack at every bu","key_machinery":"Semantic Retrieval Heads (SRHs) that attend to prompt boundaries and mid-context evidence, used to score and retain tokens, together with layer-wise budget allocation driven by offline eviction-error estimates.","core_discovery":"CompressKV identifies Semantic Retrieval Heads in GQA-based LLMs that capture both the initial and final tokens of a prompt and semantically important mid-context evidence, uses their attention scores to select which KV pairs to retain, and allocates per-layer cache budgets from offline layer-wise eviction-error estimates; on LongBench question-answering tasks this keeps over 97% of full-cache performance with only 3% of the KV cache and reaches 90% accuracy on Needle-in-a-Haystack with 0.7% storage.","pith_inferences":["If SRH patterns prove stable, the same identification step could be applied to other attention-based architectures that use grouped-query attention.","The layer-wise error estimates might be replaced by a lightweight online calibration pass if offline computation proves costly for new models.","Success here suggests that other efficiency techniques, such as speculative decoding, could also benefit from routing decisions made only through a small subset of heads.","A direct test would be to measure whether the same SRH set remains optimal when the prompt distribution shifts from question answering to summarization or code completion."],"forward_implications":["Outperforms prior KV-eviction methods at every tested memory budget on LongBench and Needle-in-a-Haystack.","Enables long-context inference on hardware whose memory would otherwise force aggressive truncation or offloading.","Reduces decoding cost proportionally to the retained KV size while accuracy remains near the uncompressed level.","Demonstrates that head specialization can be exploited for compression without retraining the underlying LLM."],"fun_headline_variants":["SRH based KV compression preserves 97% LongBench performance at 3% cache","CompressKV allocates layer budgets using eviction error estimates","Semantic heads capture initial final and mid context evidence for retention","90% accuracy on Needle tasks with 0.7% KV cache storage","GQA LLM inference uses SRHs to select retained KV pairs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The heads that qualify as Semantic Retrieval Heads stay the same and remain sufficient across models, tasks, and prompt lengths, and the offline error estimates transfer directly to online inference without adjustment.","fun_headline_variants_meta":{"raw":{"variants":["SRH based KV compression preserves 97% LongBench performance at 3% cache","CompressKV allocates layer budgets using eviction error estimates","Semantic heads capture initial final and mid context evidence for retention","90% accuracy on Needle tasks with 0.7% KV cache storage","GQA LLM inference uses SRHs to select retained KV pairs"]},"model":"grok-4.3","cost_usd":0.006712,"raw_usage":{"total_tokens":3173,"prompt_tokens":761,"num_sources_used":0,"completion_tokens":88,"cost_in_usd_ticks":67124500,"prompt_tokens_details":{"text_tokens":761,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2324,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":761,"tokens_out":88,"duration_ms":19491,"temperature":1.0,"reasoning_tokens":2324,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T00:10:03.385097+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the method on a held-out model and task at the 3% cache budget; if accuracy falls below the best uniform-eviction baseline at the same budget, the SRH-guided selection is not providing the claimed advantage.","supporting_citations":[],"review_version":1}