{"id":"97b2bad6-9e83-4f20-898f-0626a893f1b4","arxiv_id":"2605.26678","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"NestedKV is a new training-free KV cache compression technique using nested memory anchors and multi-time-scale anomaly scoring that outperforms prior methods like KeyDiff on long-context benchmarks when the retained cache fraction is small.","lead":"The paper introduces NestedKV, a training-free method for compressing KV caches in long-context LLMs using nested global/block/sliding-window anchors, multi-scale cosine anomaly scoring, and head-adaptive mixing. A smart generalist might read it to understand practical ways to reduce memory use for longer AI conversations and documents.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"The multi-scale cosine anomaly routing with head-adaptive mixing may fail to select useful tokens when key-vector similarity does not align with semantic importance.","rationale":"The reader's weakest_assumption directly identifies the same generalization risk. The empirical numbers on the two tested model families are consistent with the claim, but the absence of results on additional architectures leaves the core routing assumption as the least secure link. No internal inconsistency or formal error is visible from the abstract-level description.","tokens_in":1777,"tokens_out":347,"duration_ms":36136,"concrete_test":"Re-run the r=0.75 LongBench evaluation on Llama-3.1-8B (a model family not used in the main Qwen3 results) using the same NestedKV configuration; if average score improvement over KeyDiff drops below 8 points, the cross-model reliability of the routing weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that global, block-level, and sliding-window key anchors scored via multi-time-scale cosine anomaly, then combined by head-adaptive mixing and surprise-gating, reliably surface the tokens that matter most. This heuristic could break when importance is driven by value vectors, cross-head interactions, or patterns not visible in key cosine distances (e.g., long-range dependencies that are semantically critical but key-similar to many others). The reported gains on Qwen3-4B (up to 19.10 RULER, 19.29 LongBench at r=0.75) rest on this assumption holding for the tested contexts; the paper provides no explicit evidence that the same routing succeeds on models whose key distributions differ substantially.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces NestedKV, a training-free KV cache compression method for long-context LLMs. It maintains key anchors at global, block-level, and sliding-window scales, scores tokens using multi-time-scale cosine anomaly, and combines rankings via head-adaptive mixing and surprise-gating with adaptive per-head budgets. The approach is evaluated on Qwen3 and Llama-3.2 models across RULER (4k-32k), LooGLE, LongBench, LongBench-E, InfiniteBench, and MMLU-Pro, claiming to outperform baselines such as KeyDiff especially at low retention ratios (e.g., up to 19.10 points on RULER and 19.29 on LongBench at r=0.75 on Qwen3-4B; 37.32 vs 17.55 on LongBench at r=0.95).","tokens_in":1945,"tokens_out":603,"duration_ms":39223,"significance":"If the results hold, NestedKV would advance training-free KV compression by integrating multiple time scales in a nested routing scheme, addressing brittleness in single-signal methods. The empirical gains on multiple benchmarks at small cache sizes represent a practical contribution for efficient long-context inference without model changes or training. The training-free nature and head-adaptive components are notable strengths if robustly validated.","major_comments":[{"comment":"The core claim that global/block/sliding-window key anchors scored by multi-time-scale cosine anomaly, then routed via head-adaptive mixing and surprise-gating, reliably identify useful tokens (as stated in the method and supported by the Qwen3-4B results) rests on an untested assumption that key-vector cosine distances align with semantic importance. No ablation or counterexample analysis is provided for cases where importance derives from value vectors, cross-head interactions, or long-range patterns invisible in key similarity; this directly bears on the generalizability of the reported gains.","section":"Method (NestedKV routing description)"},{"comment":"Table reporting Qwen3-4B results at r=0.75 and r=0.95: the headline improvements (19.10 RULER, 19.29 LongBench) lack error bars, run counts, or statistical tests, making it impossible to assess whether gains exceed variance or benchmark selection effects; this undermines confidence in the central performance claim.","section":"Experiments (benchmark tables)"}],"minor_comments":[{"comment":"The abstract uses 'r' for retention ratio without an immediate definition or reference to the equation defining it.","section":"Abstract"},{"comment":"Benchmark names such as LooGLE and LongBench-E would benefit from a one-sentence description or citation on first use for readers unfamiliar with the suite.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for recognizing the potential practical contribution of NestedKV. We address each major comment below.","responses":[{"response":"NestedKV is explicitly a key-only method, chosen to support efficient, training-free compression that does not require access to value vectors at compression time. We agree that value-based signals or cross-head interactions could matter in some settings and that the paper does not include targeted ablations or counterexamples for those cases. In revision we will add a limitations paragraph explicitly discussing the key-only design choice, its rationale, and the scope of generalizability. We do not plan new experiments for this revision.","revision_made":"partial","referee_comment":"[Method (NestedKV routing description)] The core claim that global/block/sliding-window key anchors scored by multi-time-scale cosine anomaly, then routed via head-adaptive mixing and surprise-gating, reliably identify useful tokens (as stated in the method and supported by the Qwen3-4B results) rests on an untested assumption that key-vector cosine distances align with semantic importance. No ablation or counterexample analysis is provided for cases where importance derives from value vectors, cross-head interactions, or long-range patterns invisible in key similarity; this directly bears on the generalizability of the reported gains."},{"response":"We agree that the absence of error bars and statistical tests weakens confidence in the headline numbers. In the revised manuscript we will report results over multiple random seeds (where the benchmark permits stochasticity), include standard deviations, and add paired statistical tests comparing NestedKV against the strongest baseline.","revision_made":"yes","referee_comment":"[Experiments (benchmark tables)] Table reporting Qwen3-4B results at r=0.75 and r=0.95: the headline improvements (19.10 RULER, 19.29 LongBench) lack error bars, run counts, or statistical tests, making it impossible to assess whether gains exceed variance or benchmark selection effects; this undermines confidence in the central performance claim."}],"tokens_in":1535,"tokens_out":445,"duration_ms":26620,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that NestedKV improves over single-signal baselines like KeyDiff by up to 19 points on RULER and LongBench when the retained KV cache is small (r=0.75) on Qwen3-4B. At r=0.95 it still holds 37 on LongBench versus 17 for the baseline.\n\nWhat is new is the explicit nesting of anchors at three scales plus the training-free outer learner that mixes them with head-adaptive weights and surprise gating. Earlier KV compression picked one importance signal; this stacks three and routes per head without any fine-tuning or model changes. It does well by staying simple to implement and by reporting results across RULER, LongBench, LooGLE, InfiniteBench, and MMLU-Pro on both Qwen3 and Llama-3.2 families.\n\nThe soft spot is the assumption that multi-time-scale cosine anomaly on keys will surface the tokens that matter. If semantic importance lives mainly in value vectors, cross-head interactions, or long-range patterns that look similar in key space, the routing can miss them. The paper gives no targeted checks for those failure modes, so the reported gains could be tied to the specific contexts and models tested. That is the central risk flagged in the stress test.\n\nThis paper is for people who need practical, training-free KV compression for long-context inference. A reader working on deployment constraints or inference optimization would find the architecture and the head-level budget adaptation useful.\n\nI would send it to peer review. The problem is real, the incremental step is clear, and the effect sizes are large enough that referees should examine whether the routing generalizes.","headline":"NestedKV layers global/block/window key anchors with multi-scale cosine anomaly and head-adaptive mixing to beat KeyDiff at tight cache budgets on Qwen3, but the gains rest on untested alignment between key similarity and token utility.","tokens_in":2468,"tokens_out":430,"would_cite":false,"duration_ms":30749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"NestedKV routes KV cache tokens through nested global, block and window anchors scored by multi-scale cosine anomaly to preserve performance at low retention ratios.","keywords":["KV cache compression","long-context language models","token importance scoring","memory routing","training-free methods","attention cache","context length extension"],"falsifier":"On a new model architecture or benchmark where attention patterns differ markedly from Qwen3 and Llama-3.2, the method shows no gain over KeyDiff at retention ratio 0.75 on RULER or LongBench.","tokens_in":2687,"feed_emoji":"","tokens_out":650,"duration_ms":28004,"temperature":0.7,"pith_summary":"The paper proposes a training-free compression method for the key-value cache in long-context language models. It keeps three nested sets of key anchors at global, block, and sliding-window scales, then scores each token by how much its key deviates from those anchors using cosine anomaly at multiple time scales. These scores feed into a routing step that mixes them adaptively per attention head and gates them by surprise before deciding which tokens to retain under per-head budgets. The goal is to handle cases where useful context appears globally distinctive, locally episodic, or immediately relevant, rather than relying on any single importance signal. If the routing works as described, models could run longer sequences with far less memory while losing less accuracy than prior single-signal methods.","feed_headline":"Nested anchors keep accuracy in heavily compressed KV caches","feed_subtitle":"Multi-scale cosine anomaly scoring plus head-adaptive routing outperforms single-signal baselines at low retention ratios on RULER and LongB","key_machinery":"Nested memory routing that scores tokens by multi-time-scale cosine anomaly from global, block-level and sliding-window key anchors then combines them with head-adaptive mixing and surprise-gated token routing.","core_discovery":"NestedKV maintains global, block-level, and sliding-window key anchors, scores tokens by multi-time-scale cosine anomaly, and combines the resulting rankings with a training-free outer learner using head-adaptive mixing and surprise-gated token routing. The score is paired with adaptive per-head budgets and requires no training or LLM modification.","pith_inferences":["The nested-anchor approach might extend to compressing other sequence memories such as replay buffers in reinforcement learning.","Multi-scale anomaly scoring could be tested as a drop-in replacement for single-scale importance metrics in retrieval-augmented generation.","If the routing proves robust, it opens the possibility of dynamically adjusting retention per head during inference rather than fixing budgets in advance."],"forward_implications":["At retention ratio 0.75 the method yields up to 19-point gains on RULER and LongBench over KeyDiff on Qwen3-4B.","At retention ratio 0.95 it still keeps substantially higher LongBench scores than KeyDiff.","The same routing works without modification on both Qwen3 and Llama-3.2 families.","No training or architecture change is required for the gains.","The method is strongest precisely when the retained cache fraction is smallest."],"fun_headline_variants":["Nested anchors score keys across scales for KV compression","Multi-scale cosine anomaly outperforms single-signal KV methods","Head-adaptive routing retains accuracy in small KV caches","Global block and window anchors compress long-context KV cache"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The combination of global, block-level, and sliding-window key anchors scored by multi-time-scale cosine anomaly and routed via head-adaptive mixing and surprise-gating will reliably identify the most useful tokens across diverse contexts and models without any model-specific tuning or training.","fun_headline_variants_meta":{"raw":{"variants":["Nested anchors score keys across scales for KV compression","Multi-scale cosine anomaly outperforms single-signal KV methods","Head-adaptive routing retains accuracy in small KV caches","Global block and window anchors compress long-context KV cache"]},"model":"grok-4.3","cost_usd":0.003667,"raw_usage":{"total_tokens":1913,"prompt_tokens":677,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":36674500,"prompt_tokens_details":{"text_tokens":677,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1177,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":677,"tokens_out":59,"duration_ms":10622,"temperature":1.0,"reasoning_tokens":1177,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T18:09:05.266510+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"On a new model architecture or benchmark where attention patterns differ markedly from Qwen3 and Llama-3.2, the method shows no gain over KeyDiff at retention ratio 0.75 on RULER or LongBench.","supporting_citations":[],"review_version":1}