{"id":"6ded1909-d92a-434d-bf9c-497878f75f76","arxiv_id":"2608.02901","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AnchorKV compresses the KV cache 20x with near-lossless accuracy by representing each token as an anchor projection plus selectively allocated quantized residuals, beating eviction baselines at matched byte budgets on three model scales.","lead":"This paper presents AnchorKV, a way to compress the key-value memory of large language models by 20x without removing any token from attention. It stores each token as a scaled copy of a nearby anchor token and spends the savings on small corrections where they matter most, keeping long-context accuracy close to the uncompressed model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'retains every token' guarantee rests on an unverified worst-case key-coverage premise: at 20x, 82% of token-sides have no residual, and only median anchor alignment is reported, so a single poorly-covered but attended key can shift the softmax beyond what the budget can repair.","rationale":"We agree with the reader's weakest_assumption. The paper's main theoretical support for the key side is Theorem 1, which depends on the maximum key reconstruction error; the paper itself concedes that residuals cannot control that maximum. Only median alignment is measured, so the central guarantee is conditional on an empirical worst-case coverage property. This is more load-bearing than the lack of error bars or code, because it concerns the method's generality, not just the precision of the reported numbers. The testing protocol above would directly estimate the maximal attention-weighted key error. Until then, the appropriate verdict remains CONDITIONAL, as the reader concluded.","tokens_in":38795,"tokens_out":10172,"duration_ms":92888,"concrete_test":"On Llama-3.1-8B-Instruct at 20x, instrument the decode pass over RULER-32K to record, per (layer, head, sample), the key reconstruction error ||K_t - \\hat K_t|| for tokens that receive the top 5% of reconstructed attention mass at any decoding step. Report the maximum-to-median ratio of these attended-token errors, the worst-case cosine to the selected anchor, and the resulting mu in Theorem 1 using the measured ||q||. If any attended token has cosine below 0.7 or if the max-to-median ratio exceeds 3x with mu > 0.2, the worst-case assumption is violated; if all attended tokens stay above 0.85 cosine and the attention-output error is dominated by the value term, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of AnchorKV—20x compression without discarding a token—rests on the assertion that the compressed cache is a small perturbation of the exact one. Section 3.3 states this explicitly: 'a fixed number of residuals cannot control a maximum over positions,' so the key-side fidelity is delegated to the anchor representation itself. Theorem 1 (Eq. 17) makes this quantitative: the key-induced output error is bounded by 2 V_max tanh(mu), with mu = ||q|| Delta_K / sqrt(D), where Delta_K is the maximum key reconstruction error over all positions. The bound is useful only if Delta_K is small for every position that decoding attends to. The paper does not establish this. It reports the median cosine to nearest anchor (Figure 7) and the median cosine of attention outputs (Figure 10), not the worst case. At 20x, Section E.1 gives 17.3% residual density, so 82.7% of token-sides carry no residual; for those, Delta_K equals the norm of the projection residual r^K_t. A single attended token whose direction is far from every anchor can yield a large Delta_K that the residual budget cannot repair, and since the softmax is a function of all logits, that one token can shift attention weights for every position. The 'retains every token' guarantee then degenerates to retaining a coarse, possibly misleading approximation, and the claimed 99% retention on RULER may not transfer to contexts where a critical token lies in a low-coverage region. The empirical results on the tested benchmarks are extensive and honestly reported, but the worst-case key-coverage premise is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AnchorKV proposes an anchor-residual KV cache compression scheme that keeps every token inside the softmax while compressing the cache up to 20x. Each per-head key and value vector is assigned to its nearest anchor, represented as a scalar-scaled anchor projection, and a budgeted subset of tokens additionally receives 2-bit Hadamard-rotated, Lloyd-Max-quantized residuals chosen by an attention-output-aware utility. The user sets one retention knob, and an exact byte accounting (Eq. 13) converts it into a per-layer residual budget. Experiments on Llama-3.1-8B, Mistral-Small-3.1-24B, and Llama-3.1-70B across RULER, LongBench, and passkey retrieval at matched byte budgets report that AnchorKV at 20x outperforms every eviction baseline at 10x, with 99.3% RULER retention on the 70B model; a FlashAttention-style decoder reduces decode peak memory by about 19x. Appendices supply the codec and storage layout, the utility derivation, an output-error bound (Theorem 1), full per-task tables, and efficiency measurements.","tokens_in":39084,"tokens_out":11576,"duration_ms":104671,"significance":"If the empirical results hold, this is a substantial practical contribution: it demonstrates that 20x KV compression is attainable without eviction, using a principled anchor-residual representation, a parameter-free utility estimate under an explicit incoherence model, and unusually careful byte-level accounting. The worked example in Appendix A.1 reproduces the stated 3.39 MB base footprint and 91,056 residuals, and Theorem 1 is proved cleanly. The evaluation is systematic: three model scales, three benchmark families, matched byte budgets, per-task breakdowns, ablations, and direct attention-output fidelity measurements. The paper also honestly acknowledges in Section 3.3 that a fixed residual budget cannot control a worst-case key error, and it discloses the need to reproduce TurboQuant from community implementations. The main weakness is that the headline 'without discarding a single token' guarantee is stronger than the worst-case analysis actually supports, because only median alignment and fidelity are reported rather than the worst-case key reconstruction error that the bound depends on.","major_comments":[{"comment":"The key-side error term E_K = 2 V_max tanh(mu) depends on Delta_K = max_t ||K_t - hat K_t||, and the paper explicitly concedes in Section 3.3 that 'a fixed number of residuals cannot control a maximum over positions.' The claim that the compressed cache is a perturbation of the exact one therefore rests entirely on the anchor projection being accurate for every key that decoding attends to. The paper reports the median cosine to the nearest anchor (Figure 7) and the distribution of attention-output cosines (Figure 10), but not the worst case or high quantiles. At 20x, Section E.1 gives 17.3% residual density, so 82.7% of token-sides carry no residual, and for those Delta_K = ||r^K_t||; a single poorly covered but attended key can make mu large and, because softmax is a global function of all logits, shift every attention weight. To support the 'retains every token' claim, please report per-layer and per-head maxima or 99.9th percentiles of ||r^K_t|| (equivalently cosine to nearest anchor) on the evaluation workloads, or explicitly reframe the guarantee as an empirical claim validated on these benchmarks.","section":"Section 3.3 / Appendix A.4, Eq. (17)"}],"minor_comments":[{"comment":"The fixed hyperparameter list omits b_a and b_gamma, even though Eq. (13) and the worked example in Appendix A.1 set them to 2; state their values for all experiments or explain how they are derived from the anchor budget k.","section":"Section 4.1 / Appendix B.3"},{"comment":"The TurboQuant baseline is a community reproduction using a different quantization scheme (uniform base plus fp16 outliers) than the original paper's channel split, and the authors report that the verbatim method did not reproduce its published quality; this caveat should appear in the main text and the curve should be labeled as a variant so readers do not mistake it for the published algorithm.","section":"Appendix B.5 / Figure 2"},{"comment":"No variance estimates or repeated-seed runs are reported; because several headline comparisons rest on margins of a few points, please add standard errors or min/max ranges across seeds.","section":"Section 4.2 / Tables 6-15"},{"comment":"At 10x the AnchorKV needle-in-a-haystack grid has many cells below 50% at shallow depths, so the wording 'recovers the needle across nearly the whole grid' overstates the 0.80 mean; adjust the description to match the displayed grid.","section":"Section 4.2 / Figure 9"},{"comment":"In the sentence 'a cosine cleaves a residual of relative norm sqrt(1-c^2)', 'cleaves' should be 'leaves'; Figure 5 also labels the variant 'Naive 2 bit' and should be capitalized consistently.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong fit for the journal and the empirical work is substantial. The main gap is between the worst-case analysis in Theorem 1 and the 'guarantee' language in the abstract and introduction; I would expect the authors to supply the requested distributional or worst-case key-coverage evidence, or to soften the claim. The TurboQuant reproduction issue is secondary but should be cleaned up, and the missing variance estimates should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AnchorKV is one of the more interesting KV-compression papers I've read recently. The core idea—store a few anchor vectors exactly, represent every other token as an anchor index plus a scalar coefficient, then spend a fixed byte budget on quantized residuals for the tokens whose approximation most hurts attention output—is a genuine new combination. It sits between eviction and quantization, and the empirical package is unusually thorough: three model scales, RULER/LongBench/NIAH, matched byte budgets against SnapKV/PyramidKV/AdaKV/TurboQuant, per-task tables, ablations, and real efficiency measurements on an A100. I checked the math the reader flagged: the projection decomposition, pre-RoPE key storage with RoPE distributing over the decomposition, the byte accounting, and the softmax L1 perturbation bound in Theorem 1. All of it checks out. The first-order utility approximation (Eq. 6) is heuristic but sensible, and the ablation shows it beats cheaper ranking criteria.\n\nThe soft spots are real but not disqualifying. The headline 99.3% retention at 70B RULER-32K/20x is a single run with no error bars; the margin over baselines is large, but the absolute retention margin could shrink with seeds. The code is promised only on acceptance, so nothing is independently reproducible yet. The anchor budget k was selected on a RULER subset, which is also the main benchmark—mild tuning risk. And the deeper concern: Theorem 1 bounds key-induced error by a worst-case maximum over positions, and the authors explicitly say a fixed residual budget cannot control that maximum. At 20x, only 17.3% of token-sides get residuals, so fidelity for the other 82.7% depends entirely on anchor alignment. They report median alignments, not worst case. The 'retains every token' claim is true in the sense that nothing leaves the softmax, but it doesn't guarantee every token is faithfully represented. The benchmarks suggest it works in practice on tested distributions; the transfer to arbitrary contexts is not established.\n\nOverall: this deserves a serious referee. I'd send it to review with requests for code, multi-seed runs, and a worst-case alignment analysis (e.g., distribution of per-token reconstruction error or max attention-weighted error). It's a solid contribution with an honest limitation statement.","headline":"A genuinely new anchor-residual KV compression scheme with strong empirical evidence, but the 'retains every token' claim rests on an unverified worst-case alignment premise that the authors themselves acknowledge.","tokens_in":39722,"tokens_out":2408,"would_cite":true,"duration_ms":22134,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a KV cache can be compressed 20x without removing any token, retaining 93-99% of uncompressed accuracy across three model scales.","keywords":["KV cache compression","anchor-residual representation","long-context LLM inference","attention-output-aware allocation","token retention","quantized residuals","RULER","LongBench"],"falsifier":"A concrete test: on a 128K-token prompt, find the key whose cosine similarity to its nearest anchor is lowest and check whether that position receives nonzero attention; if such a worst-case key can shift the softmax and the residual budget does not cover it, the 20x accuracy on that prompt should drop measurably. Measuring $\\max_t \\lVert K_t - \\hat K_t\\rVert$ across layers and comparing it to the bound of Theorem 1 would show directly whether the guarantee is tight.","tokens_in":38526,"feed_emoji":"🗜️","tokens_out":8210,"duration_ms":67960,"temperature":0.7,"pith_summary":"AnchorKV claims that the key-value cache of a large language model can be compressed 20x while keeping every token in the attention computation. The cache is recast as a small set of exact anchor vectors plus, for each other token, a scaled projection onto its nearest anchor, with quantized residuals spent only where reconstruction error would most change the attention output. Across three model scales and three long-context benchmarks, the method retains 93-99% of the full-cache score at 20x compression, and at 20x it scores higher than every eviction baseline at 10x. A sympathetic reader would care because this breaks the trade-off eviction methods face: the full context stays reachable, so later queries can still use tokens that prefill-time scoring did not flag.","feed_headline":"20x KV-cache compression without discarding any token","feed_subtitle":"AnchorKV keeps every token in the softmax and beats eviction baselines at half the memory, gaining most at 70B scale.","key_machinery":"The load-bearing object is the anchor-residual representation: a token $x_i$ is stored as an anchor index $a(i)$, a scalar coefficient $\\gamma_i = \\langle x_i, x_{a(i)}\\rangle/\\lVert x_{a(i)}\\rVert^2$, and optionally a two-bit quantized residual $\\tilde{r}_i$, so reconstruction is $\\hat{x}_i = \\gamma_i x_{a(i)} + \\tilde{r}_i$. Anchor selection combines the SnapKV observation-window score with uniform sampling, and residual placement is driven by a first-order estimate of attention-output error that separates key-induced and value-induced channels. All of this is wrapped in a per-layer byte accounting that fixes the retained fraction $\\theta$ as the only user knob, and the supporting theorem bounds output error by an attention-weighted value term plus $2 V_{\\max}\\tanh(\\mu)$ with $\\mu = \\lVert q\\rVert \\Delta_K/\\sqrt{D}$, so the compressed cache is a perturbation of the exact one rather than a truncation.","core_discovery":"The paper's central claim is that aggressive KV-cache compression and full-context availability are not in conflict: AnchorKV shrinks the cache 20x while keeping every token inside the softmax. Each KV head stores a small set of anchors exactly and expresses every other token as a scaled copy of its nearest anchor; a byte budget then buys two-bit residuals for the tokens whose projection error would most perturb the attention output. On Llama-3.1-70B this retains 99.3% of the uncompressed RULER score at 20x against 86.8% for the strongest eviction baseline, and at 20x AnchorKV outscores every eviction baseline at 10x on all nine model-benchmark settings. The same pattern holds on LongBench real documents, with 94.1% retention at 8B and 98.4% at 70B.","pith_inferences":["Because anchor selection already mixes attention scoring with uniform sampling, one could replace uniform anchors with a coverage-optimized dictionary, such as k-means or furthest-point sampling, and likely improve the worst-case key-error term that Theorem 1 leaves uncontrolled.","The attention-output utility of Eq. 6 is computed once at prefill; a testable extension is to re-score residuals online as decoding queries arrive, since the assumption that prefill queries anticipate decode queries is the same one eviction methods make.","If the method is combined with quantization of anchors or coefficients, the 20x figure is not a floor; the anchor-residual representation could push further before accuracy falls.","The key-error bound suggests a diagnostic: monitoring the maximum key reconstruction error per layer, rather than the median cosine, would flag prompts where the no-token-removed guarantee is at risk."],"forward_implications":["At the same byte budget, 20x AnchorKV scores above every eviction baseline at 10x, so the same accuracy is reached with half the cache memory.","At Llama-3.1-70B the method retains 99.3% of the uncompressed RULER score at 20x, and retention improves with scale, the regime where the cache is most expensive.","Because no position is removed from the softmax, the compressed cache behaves as a perturbation of the exact cache; tasks requiring retrieval against distractors and whole-context aggregation retain 60-98% of the full score while eviction baselines fall below 16%.","Decode peak memory falls about 19x at a 20x target, and the concurrency ceiling roughly doubles, so serving capacity improves in addition to accuracy.","The compression runs once at the end of prefill on a frozen model, so no training or calibration corpus is needed and the method is a drop-in inference-time change."],"supporting_citations":[{"why":"Supplies the observation-window scoring (SnapKV) used to select anchors from recent queries.","marker":"Li et al. 2024"},{"why":"AdaKV is the head-adaptive eviction baseline that AnchorKV must beat at matched byte budgets.","marker":"Feng et al. 2025"},{"why":"PyramidKV is the layer-adaptive eviction baseline used in the matched-budget comparison.","marker":"Cai et al. 2024"},{"why":"TurboQuant is the every-token quantization baseline whose fixed bit width caps compression near 5x, defining the regime AnchorKV extends.","marker":"Zandieh et al. 2025"},{"why":"RULER is the synthetic long-context benchmark that supplies the main retention numbers, including 99.3% at 70B and 20x.","marker":"Hsieh et al. 2024"},{"why":"LongBench supplies the real-document long-context tasks used for matched-budget accuracy comparisons.","marker":"Bai et al. 2024"},{"why":"Provides the tiled execution pattern for the decode kernel that reconstructs keys and values without materializing the dense cache.","marker":"Dao 2024"},{"why":"Provides the randomized Hadamard rotation used before residual quantization to spread energy and reduce outlier sensitivity.","marker":"Ashkboos et al. 2024"}],"fun_headline_variants":["20x KV cache shrink without dropping a token","AnchorKV: 20x cache compression, every token kept","All tokens stay, cache shrinks 20x","20x compression, zero evictions, 99% accuracy","KV cache 20x smaller, every token intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's error bound is governed by the worst key reconstruction over all positions, and a fixed residual budget cannot repair a worst-case key, so the whole accuracy story rests on the empirical premise that every attended key projects closely enough to some anchor.","fun_headline_variants_meta":{"raw":{"variants":["20x KV cache shrink without dropping a token","AnchorKV: 20x cache compression, every token kept","All tokens stay, cache shrinks 20x","20x compression, zero evictions, 99% accuracy","KV cache 20x smaller, every token intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1198,"prompt_tokens":858,"completion_tokens":340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":474,"tokens_out":340,"duration_ms":3667,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:57:30.713724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a 128K-token prompt, find the key whose cosine similarity to its nearest anchor is lowest and check whether that position receives nonzero attention; if such a worst-case key can shift the softmax and the residual budget does not cover it, the 20x accuracy on that prompt should drop measurably. Measuring $\\max_t \\lVert K_t - \\hat K_t\\rVert$ across layers and comparing it to the bound of Theorem 1 would show directly whether the guarantee is tight.","supporting_citations":[{"cited_title":"Kevin Zhou , editor =","cited_arxiv_id":null,"evidence_quote":"AdaKV is the head-adaptive eviction baseline that AnchorKV must beat at matched byte budgets."},{"cited_title":"Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , editor =","cited_arxiv_id":null,"evidence_quote":"Provides the randomized Hadamard rotation used before residual quantization to spread energy and reduce outlier sensitivity."}],"review_version":2}