{"id":"57eb1b18-52fc-456d-b87a-68450367c6bc","arxiv_id":"2507.19906","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.","lead":"CaliDrop improves token eviction for KV caches by offloading evicted tokens and using a stored query to approximate their attention contribution, choosing between calibration and recomputation based on cosine similarity. It reports consistent accuracy gains over H2O, SnapKV, and StreamingLLM on LongBench, RULER, and Needle-in-a-Haystack, particularly at small KV budgets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on cosine similarity between stored and current queries being a reliable predictor of calibration benefit, yet the paper's evidence is one illustrative layer/head and thresholds tuned on the same benchmark used for headline results.","rationale":"This is the load-bearing assumption because the entire mechanism is a substitution of historical attention output for current output. The decomposition theorem is exact, but the substitution is not; the cosine gate is the only protection. The paper's evidence for the gate is a single illustrative heatmap and an L1 plot from an artificial prefill-only setup; it never reports per-head, per-layer calibration error at decode time. This is a correctness risk, not merely a disagreement with consensus, and a clean diagnostic would settle it. I agree with the reader's weakest-assumption identification. Secondary issues, such as offloaded KV meaning this is not pure compression and Algorithm 2's Q_{−1} typo, are real but do not change the verdict; they are already captured in the reader's conditional assessment. Since the concern is testable and the paper provides enough experimental scaffolding to address it, keeping the CONDITIONAL verdict is appropriate.","tokens_in":26267,"tokens_out":6307,"duration_ms":80227,"concrete_test":"Instrument LLaMA-3-8B-Instruct with SnapKV at a 128-token budget and, for every head, layer, and decode step, compute (a) the true attention output over evicted tokens using the full KV oracle, (b) CaliDrop's stored historical output, and (c) the cosine similarity between stored and current queries. Then plot the change in L1 error from applying calibration against cosine similarity. If a substantial fraction of steps with ρ>0.85 increase L1 error, or if error reduction is non-monotone in ρ on held-out tasks (e.g., RULER with thresholds fixed from LongBench), the cosine gate is not a reliable predictor and the headline gains are not transferable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation 1 is exact only when the same query is used for both subsets: the true α_j requires current-query softmax weights over evicted tokens. CaliDrop substitutes Cweight computed from the stored historical query, and the entire safeguard against a bad substitution is the cosine gate θ1/θ2. The paper's preliminary justification is thin: §3.1.1 shows one layer/head of one model on LongBench, and §3.1.3 tests only the first token after prefill with a historical query drawn from the preceding 0-9 tokens, not the actual decode-time regime where the cached query can be ~8 steps old (Figure 3b) and the evicted set is fixed. Because softmax(QK^T) is sensitive to small query differences, high raw-vector cosine similarity does not guarantee that the historical softmax distribution over evicted keys is close to the current one; when it is not, calibration injects a wrong attention output rather than correcting. θ1=0.7 and θ2=0.85 were selected on LongBench (Table 2) and the headline results are also LongBench, so the evidence does not establish that the gate transfers across tasks, layers, or heads. If the gate misfires on even a modest fraction of steps, CaliDrop's 'significant improvement' can become a degradation in other settings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CaliDrop, an add-on to token-eviction KV cache compression. The method offloads evicted KV pairs, precomputes during prefill an attention output over those evicted tokens using a historical query, and during decoding uses cosine similarity between the current and historical query to decide whether to reuse the precomputed calibration (θ2), recompute with the current query (θ1), or skip. The decomposition in Eq. (1) is exact for a fixed query; the approximation arises from substituting the historical query's softmax statistics for the current query's. Experiments on LongBench, RULER, and Needle-in-a-Haystack with Mistral-7B, LLaMA-3-8B, and LLaMA-3-70B show consistent accuracy gains over StreamingLLM, H2O, and SnapKV at several KV budgets, with modest throughput cost and a short analysis of threshold sensitivity.","tokens_in":26430,"tokens_out":6988,"duration_ms":82201,"significance":"If the accuracy gains are robust, CaliDrop is a useful, simple plug-in for existing eviction methods: it does not change the eviction policy, only adds an offloaded residual computation and a similarity gate. The paper's main strengths are its breadth (three models, three benchmarks, four KV sizes) and the clean statement of the exact attention decomposition. The main risk is that the core premise—that historical attention outputs approximate current ones well enough for calibration—is validated only narrowly in the preliminary section, and the two thresholds are selected on the same LongBench benchmark used for the headline results. The held-out RULER and NIAH results are encouraging but do not fully resolve the sensitivity question.","major_comments":[{"comment":"The central approximation is load-bearing and its empirical justification is too narrow. Equation (1) is exact only when the same query is used for both subsets, but Algorithm 2 replaces the current-query softmax denominator over the evicted set with Cweight computed from the stored historical query (Algorithm 1, lines 6–8). The only safeguard against a bad substitution is the cosine gate in Algorithm 2, yet the preliminary evidence in §3.1.1 is a single layer/head heatmap (Figure 1, top-right) and §3.1.3 tests only the first token after prefill with a historical query 0–9 positions old (Figure 1, bottom-right). In the actual decode regime the stored query can be about eight steps old (Figure 3b), and raw cosine similarity does not control the divergence between softmax distributions. Please provide per-layer/per-head statistics at realistic decode lags, e.g., the relationship between cosine similarity and the relative error of the calibrated attention output, and report the fraction of decode steps where calibration reduces rather than increases the attention-output error.","section":"§3.1.3, Figure 1, Algorithm 2"},{"comment":"The two thresholds are selected on the same benchmark used for the headline results. θ1 = 0.7 and θ2 = 0.85 are chosen from Table 2, which reports LongBench accuracy for LLaMA-3-8B-Instruct with SnapKV and KV size 128; the main results in Table 1 include exactly this setting, so the LongBench gains are partly in-sample. Please either select thresholds on a development split not used in the main tables, or report a sensitivity analysis of θ1/θ2 on RULER and NIAH to show that the chosen values transfer. The held-out RULER and NIAH results with the fixed thresholds are encouraging, but they do not by themselves quantify the sensitivity of the method to the gate.","section":"§4.1.3, Table 2"}],"minor_comments":[{"comment":"In the recomputation branch, the line 'Cweight = Σ exp(Q_{−1} K^T_evict / sqrt(dk))' should use the current query Qt, not Q_{−1}; as written, recomputation does not update the softmax denominator with the current query.","section":"Algorithm 2, line 5"},{"comment":"No variance or number of runs is reported for any of the accuracy numbers; since the abstract uses 'significantly improves', please add standard deviations or at least state whether each number is a single run.","section":"§4.2, Tables 1–4"},{"comment":"The paper is framed as 'KV Cache Compression', but CaliDrop offloads the evicted KV pairs rather than discarding them; the memory savings are for the active GPU cache only, while total storage is not reduced. Please clarify this framing in the abstract and introduction to avoid overclaiming.","section":"Abstract and §1"},{"comment":"It would be helpful to also report the distribution of cosine similarities across layers and heads in the decode regime, since the thresholds θ1 and θ2 are the only mechanism that prevents harmful calibration.","section":"Figure 3b"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The paper is a reasonable empirical contribution, but the core similarity-based approximation needs stronger validation, and the threshold-selection procedure should be made cleaner with a validation split or sensitivity analysis on held-out benchmarks. The offloading-versus-compression framing should be corrected, though that alone would not change my recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this is a solid, incremental KV cache compression paper with a genuinely new mechanism — stale-query calibration gated by cosine similarity — and the experiments are more thorough than most in this subfield. The headline claim, that CaliDrop recovers a lot of the accuracy lost by token eviction at small budgets, holds up on LongBench, RULER, and Needle-in-a-Haystack across three model sizes. It deserves a serious referee.\n\nWhat's new: instead of just discarding evicted tokens, CaliDrop offloads them, precomputes an attention output using a historical query during prefill, and then during decoding decides — based on cosine similarity — whether to trust that stale output, recompute with the current query, or sit tight. The attention decomposition in Eq. 1 is an algebraic identity, so no one is pretending that's a theorem; the contribution is using stored attention statistics for evicted tokens as a correction term. I don't see that in the cited prior work; RecycledAttention recomputes periodically, which is different.\n\nWhat's solid: the experimental coverage is genuinely broad — three models, three benchmarks, four KV budgets, and consistent gains, largest at small budgets. The throughput analysis is honest: they show CaliDrop adds modest overhead, and the limitations section says outright that this is offloading, not compression, and that gains shrink at larger budgets. That kind of candor is rare.\n\nWhere it's soft: the cosine-similarity gate is the load-bearing assumption, and the evidence for it is thin. The preliminary study shows one layer/head of one model, and the Q2 test uses a historical query from only 0-9 steps back, not the actual decode regime where the stored query can be ~8 steps old. The thresholds θ1=0.7, θ2=0.85 are selected on LongBench (Table 2) and the headline numbers come from the same benchmark — so that part is partly optimized on the test set. The cross-benchmark gains on RULER and NIAH with the same thresholds soften this criticism, but the gate's reliability per layer and head is not examined. No variance or error bars are reported, and no code or data is released, which limits reproducibility. Algorithm 2 also has a Q_{-1} typo in the recomputation path — should be Q_t; minor, but confusing.\n\nOverall: the central empirical claim is believable, and the weaknesses are real but not fatal. I'd send this to peer review with a request for variance bars, code, and a more realistic probe of the similarity assumption. The reader gets a fair deal from this paper.","headline":"Solid incremental KV compression paper with a genuinely new stale-query calibration mechanism; thoroughly tested, but the similarity gate is thinly evidenced and thresholds are tuned on the headline benchmark.","tokens_in":27071,"tokens_out":3735,"would_cite":false,"duration_ms":38921,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CaliDrop recovers much of the accuracy lost when a KV cache is compressed by token eviction, by reusing a stored historical query as a calibration signal.","keywords":["KV cache compression","token eviction","attention calibration","query similarity","attention decomposition","long-context inference","speculative attention","CaliDrop"],"falsifier":"Run CaliDrop on a controlled task in which adjacent-token queries are deliberately dissimilar, for example a long document with randomly interleaved unrelated passages, and compare accuracy against the underlying eviction method without calibration. If accuracy still improves despite low cosine similarity, the similarity premise is not the active mechanism; if accuracy degrades below the baseline, the premise is confirmed as load-bearing. A more direct measurement is to compute the median adjacent-query cosine similarity per layer and head on several benchmarks and check whether the layers with similarity below 0.7 are the same layers where calibration hurts.","tokens_in":25975,"feed_emoji":"🧠","tokens_out":8615,"duration_ms":93511,"temperature":0.7,"pith_summary":"Token eviction is a common way to shrink the key-value (KV) cache that lets large language models generate quickly, but throwing away tokens costs accuracy. CaliDrop proposes to soften that loss: evicted key-value pairs are offloaded rather than deleted, their attention contribution is precomputed for a stored historical query, and during decoding that stored contribution is added back as a calibration term whenever the current query resembles the historical one. The paper claims this recovers a large share of the accuracy that eviction removes, with improvements across StreamingLLM, H2O, and SnapKV, especially at high compression ratios like a 64-token cache. If the claim holds, a serving system can keep only a small KV cache in fast memory and still preserve much of the model's quality, making long-context inference far cheaper.","feed_headline":"CaliDrop restores accuracy lost by KV-cache token eviction","feed_subtitle":"By reusing a stored historical query as a calibration signal, the method recovers much of the quality that aggressive compression removes.","key_machinery":"The load-bearing object is a small calibration record kept for the evicted tokens: a historical query, the softmax denominator over the offloaded keys, and the attention output the historical query would have produced against them. The attention decomposition theorem justifies treating the retained-cache attention and the evicted-cache contribution as additive pieces of the full attention, with weights $\\alpha_i$ and $\\alpha_j$ equal to each subset's share of the total exponential score sum. The similarity gate then decides when the stored contribution is still trustworthy: below $\\theta_1=0.7$ the evicted cache is reloaded and the record is recomputed with the current query; above $\\theta_2=0.85$ the stored record is added directly; in between, calibration is skipped. Together these pieces convert a permanently lossy eviction into an occasionally refreshed approximation.","core_discovery":"The central claim is that eviction does not have to be a permanent loss: the contribution of evicted tokens can be approximated and then reinstated. CaliDrop splits the full KV cache into the retained subset and the evicted subset, offloads the evicted one, and uses the attention decomposition theorem, $\\operatorname{Att}(Q,K,V) = \\alpha_i \\operatorname{Att}(Q,K_{S_i},V_{S_i}) + \\alpha_j \\operatorname{Att}(Q,K_{S_j},V_{S_j})$, to compute the two parts separately. During the prefill phase it stores, for the evicted subset, the last query $Q_{-1}$, the softmax denominator $\\sum_t \\exp(Q_{-1} K_t^\\top/\\sqrt{d_k})$, and the attention output $\\operatorname{Att}(Q_{-1}, K_{\\mathrm{evict}}, V_{\\mathrm{evict}})$. During decoding, the cosine similarity between the current query and $Q_{-1}$ decides whether to reuse that stored output as a correction ($\\rho > \\theta_2$), recompute it by reloading the offloaded cache ($\\rho < \\theta_1$), or do nothing in between. The paper's empirical foundation is the observation, demonstrated on LongBench with LLaMA-3-8B, that queries at nearby positions have high cosine similarity and that this calibration reduces the L1 error of the approximated attention output versus the true one.","pith_inferences":["Beyond the paper's experiments, the same calibration record could be maintained during decoding-phase eviction, not just prefill eviction, which would address the long-generation regime the paper leaves to future work.","The global thresholds $\\theta_1$ and $\\theta_2$ could be made per-layer or per-head, since the paper's own recomputation-frequency analysis shows query similarity varies across layers; adaptive thresholds would let high-similarity heads reuse stored calibrations longer.","The offloaded cache is a natural target for quantization or low-rank projection, so CaliDrop's offload-and-approximate idea should compose with other KV compression techniques rather than compete with them.","More generally, CaliDrop suggests a design principle for cache compression: any eviction policy defines an error term, and if a cheap transformation of historical states can approximate that term, eviction need not be a permanent loss; the same pattern could apply to prefix caches or cross-attention states."],"forward_implications":["A serving engine can keep only a small KV cache in fast memory, offload the evicted tokens, and still recover most of the accuracy that eviction removes, which matters most when the retained cache is as small as 64 tokens.","Because CaliDrop is a wrapper, any future token-eviction policy can inherit the calibration layer without changing its token-selection rule.","The benefit is largest when compression is aggressive and shrinks as the retained cache grows, so CaliDrop is a tool for high-compression regimes, not a replacement for a large cache.","With $\\theta_1=0.7$, recomputation happens roughly every eight decoding steps, so the accuracy gain comes with only a modest throughput penalty compared with the eviction baseline."],"supporting_citations":[{"why":"Supplies the SnapKV eviction baseline and the prefilling compression setting in which CaliDrop is evaluated.","marker":"[20]"},{"why":"Supplies the H2O baseline, an eviction method that scores token importance by accumulated attention.","marker":"[43]"},{"why":"Supplies the StreamingLLM baseline, which evicts to attention sinks plus a sliding window.","marker":"[35]"},{"why":"Provides the LongBench benchmark used for the preliminary query-similarity experiments and the main accuracy tables.","marker":"[3]"},{"why":"Provides the LLaMA-3 model family on which most experiments and preliminary studies run.","marker":"[10]"},{"why":"Provides the Mistral-7B model used as a second evaluation target.","marker":"[17]"},{"why":"Provides the RULER long-context benchmark where CaliDrop shows consistent gains across KV sizes.","marker":"[16]"},{"why":"Provides the Needle-in-a-Haystack benchmark where CaliDrop produces large accuracy jumps, for example H2O rising from 48.5% to 83.8% at a 128-token cache.","marker":"[23]"}],"fun_headline_variants":["Calibrated token eviction restores lost accuracy","Reusing old queries makes eviction nearly lossless","CaliDrop: calibrate eviction to keep quality","Eviction calibration recovers attention accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the empirical premise that nearby queries look alike, so the stored historical query's attention output over evicted tokens is a trustworthy stand-in for the current query's missing attention; where that similarity is low, the calibration term can add harmful noise instead of a correction.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated token eviction restores lost accuracy","Reusing old queries makes eviction nearly lossless","CaliDrop: calibrate eviction to keep quality","Eviction calibration recovers attention accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2087,"prompt_tokens":1046,"completion_tokens":1041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":980}},"tokens_in":662,"tokens_out":1041,"duration_ms":9975,"temperature":1.0,"reasoning_tokens":980,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:51:10.982055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CaliDrop on a controlled task in which adjacent-token queries are deliberately dissimilar, for example a long document with randomly interleaved unrelated passages, and compare accuracy against the underlying eviction method without calibration. If accuracy still improves despite low cosine similarity, the similarity premise is not the active mechanism; if accuracy degrades below the baseline, the premise is confirmed as load-bearing. A more direct measurement is to compute the median adjacent-query cosine similarity per layer and head on several benchmarks and check whether the layers with similarity below 0.7 are the same layers where calibration hurts.","supporting_citations":[{"cited_title":"Barrett, Zhangyang Wang, and Beidi Chen","cited_arxiv_id":null,"evidence_quote":"Supplies the H2O baseline, an eviction method that scores token importance by accumulated attention."},{"cited_title":"Efficient streaming language models with attention sinks","cited_arxiv_id":null,"evidence_quote":"Supplies the StreamingLLM baseline, which evicts to attention sinks plus a sliding window."},{"cited_title":"Longbench: A bilingual, multitask benchmark for long context understanding","cited_arxiv_id":null,"evidence_quote":"Provides the LongBench benchmark used for the preliminary query-similarity experiments and the main accuracy tables."}],"review_version":1}