{"id":"975cb12d-4314-435e-b646-99e30d951d86","arxiv_id":"2607.21604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AgentKVShift corrects all reused KV cache entries in agentic memory by estimating a chunk-level residual offset from a small probe set, recovering near-fresh accuracy at 10–30% recompute ratios.","lead":"This paper introduces AgentKVShift, a training-free method that reuses cached key-value states in LLM agents by estimating a shared memory-level drift from a small set of probe tokens and adjusting all cached tokens. It reports near full-recompute accuracy while refreshing only 10–30% of the cache, giving 2–3.5x prefill speedups on memory-augmented agent workloads.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe KV states cannot be 'fresh' without a full forward pass; Algorithm 1's probe recomputation is underspecified, so claimed speedups and error bounds rest on an unattainable oracle.","rationale":"The reader's weakest assumption concerns probe representativeness across layers and sub-Gaussianity of value residuals. My concern is more basic: the probe states themselves may not be the 'fresh' states used in the theoretical analysis, because obtaining true fresh KVs for a subset of tokens without recomputing all tokens is not possible in a standard transformer. This is load-bearing because the entire correction mechanism estimates the chunk-level offset from probe residuals; if those residuals are biased by stale neighbor states, the correction does not recover full-recompute quality, and the theory in §3.4 does not apply. The paper's empirical F1 tables could still be correct if the actual implementation uses some approximation, but without code or a precise description of probe recomputation, the central claim is not internally consistent. This does not imply fraud or deliberate omission; it is an underspecification that a released implementation or a targeted measurement could resolve. I therefore keep the verdict conditional, with the additional condition that the probe-freshness issue be addressed directly.","tokens_in":23082,"tokens_out":10269,"duration_ms":104493,"concrete_test":"Instrument the implementation (or re-implement Algorithm 1) and, for a fixed memory chunk and several layers, compare two sets of probe KVs: (i) the KV values actually produced by the method's partial recomputation against cached non-probe KVs, and (ii) true full-recompute KVs obtained by running the full forward pass over all tokens. If the mean squared difference between these two sets is comparable to ||μ̂||², the estimated offset does not target the true residual defined in §3.1. Separately, measure end-to-end prefill time with CUDA events or a FLOP counter, including the full layer-1 probe-scoring pass; if the speedup over full recompute falls below 2–3.5× at r=0.1, or if true fresh probe states require recomputing all tokens, the efficiency claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Algorithm 1 (lines 4–8) assumes that, at every layer, one can 'recompute fresh KV' for the probe set S. But the KV of a token at layer ℓ depends on its hidden state at layer ℓ−1, which is produced by attention over all tokens at that layer. To obtain K^ℓ_fresh as defined in §3.1 — the state under full recomputation in the current context — every token's KV at layers 1..ℓ must be recomputed. The only way to keep prefill cost near r of full is to run probe tokens through the network while reusing cached (stale) KV for non-probe tokens; the resulting K^ℓ_j is not K^ℓ_fresh but a state conditioned on stale neighbors. The probe mean μ̂ is then a biased estimator of the chunk-level offset μ in the residual model r_i = μ + ξ_i, and this bias is omitted from the probe-error term E_b in Proposition 2. If, instead, the implementation performs a full forward pass to produce true fresh probe states, the measured 2–3.5× prefill speedups are not attributable to refreshing only 10–30% of the cache. The manuscript never specifies which of these is the case; the F1 tables cannot validate the central claim without this detail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"AgentKVShift proposes a training-free KV cache reuse correction for agentic memory systems. The authors observe that the residual between fresh and cached KV states for a retrieved memory unit is dominated by a chunk-level offset, with smaller token-wise fluctuations. The method selects a small probe set (b = ceil(r n)) based on layer-1 KV divergence, estimates the per-layer mean residual over the probe set, and corrects all non-probe tokens by adding a divergence-weighted mean shift. They report F1/ROUGE/BLEU on LoCoMo and AMA-Bench-Recall with four LLMs, showing within 1.5–6% of full recompute at r=0.1/0.3, prefill speedups of 2–3.5×, and gains over CacheBlend/ProphetKV, including under KV quantization. A theoretical analysis bounds attention error before and after correction under a sub-Gaussian assumption.","tokens_in":23490,"tokens_out":9920,"duration_ms":79524,"significance":"If the implementation is sound, the paper makes a useful empirical contribution: it identifies a structured component in KV reuse error for agentic memory and demonstrates that a simple mean-shift correction substantially improves over token-selection-only reuse. The evaluation is broad (two memory systems, two benchmarks, four models, quantization combinations) and the appendix contains honest diagnostics: mean-explained energy, probe-concentration checks, and explicit limitations. The method is training-free and trivial to implement, which is practically attractive. However, the central correctness of the speedup claim depends on how probe 'fresh' states are computed; the manuscript as written is ambiguous on this point.","major_comments":[{"comment":"The procedure 'recompute fresh KV for S^K_C ∪ S^V_C at layer ℓ' is not implementable as stated without a full forward pass. K^ℓ_fresh is defined under full recomputation in the current context; the KV of a token at layer ℓ depends on its hidden state at layer ℓ−1, which is produced by attention over all preceding tokens. If only probe tokens are run through the network with cached KV for other tokens, the resulting probe state is not K^ℓ_fresh but a stale-neighbor state, making the probe mean a biased estimator of µ. This bias is absent from E_b in Proposition 2. If a full forward pass is performed instead, the 2–3.5× speedups in Table 3 are not attributable to refreshing only 10–30% of the cache. Please specify the exact forward operations used to obtain probe states, and include their cost in the speedup accounting.","section":"§3.3, Algorithm 1 lines 4–8; §3.1"},{"comment":"The theoretical guarantee is proven for an unweighted correction using a uniform random probe set, with the same mean added to all tokens. Algorithm 1 instead uses per-token weights w_{K,j}=min(d_{K,j},1) and selects the top-b divergent probes at layer 1. Proposition 2's bound and Proposition 3's sufficient condition do not cover this weighted, non-random estimator. Thus the statement 'AgentKVShift admits a tighter attention-error bound' is not actually established for the algorithm evaluated in the experiments.","section":"§3.4, Props. 2–3 and App. B.3"},{"comment":"The text states that AgentKVShift 'remains the strongest KV-reuse strategy across all reasoning categories,' but the State Abstraction row of Table 7 shows ProphetKV with F1 0.254 (81.7%) versus 0.235 (75.6%) for AgentKVShift, and LLM-Judge accuracy is tied at 0.118. This contradicts the claim in the Conclusion that AgentKVShift 'outperforms CacheBlend and ProphetKV across all tested configurations.' Please correct the claim or explain the discrepancy.","section":"App. A.2, Table 7"},{"comment":"No error bars, standard deviations, or number of seeds are reported. LoCoMo contains only 10 multi-session conversations, so the headline statement of 'within 1.5–6% relative F1' is based on a single run. Given that some cells differ by only 1–2 F1 points (e.g., the weighted-average gap in Table 2 is 0.012 F1), the central quantitative comparison to full recompute requires an estimate of run-to-run variability.","section":"§4.1, Tables 1–2"}],"minor_comments":[{"comment":"The batch-size labels and values are run together (e.g., 'B=14 8 16B=14 ...'), making the table unreadable. Please reformat so each σ? column is clearly labeled.","section":"Table 3"},{"comment":"CacheBlend on Qwen2.5-3B, AMem, r=0.1 reports F1=0.178 in Table 1 but 0.150 in Table 4 under 'No Quant'. Since both settings are described as LoCoMo/AMem/Qwen2.5-3B at r=0.1, this inconsistency needs to be resolved.","section":"Tables 1 and 4"},{"comment":"Several 'retention ratio' values exceed 100% (e.g., Software F1 104.7% in Table 2). The caption should clarify that this is a ratio to full recompute and that values above 100% are possible due to noise or metric properties.","section":"Tables 2 and 7"},{"comment":"Appendix A.1 says LoCoMo and throughput experiments on 3B models use A100 80GB, but Table 3's caption says A100-40GB for Qwen2.5-3B. Please align the hardware specifications.","section":"Appendix A.1 vs Table 3 caption"},{"comment":"The inset panels with axes labeled '1.0 1.5 2.0' and F1-like values are not described in the caption. Please explain what the insets show and how they were produced.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The probe freshness issue is the main risk. I recommend that the editor ask the authors to provide the exact implementation, including how probe KVs are computed and how prefill cost is measured, before further consideration. If the implementation actually requires a full forward pass for probe states, the speedup numbers are not valid. The paper also contains a clear empirical contradiction in Table 7 that should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has one genuinely new idea—estimate a per-memory KV residual offset from a small probe set and add it (weighted) to every unrefreshed token—and it backs the idea with enough experiments that it deserves a real referee. But there is a load-bearing ambiguity in how the probe KV is 'recomputed,' and the theoretical analysis doesn't cover the actual weighted correction. Both are fixable, but they are not cosmetic.\n\nWhat's new: prior training-free reuse methods (CacheBlend, ProphetKV) decide which tokens to recompute and leave the rest stale. AgentKVShift instead corrects the stale tokens using a chunk-level offset. The spectral analysis in §3.2 provides evidence that the residual is dominated by a shared offset, and the correction is simple and training-free. Experiments span 3B–32B models, two memory systems, two benchmarks, and include quantization; the gains over baselines are consistent and sometimes large. The appendix diagnostics on sub-Gaussian behavior are honest—they show K fits well and V only at deeper layers.\n\nSoft spots:\n1. Algorithm 1 says 'recompute fresh KV' for the probe set at every layer, but a probe token's layer-ℓ KV depends on its layer-ℓ−1 hidden state, which in turn attends to all tokens. The paper never says whether probes are computed through a full forward pass (which would erase the 2–3.5× speedup) or through attention that reuses stale KV for non-probe tokens (which makes the probe states not truly fresh and biases the offset estimate). The error bound in Proposition 2 omits that bias. This is the central question for the speedup/accuracy tradeoff, and the current text does not answer it.\n2. The theory in B.3 analyzes an unweighted uniform correction, while the algorithm applies weights min(divergence,1). The paper acknowledges this but doesn't show the weighted case is covered.\n3. Reproducibility: no code, no seeds, no error bars. The layer-1 probe scoring cost is not included in the recompute-ratio accounting, though that cost is probably small.\n\nNone of these look fatal. The empirical pattern is consistent, and the idea is strong enough that a serious referee should engage. But the authors need to clarify the probe computation and either adjust the theory or report the bias-corrected measurements.\n\nRecommendation: send to peer review, conditional on the authors specifying the implementation and rerunning or re-deriving the key claims. I'd bring it to a reading group if the discussion is about inference efficiency, not as a settled result.","headline":"A genuine mean-shift correction for KV reuse with solid empirical support, but the probe 'fresh' computation is underspecified and the theory doesn't match the weighted algorithm; both need fixing before the speedup claims are credible.","tokens_in":23896,"tokens_out":4453,"would_cite":true,"duration_ms":41879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KV cache reuse errors in agentic memory reduce to a single shared chunk-level offset, and correcting it from a handful of probe tokens recovers near-fresh attention.","keywords":["KV cache reuse","agentic memory","probe-guided correction","residual decomposition","LLM prefill","attention error bound","KV quantization","memory-augmented agents"],"falsifier":"Construct or find a retrieved memory chunk where the dominant residual offset rotates strongly across layers, for example a chunk whose summary tokens are semantically stable early but heavily re-contextualized deeper in the network; if layer-1 probes miss the deep-layer offset, corrected attention error would fail to decay as the probe budget grows or would stay above the uncorrected bound. A direct measurement of ∥μ̂_b − μ∥ across layers for both K and V, checking whether it decays at the b^-1/2 sample-mean rate for values in shallow layers, would settle the core assumption.","tokens_in":23029,"feed_emoji":"⚡","tokens_out":4740,"duration_ms":39505,"temperature":0.7,"pith_summary":"AgentKVShift claims that when a remembered text chunk is retrieved under a new context, the stale key-value states differ from fresh ones mostly by a common offset shared across the entire chunk, plus small per-token noise. Estimating that offset from a tiny probe set, whose tokens are actually recomputed, and adding the estimate to every other token's cached states, recovers most of the quality of full recomputation. The paper argues this turns the recompute budget into a correction signal for the whole chunk, rather than only refreshing a few tokens. Across several open models and two agentic memory designs, the method stays within 1.5–6% relative F1 of full recompute while refreshing only 10–30% of the cache, with 2–3.5x prefill speedups. This matters because memory-augmented agents pay full re-encoding costs on every retrieval, and existing token-selection reuse methods degrade sharply on structured, metadata-rich memories.","feed_headline":"A single mean-shift fixes stale key-value caches in LLM agents","feed_subtitle":"Estimate the shared memory-level offset from a few tokens, apply it to all reused ones, and cut prefill cost 2–3.5×.","key_machinery":"The load-bearing object is the residual decomposition r_i = μ + ξ_i for key and value states of every token in a retrieved memory chunk, with μ the shared chunk-level offset and ξ_i the token-wise fluctuation. AgentKVShift selects top-b tokens by layer-1 divergence as probes, recomputes their fresh KVs at every layer, estimates μ as the probe mean, and applies a weighted vector addition (K_reuse + w·μ̂) to every non-probe token. This turns the refresh budget into a global correction rather than a per-token decision. The theoretical engine is the deterministic attention-perturbation bound of Lemma 1, combined with Proposition 3, which shows correction wins when the common bias exceeds the pro","core_discovery":"The paper's central claim is that the per-memory KV reuse residual, fresh minus cached, decomposes as R_i = μ + ξ_i, where μ is a shared chunk-level offset and ξ_i are centered, sub-Gaussian-style token-wise fluctuations. A small set of probe tokens whose fresh KVs are recomputed at layer 1 provides an estimator for μ at every layer; subtracting that estimate from all reused tokens removes the common bias and leaves only probe-estimation error plus fluctuations. The theoretical analysis (Propositions 1–3) bounds attention error before and after correction and shows the corrected bound is strictly tighter whenever the common-bias term exceeds the probe-estimation term. Empirically, this singl","pith_inferences":["The same residual-decomposition argument may extend beyond agentic memory: any dynamic retrieval setting where identical text is re-encoded under different contexts, such as RAG with template-heavy metadata, could benefit from probe-guided mean correction.","Because probes are fixed at layer 1 but offsets are re-estimated per layer, a natural extension is layer-dependent probe selection or asymmetric K/V probe budgets; the paper's own diagnostics show values at shallow layers deviate from the sub-Gaussian assumption, so a value-specific correction scheme might close the remaining gap.","The cross-chunk reasoning gap reported in the appendix suggests memory-local correction has a ceiling; a state-aware variant that conditions the offset on the agent's current reasoning state could address capabilities like state updating and causal inference.","The theory predicts the corrected bound tightens as probe budget b grows; a direct experiment sweeping b and measuring attention error against the b^-1/2 prediction would give a sharper test of the sub-Gaussian core assumption."],"forward_implications":["Near-full accuracy at 10% refresh: on a long-horizon dialogue benchmark, relative F1 drop to full recompute stays within 1.5–6% while refreshing only 10% of tokens.","Fivefold lower recompute: AgentKVShift reaches quality that prior token-selection reuse methods achieve only at 45–55% refresh, using roughly 10% instead.","Prefill speedups of 2–3.5x over full recompute on a single GPU in the 10–30% refresh regime, growing with context length and batch size.","Composition with KV quantization: the probe estimate absorbs average quantization error, retaining over 2x the F1 of prior reuse methods at aggressive 2- and 4-bit settings.","Generality: the correction transfers across model scales from 3B to 32B parameters and across both note-based and graph-based agentic memory systems."],"fun_headline_variants":["One offset corrects all reused KV tokens in agent memory","Probe-guided offset: KV reuse with near-perfect accuracy","Refresh only 10-30% and match full recompute","AgentKVShift: cheap KV reuse, 2-3.5x prefill speedup","Mean-shift correction turns KV reuse into full quality"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method stands or falls on the premise that probe tokens selected by layer-1 divergence remain representative of the chunk-level residual offset at every later layer, and that token-wise fluctuations are light-tailed enough for the probe mean to concentrate around the true offset.","fun_headline_variants_meta":{"raw":{"variants":["One offset corrects all reused KV tokens in agent memory","Probe-guided offset: KV reuse with near-perfect accuracy","Refresh only 10-30% and match full recompute","AgentKVShift: cheap KV reuse, 2-3.5x prefill speedup","Mean-shift correction turns KV reuse into full quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2782,"prompt_tokens":910,"completion_tokens":1872,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":1782}},"tokens_in":654,"tokens_out":1872,"duration_ms":12284,"temperature":1.0,"reasoning_tokens":1782,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:51:11.944644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or find a retrieved memory chunk where the dominant residual offset rotates strongly across layers, for example a chunk whose summary tokens are semantically stable early but heavily re-contextualized deeper in the network; if layer-1 probes miss the deep-layer offset, corrected attention error would fail to decay as the probe budget grows or would stay above the uncorrected bound. A direct measurement of ∥μ̂_b − μ∥ across layers for both K and V, checking whether it decays at the b^-1/2 sample-mean rate for values in shallow layers, would settle the core assumption.","supporting_citations":[],"review_version":1}