Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Practical Online KV Cache Compaction for LLM Agents: An Empirical Study

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that online KV-cache compaction for LLM agents should be delayed until the agent's own future queries are available, that simple token eviction is often more robust than attention matching under imperfect proxies, and

desk verdict Useful, credible findings on proxy-query choice for online KV compaction, but the Gemma throughput simulation is internally inconsistent and needs fixing before the 'across models' claim stands. read the letter →

arxiv 2608.00902 v1 pith:LWJPFAZE submitted 2026-08-02 cs.CL

Yujian Liu , Jiabao Ji , Li An , Rohit Jain , Gungor Polatkan , Siyu Zhu , Shiyu Chang This is my paper · ORCID
classification cs.CL
keywords onlineKVcachecompactionLLMagentstokenevictionattentionmatchingproxyquerieslong-horizonagenttrajectoriesmemoryservingthroughput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM agents keep growing a key-value (KV) cache as they reason and call tools, and that cache becomes the main memory bottleneck of long trajectories. This paper argues that compressing the cache inside a live agent loop is a different problem from compressing a static prompt: the queries that will later read a turn do not yet exist when the turn ends. It shows empirically that compacting immediately often degrades task accuracy, while deferring compaction until the agent has generated its next turn and using those real query vectors as a guide recovers most of the gap. Across model scales, simple token eviction guided by such delayed proxy queries preserves most no-compaction accuracy while cutting the KV cache to 20% of its size and raising simulated serving throughput by 1.5-4.2x. The paper's central claim is that proxy-query selection, not the sophistication of the compactor, is the decisive design choice for online KV compaction.

What carries the argument

The load-bearing object is the proxy query: a set of query vectors that stand in for how the compacted cache will later be read. Both compaction families rank or fit against these queries, where token eviction selects the top-m positions by root-mean-square attention mass and attention matching adds an optimized bias and compacted values to match full-cache attention outputs. The decisive mechanism in the online setting is the delay window: keeping a turn uncompressed for k future turns harvests the agent's own subsequent query vectors as proxies, and the empirical claim is that these real future queries are a far better signal than any current-turn substitute.

What would settle it

Run the recommended one-turn-delayed token eviction at a 0.2 compaction ratio on an agent benchmark whose future turns are driven by state changes—say a code-editing task where a later turn must read an earlier file diff or execution log—and compare final task success against no compaction. If the accuracy gap exceeds the few-point range seen on the search benchmarks, or if the agent's later queries consistently attend to tokens the one-turn delay could not have seen, then delayed future-generation proxies are not a universal fix.

Watch

Extended reading notes

Core claim

On two agentic search benchmarks, BrowseComp-Plus and WideSearch, the paper studies two compaction families adapted to run online: token eviction (TE), which scores cache positions by the attention mass they receive from proxy queries and keeps the top-scoring original keys and values, and attention matching (AM), which additionally fits a bias and reconstructed values to mimic full-cache attention. Comparing current-turn proxies (boundary tokens, repeat-prefill) against one-turn-delayed future-generation queries, the paper finds that immediate compaction hurts performance, whereas delaying compaction so that the agent's own subsequent assistant-generation queries serve as proxies consistent

Load-bearing premise

The load-bearing premise is that the structure of future relevance in BrowseComp-Plus and WideSearch—long-horizon search with repeated retrieval of evidence—represents agent workloads generally; the paper's own Limitations section notes that code editing, GUI control, and state-changing web tasks may have different future-relevance structure.

Editorial extensions

If this is right

  • Immediate compaction should be avoided in agent serving: current-turn proxies such as boundary tokens or repeat-prefill consistently underperform a one-turn delay on the studied benchmarks.
  • Token eviction is the practical default: its simpler selection-only design matches or beats attention matching under imperfect proxy queries across delays, budgets, and model scales.
  • An 80% KV-cache reduction (compaction ratio 0.2) preserves most task accuracy on agentic search and, because peak KV falls, supports larger serving batches and 1.5-4.2x throughput.
  • Compaction changes agent behavior, not just the cache: compacted agents take more turns and issue more similar search queries, so evaluations should include behavioral or cost measures beyond final accuracy.
  • Longer delay windows and larger budgets generally help, but the best settings are model- and benchmark-dependent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark representativeness caveat is taken seriously, the headline recipe (one-turn delay, TE, ratio 0.2) is a hypothesis for code-editing, GUI-control, and state-changing web agents rather than a proven recipe; tests in those domains could either confirm or overturn the delay recommendation.
  • The paper's own behavioral evidence suggests a memory-accounting subtlety: 'savings' computed from cache size may be partially spent as extra tool-call turns, so a full cost model should count environment calls and total decoding tokens, not only peak KV.
  • Delayed compaction could be paired with adaptive policies: using memory pressure to choose when to compact and how long to wait, since the paper only tests fixed ratios and windows; such controllers would directly operationalize the proxy-query design tradeoff.
  • The delay-as-proxy idea may transfer to other online context-management decisions such as summarization or retrieval ordering: whenever future reads are endogenous to the model's own trajectory, using a short horizon of real future computation as supervision is likely better than static heuristics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies online KV-cache compaction for LLM agents, where turns must be compacted before future relevance is known. It adapts token eviction (TE) and attention matching (AM) to agent trajectories and compares proxy-query sources—current-turn boundary queries, repeat-prefill, and delayed future-generation queries—on BrowseComp-Plus and WideSearch with Qwen3.5 and Gemma-4 at multiple scales. The main empirical findings are that immediate compaction often hurts, that delayed future-turn proxies recover much of the accuracy gap, that TE is competitive with or more robust than AM under imperfect proxies, and that at a 0.2 compaction ratio TE preserves most accuracy while reducing peak KV and increasing simulated serving throughput.

Significance. If the empirical findings hold, the paper makes a useful contribution by mapping the proxy-query design space for online KV compaction and by showing that delay is a first-order design choice. The paper's strengths include bootstrap confidence intervals for the accuracy comparisons, evaluation across two model families and scales, and the use of realized trajectory shapes for the serving replay. However, the serving-throughput simulation for Gemma-4-31B is internally inconsistent as written, and the no-compaction baseline for that model is not fully defined. Since the abstract generalizes the throughput claim 'across models,' that part is not currently supported for one of the two model families even though the accuracy findings appear more robust.

major comments (4)
  1. [Appendix A.7 / Table 2 / §5.1] The Gemma-4-31B serving simulation is internally inconsistent. Appendix A.7 states the vLLM hybrid KV pool is 489,712 tokens. Table 2 reports peak per-turn KV lengths of 272.1k (no compaction), 99.6k (AM), and 121.3k (TE). With a fixed pool, the maximum admissible batches are floor(489,712/272,100)=1, floor(489,712/99,600)=4, and floor(489,712/121,300)=4, not the 13, 34, and 28 stated in A.7, and not the chosen serving batches 8, 8, and 16 in Table 2. Moreover, §5.1 says some no-compaction Gemma trajectories OOM even at batch size 1, yet the no-compaction row is simulated at batch 8 with peak KV 272.1k, which should fit under the stated pool. As written, the Gemma throughput multipliers (1.7x, 1.5x) cannot be reproduced, and the 'across models' throughput claim in the abstract is unsupported for one of the two model families. Please reconcile the pool definition, batch computation, and O
  2. [§5.1 / Table 2 / Appendix A.7] The no-compaction Gemma baseline is not cleanly defined. The accuracy and turn counts are said to come from batch-size-1 runs, but the text also says that some no-compaction trajectories run out of memory even at batch size 1. If those examples are excluded from the baseline, the reported 48.75% accuracy is computed over a different subset than the compacted runs, making the statement that AM and TE 'slightly exceed the baseline' unsupported. If they were completed by some mechanism, the sentence is misleading and the simulation setup should describe it (e.g., offloading, retry, or a different memory budget). This matters because the Gemma rows are used to argue that compaction is lossless or better at scale.
  3. [§5.1 / Table 2 / Appendix A.7] The reported 'throughput' is decode throughput only; the time and memory cost of performing the compaction itself is not included. This is especially relevant for AM, which fits an attention bias and compact values per layer and per KV head (Section 3.1 and Appendix A.3), and for the repeat-prefill proxy, which requires an extra teacher-forced forward pass. Since the paper's framing is 'practical online KV compaction,' the queries-per-hour comparison should either include the compaction step's latency or be explicitly labeled as decode-only serving throughput. Without this, the TE-vs-AM throughput comparison is incomplete and may overstate the practical gain of the more expensive method.
  4. [§4 / §5 / Limitations] All headline recommendations—delay compaction, use future-generation proxies, prefer TE—are measured only on BrowseComp-Plus and WideSearch, both long-horizon search-and-document-reading tasks. The Limitations section correctly notes that code editing, GUI control, and state-changing web tasks may have a different structure of future relevance. Because the paper's title and abstract make general claims about LLM agents, the conclusions should either be explicitly restricted to information-gathering agents or supported by at least one task family with a different relevance structure. A concrete test would be a multi-step tool-use benchmark (e.g., code editing or state-changing web interaction) at the same 0.2 compaction ratio.
minor comments (5)
  1. [§3.2] The description of repeat-prefill proxies is ambiguous: 'We append a reconstruction prompt after the previous turn's content and teacher-force the model to repeat the completed turn.' Please specify whether the repeated content includes the full previous context or only the immediately preceding turn, and how the reconstruction prompt interacts with the chat template.
  2. [Table 3] Table 3 lacks context: it does not state which model, benchmark, and compaction configuration is used. The text in §5.2 implies a specific setting (likely Qwen3.5-4B on BrowseComp-Plus), but the caption should say so explicitly, and the duplicate threshold of 0.9 should be justified or accompanied by a sensitivity check.
  3. [Appendix A.7] For Qwen, the largest admissible integer batches are 11, 39, and 38, but the simulation uses power-of-two batches 8, 32, and 32. The choice of rounding down to powers of two is reasonable, but should be stated as a modeling choice, and the same convention should be applied consistently to Gemma.
  4. [Table 1] The table layout in the submitted text is hard to read: several cells run together (e.g., the WideSearch F1 and turn-count entries). Please reformat so each column is clearly separated.
  5. [Appendix A.3] The AM regularization weights are set to 10^{-3} 'with spectral scaling by default.' Please provide the exact scaling rule and, ideally, a sensitivity check, since these weights could affect the AM results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical study with externally cited baselines and independent evaluation.

full rationale

This paper is an empirical measurement study, not a derivation. The headline claims (delayed future-generation proxies help; TE is robust; memory savings from a 0.2 ratio; throughput gains) are all evaluated on held-out benchmark trajectories with external judges and independent serving simulators. The proxy queries are experimental interventions, not fitted parameters: delayed compaction waits for actual future turns, records their query vectors, and then evaluates final task accuracy; the accuracy is not recovered from the proxy queries themselves. TE and AM are imported from external cited papers (SnapKV, attention matching) and are not justified by self-citations; no uniqueness theorem or ansatz is smuggled via the authors' own prior work. The '80% KV reduction' is a chosen experimental condition (ratio 0.2), not a predicted output. The throughput simulation replays measured trajectory shapes with SGLang/vLLM latencies; even if the Gemma batch-size numbers are internally inconsistent (Appendix A.7 gives a 489,712-token pool while Table 2 peaks imply much smaller admissible batches), that is a numerical correctness issue, not a definitional reduction, and per the rules cannot be scored as circularity. The Limitations section honestly concedes the two-benchmark scope and fixed-ratio design; acknowledging a scope limit is not a circular step. No load-bearing reduction of a result to its own input was found.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is an empirical paper, so the ledger records the experimental design choices and background assumptions the headline results depend on, rather than fitted constants in a derivation. The two benchmarks, the fixed turn segmentation, the fixed compaction ratio and delay defaults, and the judge model are all premises a reader must accept to generalize the findings.

free parameters (5)
  • Compaction ratio (main operating point) = 0.2
    Chosen as the default memory budget for the headline results; not fitted, but the paper shows performance varies with this ratio and the findings are conditional on it.
  • Delay window (default) = 1 turn
    The paper's main comparisons use a one-turn delay; longer delays generally help, so the quantitative results depend on this choice.
  • AM regularization weight = 1e-3 (with spectral scaling)
    Hand-set for the attention matching optimization; affects AM's fitted bias and values and thus the TE-vs-AM comparison.
  • Gemma-4-E4B system prompt (search floor 20, document-read floor 5)
    Hand-crafted prompt that encourages exhaustive search and differs from Qwen's prompt; this could interact with compaction effects and cross-model comparisons.
  • Generation hyperparameters (temperature, top-p, top-k, presence penalty) = Temp 1.0, top-p 0.95, top-k 20/64, presence penalty 1.5
    Standard decoding settings chosen by hand; the agent behavior and accuracy results are produced under these settings.
assumptions (4)
  • domain assumption Agent trajectories decompose into discrete turns, each containing an assistant generation and a tool response, and compaction of a completed turn freezes it permanently.
    This segmentation is introduced in Section 3.2 and underlies the entire online compaction protocol.
  • domain assumption Proxy queries sampled from the agent's own subsequent turns (with a k-turn delay) are representative of how the compacted turn will actually be read later.
    The delayed-query benefit (Section 4.2) depends on this premise; if post-compaction queries shift, the proxies may be stale.
  • domain assumption The Qwen3.5-397B judge and official templates produce valid correctness scores for both benchmarks.
    All accuracies and F1 scores are judged by this LLM; no human validation or inter-judge agreement is reported.
  • standard math Standard softmax attention formulas in Eqs. (1)-(7) and the optimization steps for TE and AM are correct and applicable to hybrid-attention models.
    The compaction families follow existing methods (SnapKV, Fast KV compaction); the paper adapts them without changing the underlying math.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Practical Online KV Cache Compaction for LLM Agents: An Empirical Study." pith.science (2026). https://pith.science/paper/LWJPFAZE

@misc{pith2026260800902,
  author       = {Pith},
  title        = {Pith review of: Practical Online KV Cache Compaction for LLM Agents: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LWJPFAZE}},
  note         = {Machine review of arXiv:2608.00902}
}
read the original abstract

LLM agents accumulate long trajectories of reasoning steps, tool calls, and environment feedback, making the KV cache a major inference bottleneck. KV cache compaction can reduce this cost, but most prior methods assume a static context where future queries are known or can be approximated offline. Agents instead require online compaction: new information must be compressed before future relevance is known, using proxy queries cheap enough for the inference path. We study online compaction across token eviction (TE) and attention matching (AM), adapting both to compact agent turns and comparing cheap proxy sources such as boundary, repeat-prefill, and delayed future-generation queries. Experiments on BrowseComp-Plus and WideSearch show that immediate compaction often hurts performance, whereas delaying compaction to use the agent's future queries recovers much of the gap. Moreover, TE is often more robust than AM under imperfect proxies. Across models at different scales, TE preserves most of the accuracy while reducing KV cache by 80%, and can improve throughput over the no compaction baseline. These results position proxy-query selection as a core design choice for practical online KV compaction.

Figures

Figures reproduced from arXiv: 2608.00902 by the authors.

Figure 1
Figure 1. Overview of the online KV compaction design space. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Sensitivity to compaction budget and delay window. The top row varies the delay window over [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    You already have all the clues you need; your job is to use the tools to find the answer

    NEVER ask the user for clarification or more information. You already have all the clues you need; your job is to use the tools to find the answer

    work page

  2. [2]

    This is a HARD floor -- the investigation is INCOMPLETE until you reach 20

    SEARCH FLOOR -- You MUST issue AT LEAST 20 distinct`local_knowledge_base_retrieval `calls before you are permitted to output a final answer. This is a HARD floor -- the investigation is INCOMPLETE until you reach 20

    work page

  3. [3]

    Preprint, arXiv:2403.05527

    Gear: An efficient kv cache compression recipe for near-lossless generative inference of llm. Preprint, arXiv:2403.05527. Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. 2025. Kvzip: Query-agnostic kv cache compression with context reconstruction.Preprint, arXiv:2505.23416. Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Z...

    arXiv 2025

  4. [4]

    I cannot determine

    NEVER GIVE UP. NEVER write phrases like "I cannot determine", "I am unable to find ", "I do not have enough information", " please provide more details", "based on the available information I cannot answer", " the knowledge base does not contain", or anything similar. If you don't have enough information, KEEP SEARCHING

    work page

  5. [5]

    Preprint, arXiv:2602.16284

    Fast kv compaction via attention matching. Preprint, arXiv:2602.16284. 11 HyperparameterQwen3.5 Gemma-4 Temperature 1.0 1.0 Top-p0.95 0.95 Top-k20 64 Presence penalty 1.5 – Max tokens/turn 4096 4096 Max turns 100 100 Thinking mode enabled enabled Table 4: Agent generation hyperparameters. A Implementation Details A.1 Package and Hardware We implement onli...

    arXiv

  6. [6]

    Once you have a top candidate, you MUST issue additional verification searches from new angles to confirm the candidate satisfies EVERY criterion

    CROSS−VERIFY BEFORE COMMITTING. Once you have a top candidate, you MUST issue additional verification searches from new angles to confirm the candidate satisfies EVERY criterion

    work page

  7. [7]

    Start by decomposing the question into all its individual criteria

    DECOMPOSE AND DRILL. Start by decomposing the question into all its individual criteria. Issue focused searches for each criterion. Then combine criteria progressively to narrow down the candidate set. Remember: more searches always produce more confident answers. The 20−search floor exists because shallow investigation routinely misses the correct answer...

    work page

  8. [8]

    You MUST call` get_document`AT LEAST 5 times across the investigation to read full document content (snippets are truncated at ~512 tokens and the answer often sits past them)

    DOCUMENT−READ FLOOR. You MUST call` get_document`AT LEAST 5 times across the investigation to read full document content (snippets are truncated at ~512 tokens and the answer often sits past them)

    work page

Show all 12 references
  1. [10]

    Never repeat a search that already failed

    MULTI−ANGLE COVERAGE -- Each of your 20+ searches must approach the question from a DIFFERENT angle. Never repeat a search that already failed

  2. [2024]

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou

    Deepseek-v2: A strong, economical, and effi- cient mixture-of-experts language model.Preprint, arXiv:2405.04434. Alessio Devoto, Maximilian Jeblick, and Simon Jégou

    arXiv

  3. [2025]

    Preprint, arXiv:2510.00636

    Expected attention: Kv cache compression by estimating attention from future queries distribution. Preprint, arXiv:2510.00636. Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christo- ...

    2025

  4. [2026]

    Preprint, arXiv:2511.02230

    Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live. Preprint, arXiv:2511.02230. Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu. 2025....

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.