Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Semantic structure in agent prompts is a robust signal for KV-cache eviction: region-aware decay preserves critical instructions and stays usable as context grows while pure recency collapses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:39 UTC pith:ZEQTB6K2

load-bearing objection Solid, honest systems paper: region half-lives really do differ by ~10× and pinning works by design; unpinned recall loses to H2O and the eval is still synthetic/small-scale. the 3 major comments →

arxiv 2607.10582 v1 pith:ZEQTB6K2 submitted 2026-07-12 cs.LG cs.AI

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Venkatesha Matam , Keon Kim This is my paper
classification cs.LG cs.AI
keywords KV cacheKV cache compressioncache evictionlarge language modelsLLM agentsagentic workflowsefficient LLM inferencelong-context inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agents pile up heterogeneous context—system instructions, plans, tool outputs, retrievals, and scratchpad reasoning—whose useful lifetimes differ sharply, yet standard KV-cache eviction treats every token the same. MemDecay is a training-free policy that takes region labels already known to the orchestrator, assigns each region a base priority and decay rate, refreshes a token’s score when attention hits it, and evicts the lowest-scoring pages under a fixed budget while pinning critical regions. Across two context lengths and two model sizes, measured attention half-lives differ by an order of magnitude (system roughly 150–190 decode steps versus scratchpad 14–16), and the ordering of movable regions is robust to insertion order. Pinning keeps system-region facts at full-cache accuracy where baselines fail, and region-aware retention remains effective as context grows while recency collapses; accumulated attention still wins on unpinned content, and ablations show the attention term needs magnitude normalization. A reader who cares about long-horizon agent serving gets a free structural signal and a clear diagnosis of how to combine it with attention.

Core claim

Across controlled agent episodes at roughly 450 and 1,700 tokens with Qwen2.5-1.5B and 3B, region-conditioned attention lifetimes differ by an order of magnitude—system half-lives of 148–189 decode steps against 14–16 for scratchpad—with non-overlapping intervals and an ordering of movable regions that is robust to insertion order. Pinning system pages preserves system-region facts at full-cache accuracy in every setting while no baseline preserves more than a small fraction of those probes; region-aware retention remains usable as context grows whereas recency-based retention collapses. Accumulated-attention retention is stronger on unpinned content, and ablations show the current attention

What carries the argument

MemDecay’s retention score: each token receives a region-specific base priority times exponential decay on time since last attention-triggered refresh, plus a weighted attention-importance term; pages are scored by the mean of their tokens and the lowest non-pinned pages are evicted under a fixed budget. That score turns orchestrator region labels into a scheduled, refreshable forgetting policy that maps onto page-granular KV layouts.

Load-bearing premise

The load-bearing premise is that synthetic multi-stage episodes with planted facts in fixed region templates, and decay rates fitted from attention mass versus age on those episodes, are representative enough of real agent workloads for the lifetime hierarchy and recall results to transfer.

What would settle it

On real multi-turn agent traces (coding or function-calling sessions), measure attention half-lives by region and re-run the same budgeted eviction probes: if system and scratchpad half-lives no longer differ by roughly an order of magnitude, or if pinning system pages no longer preserves instruction-fact recall at full-cache levels, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Agent serving stacks can pin system (and optionally plan) pages and apply calibrated region decay without retraining models.
  • Recency-only windows become unreliable for multi-stage agent traces beyond short contexts; structural priors age more gracefully.
  • Decay rates should be fitted from measured region-conditioned attention lifetimes rather than fixed qualitative defaults, because calibration can overturn defaults such as fast retrieval decay.
  • Combining structure with attention requires magnitude normalization of the attention term, not merely retuning its weight.
  • Page-granular eviction maps cleanly onto existing paged KV layouts when regions are contiguous spans.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If orchestrators already emit region metadata, MemDecay-style scoring could sit as standard middleware between agent frameworks and inference engines without model changes.
  • The same lifetime hierarchy suggests sizing retention budgets differently for tool-output and retrieval versus scratchpad in multi-agent workflow schedulers.
  • Magnitude-normalized attention plus region priors is a concrete hybrid that could close the unpinned-recall gap against pure accumulated-attention methods.
  • At larger models and real coding or function-calling traces, re-measuring straddle rates and position-confounded fixed regions would be needed before trusting the fitted rates in production.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. MemDecay proposes a training-free, region-aware KV-cache eviction policy for LLM agents. Tokens receive orchestrator-supplied region labels (system, plan, user, tool_in/out, retrieval, scratchpad); each region has a base priority b_r and decay rate λ_r. Retention scores combine a decayed structural prior with an EWMA of sampled attention, with attention above a threshold refreshing the decay clock (Eqs. 2–4). Under a fixed budget the policy evicts lowest-scoring pages while pinning system pages. Decay rates are calibrated from measured region-conditioned attention half-lives (Eqs. 6–7). On synthetic multi-stage episodes (~450 and ~1,700 tokens) with Qwen2.5-1.5B/3B, system half-lives are 148–189 decode steps versus 14–16 for scratchpad; pinning keeps system probes at full-cache accuracy while baselines keep at most a small fraction; recency collapses at longer context while structural retention does not; accumulated-attention (H2O-style) wins on unpinned content, and an ablation attributes MemDecay’s unpinned loss to magnitude mismatch of the attention term.

Significance. If the measured region-lifetime hierarchy and the pinning guarantee transfer beyond the controlled setting, the paper supplies a practical, interpretable prior for agent KV management that existing attention- or recency-only policies lack, and a calibration protocol that turns orchestrator metadata into fitted decay rates. Strengths that should be credited: (i) the lifetime hierarchy is replicated across three settings with scenario-bootstrap CIs and non-overlapping system vs scratchpad intervals (Table II); (ii) movable-region ordering is stress-tested with insertion-order variants; (iii) the evaluation is unusually honest about the unpinned loss and diagnoses a concrete design fix (attention magnitude normalization); (iv) a released harness and 4,320 scored probes support reproducibility. The work is a useful systems contribution at the intersection of agent orchestration and cache eviction, even if the full policy is not yet competitive on unpinned recall.

major comments (3)
  1. [Section IV-A, Limitations] Section IV-A and Limitations: all lifetime fits (Table II) and eviction recall (Tables III–IV) come from eight synthetic scenarios with planted facts, fixed opening/closing stages, and only three insertion-order variants for movable blocks. System/plan/scratchpad lifetimes remain position-confounded by construction. The central transfer claim—that semantic region structure is a robust signal for agent KV management—therefore rests on a workload class the paper itself flags as non-representative of SWE-bench/BFCL-style traces. Either add at least one real multi-turn agent workload, or substantially narrow the abstract/intro claims to “controlled synthetic episodes” and treat real-agent transfer as future work.
  2. [Tables III–IV, §IV-D] Tables III–IV and §IV-D ablation: on non-system probes MemDecay is consistently weaker than the H2O-style baseline (e.g., 3B long, 50% budget: 0.22 vs 0.60 non-sys), and varying α ∈ {0.2,0.5,0.8} leaves keep sets essentially unchanged because the page-mean and EWMA make α a_i one to two orders of magnitude below the structural term for old tokens. The paper correctly identifies magnitude normalization as the needed design change, but does not implement or evaluate it. Without that fix (or an equivalent), the claim that region-aware retention is an effective full eviction policy—not merely a pinning mechanism—is not supported by the unpinned results.
  3. [§IV-B, Eqs. (6)–(7)] §IV-B / Eqs. (6)–(7): calibration fits λ_r under a 48-step final observation burst while eviction-time signals use an 8-step burst, and the count-weighted fit anchors the intercept toward low-age buckets. Both choices are constant across policies, but they couple the fitted half-lives to a measurement schedule that is not the production observation interval (k>1). A sensitivity check of fitted λ_r and of eviction recall under matched observation budgets would strengthen the claim that the calibrated rates are the right ones for the policy as deployed.
minor comments (5)
  1. [Fig. 2] Fig. 2 is illustrative only (α a_i = 0); label it explicitly as schematic so readers do not confuse it with measured trajectories in Fig. 3.
  2. [Table I] Table I defaults assign retrieval a large decay rate, but calibration overturns this (retrieval is the longest-lived movable region). Consider moving the qualitative defaults to an appendix and leading with calibrated values to avoid readers adopting the overturned prior.
  3. [§III-D, §IV-A] Page-straddle rates (35% short tier, 17% long tier) are reported; a short note on how often pinned-page collateral protects non-system tokens would clarify how much of the “pinning win” is pure region protection versus page-level side effect.
  4. [§II-E] Related work cites concurrent RoleKV, IntentKV, and retention-gated caching; a one-row comparison table (training-free vs learned, single-context eviction vs cross-request reuse, region-conditioned λ vs global decay) would make the positioning in §II-E easier to scan.
  5. [Eqs. (3), (6)] Notation: Δt_i is used both for age since insertion/refresh and as the argument of the measured attention curves; a brief reminder at Eq. (6) that the measurement age is wall-clock decode age (not necessarily the policy’s refresh clock) would avoid ambiguity.

Circularity Check

1 steps flagged

No load-bearing circular derivation: half-lives are measured, decay rates are ordinary leave-one-out calibration, and the pinning win is an acknowledged designed property.

specific steps
  1. self definitional [Section IV-C; also Abstract and contributions]
    "Pinning preserves system-region facts at the full-cache ceiling in every setting, no baseline preserves more than 13 of 24... Guaranteed instruction survival under memory pressure is a designed property that no attention- or recency-based baseline provides."

    If F contains all system tokens and pinned tokens are never evicted, system-region probe survival at the full-cache ceiling is true by construction of the pin set, not an independent test of the decay/attention terms. The paper correctly calls this a designed property, so the circularity is mild and disclosed rather than a hidden reduction of a claimed derivation.

full rationale

MemDecay is an empirical systems paper, not a first-principles derivation. Region half-lives (Table II, Eqs. 6–7) are fitted from measured attention-mass-vs-age curves; the paper does not present those fits as independent predictions of the same quantity. Leave-one-scenario-out calibration uses attention statistics to set λ_r, then evaluates a different outcome (query-agnostic planted-fact recall after eviction). That is standard calibration, not fitted-input-as-prediction. The retention score (Eq. 4) is a designed combination of structural priors plus attention EWMA; evaluation against full-cache, random, streaming, and H2O-style baselines is external to the score definition. The only mild self-definitional element is that pinning system pages guarantees system-probe survival by construction of F—which the paper itself labels a designed property rather than an empirical discovery. No self-citation uniqueness theorem, ansatz smuggling, or renaming of a known result carries the central claims. Score 1 reflects that single transparent tautology; the lifetime hierarchy and unpinned-recall comparisons remain non-circular measurements.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on measuring attention mass by region and age under a fixed agent-template workload, then using fitted λ_r plus hand-set base priorities and hyperparameters in a composite retention score. No new physical entities; the invented objects are the scoring rule and region configuration. Load-bearing domain assumptions are orchestrator region labels, attention as a proxy for future utility, and synthetic planted-fact recall as a stand-in for agent correctness under eviction.

free parameters (6)
  • region decay rates λ_r = half-lives e.g. system 148–189, scratchpad 14–16 steps (Table II)
    Fitted per region and setting via weighted log-linear regression of mean attention vs age (Eqs. 6–7); leave-one-scenario-out. Directly set the structural forgetting schedule.
  • region base priorities b_r = defaults in Table I (e.g. system high, retrieval low)
    Default qualitative values in Table I (system high, scratchpad low, etc.); not learned from data. Set initial retention before decay.
  • attention weight α = 0.5 default
    Hand-set (default 0.5); ablation over {0.2,0.5,0.8} shows near-invariance because α a_i is magnitude-dominated by the structural term.
  • EWMA smoothing ρ = 0.9
    Hand-set smoothing for attention importance a_i (Eq. 2).
  • refresh threshold τ = 2/|K_t|
    Scaled as 2/|K_t|; controls when attention resets the decay clock.
  • page size = 16 tokens
    Serving-layout hyperparameter for page-mean scores and eviction granularity.
axioms (5)
  • domain assumption Orchestration layer supplies accurate contiguous region labels for tokens (system, plan, user, tool_in, tool_out, retrieval, scratchpad).
    Section III-A; without labels the policy cannot apply region priors. Serving metadata is described as emerging, not universal.
  • domain assumption Mean attention mass vs token age is a valid proxy for region-conditioned future utility for cache retention.
    Calibration protocol Section IV-B; eviction quality is then tested separately via planted-fact probes.
  • ad hoc to paper Additive token retention scores and equal-size page means yield a practical optimum under a fixed budget (greedy eviction of lowest Sp).
    Eqs. (1) and (5); authors note deviation when pages straddle regions.
  • domain assumption Pinned system pages (and collateral co-page tokens) should never be evicted for stable decoding and instruction survival.
    Section III-A default F; aligns with attention-sink practice but is a policy choice that guarantees the system-probe win.
  • ad hoc to paper Synthetic multi-stage episodes with planted facts and greedy 16-token probes measure eviction quality relevant to agent inference.
    Section IV-A evaluation design; Limitations admit real agent workloads remain future work.
invented entities (2)
  • MemDecay retention score s_i(t) = b_r exp(-λ_r Δt_i) + α a_i with attention-triggered decay refresh no independent evidence
    purpose: Single interpretable score combining structural prior, temporal decay, and observed importance for page eviction.
    Defined in Eqs. (2)–(4); not a physical entity but the paper’s core postulated mechanism. Independent evidence is the empirical study, not an external measurement of the score itself.
  • Default seven-region agent prompt ontology with per-region (b_r, λ_r, pin) configuration independent evidence
    purpose: Map orchestrator segments onto cache classes with different lifetimes.
    Table I; regions are standard agent template names, but the specific priority/decay/pin assignment is paper-defined and workload-calibrated.

pith-pipeline@v1.1.0-grok45 · 17790 in / 4026 out tokens · 43714 ms · 2026-07-14T10:39:33.652683+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference." pith.science (2026). https://pith.science/paper/ZEQTB6K2

@misc{pith2026260710582,
  author       = {Pith},
  title        = {Pith review of: MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZEQTB6K2}},
  note         = {Machine review of arXiv:2607.10582}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.

Figures

Figures reproduced from arXiv: 2607.10582 by Keon Kim, Venkatesha Matam.

Figure 1
Figure 1. Figure 1: MemDecay in the serving stack. The orchestrator labels each prompt [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Retention score trajectories from (4) under example parameters, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Measured mean attention mass per token by region and token age [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 15 linked inside Pith

  1. [1]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” inProc. ACM Symp. Operating Systems Principles (SOSP), 2023

    2023

  2. [2]

    A survey on large language model acceleration based on KV cache management,

    H. Li et al., “A survey on large language model acceleration based on KV cache management,” arXiv:2412.19442, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Model tells you what to discard: Adaptive KV cache compression for LLMs,

    S. Ge et al., “Model tells you what to discard: Adaptive KV cache compression for LLMs,” inProc. Int. Conf. Learning Representations (ICLR), 2024

    2024

  4. [4]

    Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference,

    Y . Feng, J. Lv, Y . Cao, X. Xie, and S. K. Zhou, “Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference,” arXiv:2407.11550, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    H 2O: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang et al., “H 2O: Heavy-hitter oracle for efficient generative inference of large language models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  6. [6]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” inProc. Int. Conf. Learning Representations (ICLR), 2024

    2024

  7. [7]

    Transformers are multi-state RNNs,

    M. Oren, M. Hassid, N. Yarden, Y . Adi, and R. Schwartz, “Transformers are multi-state RNNs,” arXiv:2401.06104, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    A survey on the memory mechanism of large language model based agents,

    Z. Zhang et al., “A survey on the memory mechanism of large language model based agents,” arXiv:2404.13501, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Not all tokens are worth caching: Learning semantic- aware eviction for LLM prefix caches,

    S. Fang et al., “Not all tokens are worth caching: Learning semantic- aware eviction for LLM prefix caches,” arXiv:2605.18825, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    RoleKV: Role-aware KV cache management for the inverted age-importance of LLM agent context,

    Anonymous, “RoleKV: Role-aware KV cache management for the inverted age-importance of LLM agent context,” under review, Open- Review forumof1W47Odj1, 2026

    2026

  11. [11]

    ChunkAttention: Efficient self-attention with prefix-aware KV cache and two-phase partition,

    L. Ye, Z. Tao, Y . Huang, and Y . Li, “ChunkAttention: Efficient self-attention with prefix-aware KV cache and two-phase partition,” arXiv:2402.15220, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    CacheBlend: Fast large language model serving for RAG with cached knowledge fusion,

    J. Yao et al., “CacheBlend: Fast large language model serving for RAG with cached knowledge fusion,” inProc. European Conf. Computer Systems (EuroSys), 2025

    2025

  13. [13]

    KV cache compression for inference efficiency in LLMs: A review,

    Y . Liu et al., “KV cache compression for inference efficiency in LLMs: A review,” arXiv:2508.06297, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time,

    Z. Liu et al., “Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  15. [15]

    SnapKV: LLM knows what you are looking for before generation,

    Y . Li et al., “SnapKV: LLM knows what you are looking for before generation,” arXiv:2404.14469, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    NACL: A general and effective KV cache eviction framework for LLM at inference time,

    Y . Chen et al., “NACL: A general and effective KV cache eviction framework for LLM at inference time,” inProc. 62nd Annu. Meeting Assoc. Computational Linguistics (ACL), 2024

    2024

  17. [17]

    Expected attention: KV cache compression by estimating attention from future queries distribution,

    A. Devoto, M. Jeblick, and S. J ´egou, “Expected attention: KV cache compression by estimating attention from future queries distribution,” arXiv:2510.00636, 2025

    arXiv 2025

  18. [18]

    Quest: Query-aware sparsity for efficient long-context LLM inference,

    J. Tang et al., “Quest: Query-aware sparsity for efficient long-context LLM inference,” inProc. Int. Conf. Machine Learning (ICML), 2024

    2024

  19. [19]

    ClusterKV: Manipu- lating LLM KV cache in semantic space for recallable compression,

    G. Liu, C. Li, J. Zhao, C. Zhang, and M. Guo, “ClusterKV: Manipu- lating LLM KV cache in semantic space for recallable compression,” arXiv:2412.03213, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    ChunkKV: Semantic-preserving KV cache compression for efficient long-context LLM inference,

    X. Liu et al., “ChunkKV: Semantic-preserving KV cache compression for efficient long-context LLM inference,” arXiv:2502.00299, 2025

    arXiv 2025

  21. [21]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache,

    Z. Liu et al., “KIVI: A tuning-free asymmetric 2bit quantization for KV cache,” inProc. Int. Conf. Machine Learning (ICML), 2024

    2024

  22. [22]

    LKV: End-to-end learning of head-wise budgets and token selection for LLM KV cache eviction,

    E. Zhou et al., “LKV: End-to-end learning of head-wise budgets and token selection for LLM KV cache eviction,” arXiv:2605.06676, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live,

    H. Li et al., “Continuum: Efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live,” arXiv:2511.02230, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    KVFlow: Efficient prefix caching for accelerating LLM- based multi-agent workflows,

    Z. Pan et al., “KVFlow: Efficient prefix caching for accelerating LLM- based multi-agent workflows,” inAdvances in Neural Information Pro- cessing Systems (NeurIPS), 2025

    2025

  25. [25]

    IntentKV: Cross-turn intent-aware KV cache pruning for agent inference,

    J. Li, J. Lou, and J. Li, “IntentKV: Cross-turn intent-aware KV cache pruning for agent inference,” arXiv:2606.09916, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Cache what lasts: Token retention for memory-bounded KV cache in LLMs,

    N. Bui, S. Sharma, S. Lamba, S. Mishra, and R. Ying, “Cache what lasts: Token retention for memory-bounded KV cache in LLMs,” arXiv:2512.03324, 2025

    arXiv 2025

  27. [27]

    From human memory to AI memory: A survey on memory mechanisms in the era of LLMs,

    Y . Wu et al., “From human memory to AI memory: A survey on memory mechanisms in the era of LLMs,” arXiv:2504.15965, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    CONF-KV: Confidence-aware KV cache eviction with mixed-precision storage for long-horizon LLM,

    Y . Li and Y . Miao, “CONF-KV: Confidence-aware KV cache eviction with mixed-precision storage for long-horizon LLM,” arXiv:2605.24786, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    DynamicKV: Task-aware adaptive KV cache compres- sion for long context LLMs,

    X. Zhou et al., “DynamicKV: Task-aware adaptive KV cache compres- sion for long context LLMs,” arXiv:2412.14838, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    KVzip: Query-agnostic KV cache compression with context reconstruction,

    J.-H. Kim et al., “KVzip: Query-agnostic KV cache compression with context reconstruction,” arXiv:2505.23416, 2025

    arXiv 2025

  31. [31]

    Improving WWW proxies performance with Greedy- Dual-Size-Frequency caching policy,

    L. Cherkasova, “Improving WWW proxies performance with Greedy- Dual-Size-Frequency caching policy,” Hewlett-Packard Laboratories, Tech. Rep. HPL-98-69R1, 1998

    1998

  32. [32]

    RFC: Agent-aware KV cache phase 1 for agen- tic workloads,

    SGLang Project, “RFC: Agent-aware KV cache phase 1 for agen- tic workloads,” GitHub issue 24656, 2026. [Online]. Available: github.com/sgl-project/sglang/issues/24656

    2026

  33. [33]

    KV Cache Compression and Its Infra Problems,

    NVIDIA Research, “KV Cache Compression and Its Infra Problems,” June 2026. [Online]. Available: research.nvidia.com/labs/eai/blogs/

    2026