Pith. sign in

REVIEW 4 major objections 5 minor 24 references

PagedEviction claims that a block-wise KV-cache eviction scheme using a static K/V importance ratio keeps long-context accuracy near full-cache levels while shrinking memory and raising throughput.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:12 UTC pith:HP3FF5LW

load-bearing objection Block-aligned eviction is a real idea, but the paper's headline accuracy claim doesn't survive its own numbers. the 4 major comments →

arxiv 2509.04377 v1 pith:HP3FF5LW submitted 2025-09-04 cs.LG

PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

Krishna Teja Chitty-Venkata , Jie Ye , Xian-He Sun , Anthony Kougkas , Murali Emani , Venkatram Vishwanath , Bogdan Nicolae This is my paper
classification cs.LG
keywords KV cachecache evictionPagedAttentionvLLMblock-wise pruningattention-free importanceLongBenchLLM inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PagedEviction is a way to shrink the KV cache during LLM inference without touching the attention kernels. It ranks each token by the ratio of its Value L2 norm to its Key L2 norm, evicts the least important tokens during prefill, and during decoding evicts whole memory pages one at a time only when a new page fills. On LongBench tasks with Llama-1B, 3B, and 8B models, the paper reports staying within about 0.5–1.5 ROUGE points of a full cache at a 1024-token budget while improving throughput by up to 3.1x and cutting per-token latency by 10–12%. If correct, this gives serving systems a drop-in, kernel-compatible way to fit longer contexts and larger batches into the same GPU memory.

Core claim

The central claim is that token importance for KV-cache eviction can be read directly from the already-cached Key and Value tensors using the score S_i = ||V_i||2 / ||K_i||2, and that evicting whole pages based on the mean of these scores preserves accuracy while staying aligned with vLLM's paged memory. In prefill, the least-important tokens are dropped before pages are formed, down to a fixed cache budget. In decode, when the most recent page becomes full, the page with the lowest mean ratio is evicted wholesale and the page table is updated. The paper argues that this matches PagedAttention's block granularity, avoids extracting attention scores from FlashAttention-style kernels, and keep

What carries the argument

The importance proxy is S_i = ||V_i||2 / ||K_i||2, together with its block-level mean. The paper treats the Value L2 norm as carrying token content and, following the previously observed inverse correlation between Key L2 norm and cumulative attention, uses the Key L2 norm as a proxy for attention weight. The ratio combines both into a static, attention-free importance score, which is what lets eviction decisions be made from the cached K/V tensors alone so that whole pages can be scored and dropped without any CUDA kernel modification.

Load-bearing premise

Token importance can be ranked by the ratio ||V_i||2 / ||K_i||2, and a page's mean ratio ranks whole-page importance; the paper offers no derivation that this ratio tracks true attention relevance, and the cited inverse-key-norm result supports only the denominator part.

What would settle it

On one LongBench task at a 1024-token budget, run the identical block/paging code with the proposed ratio replaced by (i) ||V_i||2 alone, (ii) ||K_i||2 alone, and (iii) random block eviction. If any of these matches the ratio's accuracy within noise, the proposed score is not what preserves quality; additionally, label which pages containing gold answer tokens are evicted and check whether high-scoring pages actually contain the answer tokens the model needs.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Serving systems can drop the KV cache to a fixed token budget (e.g., 1024) on long-context tasks and stay within roughly 0.5–1.5 ROUGE points of full-cache quality, per the paper's LongBench results.
  • Because eviction is whole-page and aligned to the paged layout, the method avoids the fragmentation and per-step block-table churn of token-level eviction, preserving PagedAttention's memory-management benefits.
  • The kernel-agnostic design means the method can be added to vLLM-style serving runtimes without changing attention kernels, lowering the deployment cost of cache compression.
  • On the paper's throughput runs, a 1024-token budget yields up to about 3020 tokens/sec on Llama-3.2-1B (37% over full cache) and 10–12% lower time-per-output-token across 1B, 3B, and 8B models.
  • The method is complementary to other cache optimizations such as layer-wise budget allocation and KV quantization, so further memory savings can be layered on top of page-aligned eviction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct ablation swapping the ratio for either ||V_i||2 alone or ||K_i||2 alone would isolate which term actually preserves accuracy; the paper does not report this, so the value-norm contribution remains an open question.
  • Because the importance score uses only the K/V tensors and the block table, the same scheme should transfer to other paged serving systems or attention kernels that expose KV-cache tensors, not just vLLM.
  • The reported page-size robustness at 16 and 32 suggests block granularity is not the main accuracy bottleneck; a natural extension is to test adaptive page sizes or per-layer budgets, keeping larger pages in early layers and smaller pages in later layers.
  • If the ratio truly tracks attention relevance, the same score could be repurposed for KV-cache quantization by deciding which blocks retain higher precision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PagedEviction, a block-aligned KV cache eviction method for vLLM's PagedAttention. Each token is scored with ||V_i||2/||K_i||2; during prefill the lowest-scoring tokens are evicted to a cache budget, and during decode entire pages are evicted when a page fills, always replacing the lowest-scoring page. The authors evaluate accuracy on five LongBench tasks with LLaMA-3.2-1B/3B and LLaMA-3.1-8B across cache budgets 256-4096, and report throughput/latency on a synthetic workload. The central claims are attention-free, block-structured eviction, no CUDA kernel changes, improved memory efficiency, and better accuracy than attention-free baselines.

Significance. The core idea is practical and timely: block-granular eviction preserves vLLM's paged layout and avoids per-step token-level fragmentation, and using static K/V states avoids the need to materialize attention scores. Notable strengths are that the method has no fitted parameters, is deterministic, integrates with PagedAttention without kernel changes, and is compared against relevant attention-free baselines. If the accuracy and throughput claims are validated, this would be a useful systems contribution. However, the empirical validation is currently undermined by a direct contradiction in the headline accuracy claim and by the absence of an ablation for the importance metric.

major comments (4)
  1. [Section 6 vs Section 5.3/Figure 2] The conclusion states that PagedEviction is 'within 0.5-1.5 ROUGE points of full-cache performance at tight budgets (1024 tokens)'. This is contradicted by the GovReport/LLaMA-3.2-1B results: at budget 1024, PagedEviction is ~24.5 while Full Cache is ~30, a gap of ~5.5 ROUGE points (roughly 18% relative). The 0.5-1.5 number appears to correspond to the 4096 budget (~29.5 vs 30), not 1024. Figure 4's caption also claims 'less than 3-5% degradation', which is inconsistent with this 18% gap. This is load-bearing because it is the central accuracy claim in the abstract and conclusion. Please report exact numbers and revise the claim.
  2. [Section 4.1, Algorithm 1] The importance metric ||V_i||2/||K_i||2 is asserted without derivation. The cited Devoto et al. result supports an inverse relationship with the key L2 norm, not the value-norm term or their ratio. No ablation isolates the contribution of the value norm or tests whether the block-level average is a faithful proxy for token-level importance. Since the correctness of every eviction decision depends on this score, please add ablations (inverse key norm only, value norm only, product/ratio, and block-level average vs token-level selection) and report whether the differences are material.
  3. [Section 5.1 / Section 5.4, Figure 3] Throughput and latency are measured only on a synthetic workload (input length 1024, output length 8192, 64 concurrent requests). The abstract and conclusion present these as general hardware benefits. No LongBench-based generation trace or variable-batch experiment is reported, and the throughput figures have no error bars. Please either add realistic workload results or explicitly limit the throughput and latency claims to the synthetic configuration used.
  4. [Section 5.3] All accuracy results are reported as approximate values read from plots (e.g., '~24.5'), with no numerical tables and no standard deviations. The five datasets are a small subset of LongBench, while the abstract says 'on the LongBench benchmark suite'. Exact tables with variance, and a clear statement of dataset coverage, are needed to verify the empirical claims and to resolve the contradiction in the headline accuracy statement.
minor comments (5)
  1. [Section 5.2] The description of the Inverse Key L2-Norm baseline is inconsistent with Section 4.1. Section 4.1 says key L2 norm is inversely proportional to cumulative attention, while Section 5.2 says Inverse Key L2-Norm evicts high-norm keys under the assumption that lower-norm keys are less influential. Please clarify the direction of the heuristic.
  2. [Algorithm 2] Line 12 says 'Evict(K,E), Evict(V,E)' but does not state that the E tokens are those with the lowest S_i from line 8. Make the dependency explicit.
  3. [Figure 4] The y-axis labels read 'Rogue Score' in panels (d)-(i); should be 'ROUGE Score'.
  4. [Notation] The symbol B is overloaded: it denotes block/page size in Algorithms 1 and 3, while the cache budget is C. Please use distinct symbols consistently, especially since both appear in the same algorithms.
  5. [Section 5.1] Decoding settings for LongBench evaluation are not stated (e.g., max_new_tokens, sampling parameters, number of runs). Please provide them for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: PagedEviction's accuracy and throughput results are empirical, and its importance heuristic is a fixed, externally-motivated proxy rather than a fitted or self-cited input.

full rationale

The claimed derivation chain is: (1) choose a static token importance proxy S_i = ||V_i||_2 / ||K_i||_2 (Algorithm 1, Sec 4.1); (2) aggregate it into block averages; (3) evict lowest-scoring tokens/pages (Algorithms 2-3); (4) measure ROUGE/F1 and throughput against Full Cache and attention-free baselines (Sec 5). No parameter is fitted to the evaluation data, no predicted quantity is a renamed fit, and no load-bearing result is imported from the authors' own prior work. The importance proxy is an ansatz; the citation to Devoto et al. (2024) supports only the inverse-key-L2 component, so the value-norm numerator is a new heuristic, but that is a motivation/validation gap, not circularity. The paper's internal inconsistency (the Conclusion claims 0.5-1.5 ROUGE points at 1024 while Sec 5.3 shows roughly a 5.5-point gap on GovReport with LLaMA-3.2-1B) is a factual/consistency error, not a self-referential reduction. External LongBench benchmarks and vLLM throughput measurements are independent evidence. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The method introduces no new particles, forces, or entities. It relies on four unproven modeling assumptions about the importance score and block-level aggregation. There are no fitted free parameters; the page size and cache budgets are experimental settings, not learned constants.

axioms (4)
  • ad hoc to paper Token importance for future attention can be ranked by ||V_i||2 / ||K_i||2.
    Introduced in Section 4.1 with no derivation or ablation; Devoto et al. supports only the inverse key L2 norm, not the ratio with the value norm.
  • domain assumption Key L2 norm is inversely proportional to cumulative attention score (Devoto et al., 2024).
    Cited in Section 4.1 to justify the heuristic; accepted from prior work, but the cited paper does not establish the value-norm component.
  • ad hoc to paper The average token score of a block is a reliable proxy for block-level importance.
    Used in Algorithm 1 for block mode; no analysis of score variance within a block or comparison of mean versus other aggregations.
  • domain assumption Evicting only when the most recent block is full (L % B == 0) preserves the cache budget and avoids fragmentation.
    Core design choice in Algorithm 3; evaluated empirically but not analyzed, and no memory measurements confirm the claimed budget behavior.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 15029 in / 9726 out tokens · 83494 ms · 2026-08-05T10:12:22.929317+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference." pith.science (2026). https://pith.science/paper/HP3FF5LW

@misc{pith2026250904377,
  author       = {Pith},
  title        = {Pith review of: PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HP3FF5LW}},
  note         = {Machine review of arXiv:2509.04377}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM's PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks.

Figures

Figures reproduced from arXiv: 2509.04377 by Anthony Kougkas, Bogdan Nicolae, Jie Ye, Krishna Teja Chitty-Venkata, Murali Emani, Venkatram Vishwanath, Xian-He Sun.

Figure 1
Figure 1. Figure 1: Illustration of the PagedEviction technique integrated into vLLM with a block size ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy vs Cache Budget of 1B, 3B, and 8B models on GovReport, HotpotQA, MultiFieldQA, Qasper and Multi News [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a,b,c) Cache Budget vs Throughput of Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Throughput (a–c) and ROUGE Score (d–i) across different LLaMA models (1B, 3B, 8B) and datasets (MultiNews, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the StreamingLLM KV cache eviction strategy. During prefill (Step 0), the last tokens are evicted to fit [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of Unstructured KV Cache Eviction using token-wise importance metrics such as Inverse Key L2-Norm or [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages

  1. [4]

    arXiv preprint arXiv:2406.11430

    A Simple and Effective L 2 Norm-Based Strategy for KV Cache Compression. arXiv preprint arXiv:2406.11430. Dong, S.; Cheng, W.; Qin, J.; and Wang, W

    Pith/arXiv arXiv

  2. [5]

    arXiv preprint arXiv:2403.04643

    QAQ: Quality Adaptive Quantization for LLM KV Cache. arXiv preprint arXiv:2403.04643. Fabbri, A. R.; Li, I.; She, T.; Li, S.; and Radev, D. R

    Pith/arXiv arXiv

  3. [7]

    arXiv preprint arXiv:2407.11550

    Ada-kv: Optimizing kv cache eviction by adaptive bud- get allocation for efficient llm inference. arXiv preprint arXiv:2407.11550. Ge, S.; Zhang, Y .; Liu, L.; Zhang, M.; Han, J.; and Gao, J

    Pith/arXiv arXiv

  4. [8]

    arXiv preprint arXiv:2310.01801

    Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al

    Pith/arXiv arXiv

  5. [10]

    arXiv preprint arXiv:2405.14256

    ZipCache: Accurate and Efficient KV Cache Quan- tization with Salient Token Identification. arXiv preprint arXiv:2405.14256. Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A

    Pith/arXiv arXiv

  6. [12]

    arXiv preprint arXiv:2401.18079

    Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079. Huang, L.; Cao, S.; Parulian, N.; Ji, H.; and Wang, L

    Pith/arXiv arXiv

  7. [13]

    arXiv preprint arXiv:2104.02112

    Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112. Kang, H.; Zhang, Q.; Kundu, S.; Jeong, G.; Liu, Z.; Krishna, T.; and Zhao, T

    Pith/arXiv arXiv

  8. [14]

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y .; Zheng, L.; Yu, C

    Gear: An efficient kv cache compres- sion recipefor near-lossless generative inference of llm.arXiv preprint arXiv:2403.05527. Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y .; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; and Stoica, I

    Pith/arXiv arXiv

  9. [15]

    In Proceedings of the 29th Symposium on Operating Systems Principles, 611–626

    Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 611–626. Li, H.; Li, Y .; Tian, A.; Tang, T.; Xu, Z.; Chen, X.; Hu, N.; Dong, W.; Li, Q.; and Chen, L. 2024a. A Survey on Large Language Model Acceleration based on KV Cache Management. arXiv preprint arXi...

    Pith/arXiv arXiv

  10. [17]

    arXiv preprint arXiv:2504.15364

    KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments. arXiv preprint arXiv:2504.15364. Shi, L.; Zhang, H.; Yao, Y .; Li, Z.; and Zhao, H

    arXiv

  11. [18]

    arXiv preprint arXiv:2407.18003

    Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption. arXiv preprint arXiv:2407.18003. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi `ere, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al

    Pith/arXiv arXiv

  12. [19]

    arXiv preprint arXiv:2302.13971

    Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971. Wan, Z.; Wu, X.; Zhang, Y .; Xin, Y .; Tao, C.; Zhu, Z.; Wang, X.; Luo, S.; Xiong, J.; and Zhang, M

    Pith/arXiv arXiv

  13. [20]

    arXiv preprint arXiv:2406.13035

    D2O: Dynamic Discriminative Operations for Efficient Genera- tive Inference of Large Language Models. arXiv preprint arXiv:2406.13035. Wang, Z.; Jin, B.; Yu, Z.; and Zhang, M

    Pith/arXiv arXiv

  14. [21]

    arXiv preprint arXiv:2407.08454

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454. Xiao, G.; Tian, Y .; Chen, B.; Han, S.; and Lewis, M

    Pith/arXiv arXiv

  15. [22]

    arXiv preprint arXiv:2309.17453

    Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. Yang, D.; Han, X.; Gao, Y .; Hu, Y .; Zhang, S.; and Zhao, H. 2024a. PyramidInfer: Pyramid KV Cache Compres- sion for High-throughput LLM Inference. arXiv preprint arXiv:2405.12532. Yang, J. Y .; Kim, B.; Bae, J.; Kwon, B.; Park, G.; Yang, E.; Kwon, S. J.; and Lee, ...

    Pith/arXiv arXiv

  16. [24]

    Proceedings of Machine Learning and Systems, 6: 196–209

    Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6: 196–209. A Appendix A.1 StreamingLLM Figure 5 illustrates the StreamingLLM eviction strategy. Ini- tially, prompt tokens are processed, where only the earliest tokens (used as attention sinks) are retained, and the remain- ing prompt tokens a...

    work page 2024

  17. [36]

    Liu, Z.; Yuan, J.; Jin, H.; Zhong, S.; Xu, Z.; Braverman, V .; Chen, B.; and Hu, X. 2024c. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750. Meta

    Pith/arXiv arXiv

  18. [2018]

    In Conference on Empirical Methods in Natural Language Processing (EMNLP)

    HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question An- swering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Zhang, Y .; Du, Y .; Luo, G.; Zhong, Y .; Zhang, Z.; Liu, S.; and Ji, R. 2024a. Cam: Cache merging for memory-efficient llms inference. In Forty-first international conference on machine learning. Zhang, Y...

    Pith/arXiv arXiv

  19. [2019]

    arXiv preprint arXiv:1906.01749

    Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.01749. Feng, Y .; Lv, J.; Cao, Y .; Xie, X.; and Zhou, S. K

    Pith/arXiv arXiv 1906

  20. [2020]

    arXiv preprint arXiv:2011.01060

    Constructing a multi-hop QA dataset for com- prehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. Hooper, C.; Kim, S.; Mohammadzadeh, H.; Mahoney, M. W.; Shao, Y . S.; Keutzer, K.; and Gholami, A

    Pith/arXiv arXiv 2011

  21. [2021]

    Devoto, A.; Zhao, Y .; Scardapane, S.; and Minervini, P

    A dataset of information-seeking ques- tions and answers anchored in research papers.arXiv preprint arXiv:2105.03011. Devoto, A.; Zhao, Y .; Scardapane, S.; and Minervini, P

    Pith/arXiv arXiv

  22. [2023]

    arXiv preprint arXiv:2308.14508

    Longbench: A bilingual, multitask benchmark for long context understand- ing. arXiv preprint arXiv:2308.14508. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Pith/arXiv arXiv

  23. [2024]

    arXiv preprint arXiv:2401.06066

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066. Dasigi, P.; Lo, K.; Beltagy, I.; Cohan, A.; Smith, N. A.; and Gardner, M

    Pith/arXiv arXiv

  24. [2025]

    arXiv preprint arXiv:2501.12948

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. He, Y .; Zhang, L.; Wu, W.; Liu, J.; Zhou, H.; and Zhuang, B

    Pith/arXiv arXiv