Pith. sign in

REVIEW 5 major objections 5 minor 37 references

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Per-layer KV-cache budgets cut long-context LLM memory 61.6% without accuracy loss.

desk verdict Useful empirical extension of PyramidKV/SnapKV with a shaky theoretical bridge; the per-layer dynamic allocation idea is worth a round of revision, not rejection. read the letter →

arxiv 2412.05896 v1 pith:6GLTI54Z submitted 2024-12-08 cs.LG cs.CL

Weizhuo Li , Zhigang Wang , Yu Gu , Ge Yu This is my paper · ORCID
classification cs.LGcs.CL
keywords KVcachelong-contextinferencelayer-wiseallocationattentionimportanceevictiongreedyoptimizationLLMefficiencyprefillcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the key-value (KV) cache of a large language model can be compressed much more aggressively if the cache budget is allocated per layer rather than uniformly or by a fixed pyramid pattern. Its central observation, called the Dynamic Differences of Importance Distribution (DDID), is that attention scores assign very different token importance at different transformer layers, and that these differences shift with the task. The paper defines an Importance Retention Ratio for each layer, argues on theoretical and experimental grounds that end-to-end accuracy tracks the average of these ratios, and turns cache budgeting into a combinatorial optimization problem with a greedy optimal solution. The resulting XKV pipeline uses a lightweight, cache-free mini-pass over the input to collect per-layer importance statistics before the actual full pass, plus a sampling step that reuses allocation patterns across tasks of the same type. On a long-context benchmark with Llama-3.1-8B, XKV reduces KV-cache memory by 61.6% on average, improves computational efficiency by 2.1x, and increases throughput by up to 5.2x while matching or beating static baselines on accuracy.

What carries the argument

The load-bearing quantities are the per-layer Importance Retention Ratio $R_i = \mathrm{Sum}(\mathrm{Topk}(n_i, w_i))/\mathrm{Sum}(w_i)$, which measures what share of a layer's attention-score mass survives after keeping $n_i$ tokens, and the Importance-to-Size Ratio $ISR_i = R_i/\log_2(n_i)$, which measures how efficiently a layer converts cache budget into retained importance. The algorithmic core is a greedy allocation that repeatedly gives one more cache slot to the layer whose next token adds the most to the average ratio, with a wait list tracking each layer's next best candidate; the paper proves optimality for the two-layer case. The framework also includes a mini-prefill pass over a lightweight variant of the model that computes all layers' attention-score distributions without caching, plus a sampling strategy that averages allocation lists across tasks of the same type to avoid repeating the mini-prefill.

What would settle it

Pick two per-layer cache allocations with the same total budget and the same average retention ratio but very different distributions of tokens across layers, run them on the same long-context benchmark, and compare end-of-task accuracy; if accuracy differs noticeably despite equal average retention, the equivalence that XKV optimizes is not sufficient.

Watch

Extended reading notes

Core claim

The paper's central claim is that existing KV-cache eviction leaves memory savings on the table because it treats all layers as equally important. When the same number of cached tokens is kept in every layer, layers with sparser attention retain nearly all their importance while other layers lose much more; equivalently, the cache size needed to hold a fixed retention ratio varies strongly across layers. XKV exploits this by computing, for each layer, the attention-score distribution over the input relative to a small observation window at the end of the context, then solving for the per-layer cache sizes that maximize the average Importance Retention Ratio under a total-memory budget, or minimize total memory under a target ratio. The paper argues this greedy allocation is globally optimal, and reports that the resulting personalized budgets beat static layer-uniform and pyramid-shaped competitors on accuracy at equal compression, and reach lower memory than the best competitor at extreme compression ratios.

Load-bearing premise

The whole scheme rests on the assumption that maximizing the average importance-retention ratio across layers is equivalent to maximizing end-to-end inference accuracy, so a good cache allocation can be chosen from attention-score statistics alone.

Editorial extensions

If this is right

  • A fixed total KV-cache budget can support longer contexts or larger batches if the budget is shifted toward layers whose attention distribution is sparse and away from layers where extra tokens barely raise retention.
  • Prefill-stage eviction becomes practical with observation-window attention, so the peak memory spike that currently dominates long-context inference can be cut before generation starts.
  • Because allocation patterns are similar across tasks of the same type, a small sampled set of tasks can calibrate the per-layer budgets for an entire application without per-query overhead.
  • XKV's layer-wise allocation is orthogonal to quantization and merging techniques, so its memory savings should stack with those approaches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test transfer across model scales, but the DDID pattern suggests the allocation learned on a small model could initialize a larger model's budgets, cutting the mini-prefill cost further.
  • A serving system could use the same retention-ratio statistic as a live monitor: when average retention drifts, re-run sampling; that operational use is implicit in the paper's sampling result.
  • Because importance is measured only through attention scores, an obvious next experiment is to check whether value-vector norms or attribution scores change which layer-specific budgets are optimal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper describes XKV, a layer-personalized KV cache eviction scheme for long-context LLM inference. XKV runs a mini-prefill pass to collect per-layer attention score distributions, uses those statistics to solve a cache-allocation optimization over layers (via an adaptive greedy algorithm it claims is globally optimal), and then evicts tokens in each layer according to the allocated budget before the actual prefill proceeds. The central premise is that the average importance-retention ratio R_avg across layers is a faithful proxy for end-to-end accuracy, so maximizing R_avg under a memory budget should preserve accuracy. The authors report on LongBench with Llama-3.1-8B that XKV cuts KV cache memory by 61.6% on average and improves throughput by up to 5.2x compared with FullKV.

Significance. If the central claims hold, XKV is a practical contribution to KV-cache management for long-context inference, specifically addressing the prefill stage that often determines peak memory. The direction of personalizing per-layer budgets based on observed attention statistics is sensible and complementary to quantization and cache merging. The empirical evaluation covers five task families and 14 LongBench datasets, and the reported memory savings are large. However, the theoretical validation—the equivalence between R_avg and accuracy, and the global optimality of the allocation algorithm—is not rigorously established; the paper currently asserts rather than proves these connections. The empirical results are suggestive, but the theoretical section will need rewriting or substantive hedging before the paper can be accepted.

major comments (5)
  1. [Sec. III-B, Eqs. (7)-(8)] The proportionality a1 ∝ R1 and a2 ∝ R2 is asserted without derivation. a_i is a vector-valued attention output, a weighted sum of value vectors, while R_i is a scalar ratio of sums of attention weights. A vector cannot be proportional to a scalar unless the value vectors are collinear, which is not stated or plausible. This step is load-bearing because it leads to a1 + a2 ∝ R1 + R2 and hence to the separable objective in Eq. (17). Please either provide a rigorous derivation (e.g., in expectation or under a norm) or explicitly label this as a heuristic.
  2. [Sec. III-B, Eq. (13)] The approximation from Eq. (11) to Eq. (13) replaces log2(n2)/log2(n2 - 1) by 1, which is accurate only for large n2. The experimental operating regime includes per-layer budgets around 100 tokens (e.g., compression ratios of 1.2-1.6% on 8192-token inputs), where this approximation can introduce non-negligible error. The authors should quantify the error or avoid the approximation.
  3. [Sec. III-B, paragraph after Eq. (16)] The derivation assumes that the observation-window inputs X^2_ows in Eq. (6) do not depend on the output of Eq. (5). In an actual transformer, layer 2 receives the output of layer 1, so this independence is false. The paper's claim that this is 'a characteristic of the token selection process during the prefill stage' is not explained. Since the independence is used to justify the additive separability of R across layers, it should be justified analytically or abandoned in favor of an explicit approximation.
  4. [Sec. IV-B, Algorithm 1 proof] The optimality proof in Sec. IV-B considers only a two-layer model and a single-token swap, and it assumes the greedy inequality Sort(w1)[-(n1+1)]/Sum(w1) > Sort(w2)[-n2]/Sum(w2) for the selected pair. This does not establish global optimality for L layers, nor does it cover the constraint in Eq. (18) (minimizing total size for a given R_avg). The claim that Algorithm 1 computes a 'globally optimal' allocation is therefore not supported. Please restate the guarantee as local exchange optimality for the R_avg objective and validate the L-layer behavior experimentally.
  5. [Sec. III-B, Fig. 5] The relationship between R_avg and end-to-end accuracy is the central premise of the optimization, but Fig. 5 provides only a single scatter plot without error bars, without a description of how the allocation points were generated, and without counterfactual allocations. The paper should provide more evidence for this link (e.g., ablations over several allocation strategies across datasets) or explicitly present the R_avg proxy as a heuristic with supporting evidence.
minor comments (5)
  1. [Abstract vs. Sec. V-D] Abstract reports throughput improvement 'up to 5.5x' while the evaluation section (Sec. V-D, Table IV) reports 5.2x; these numbers should be reconciled.
  2. [Sec. II-C] Sec. II-C uses 'coarsen-grained' twice; it should be 'coarse-grained'.
  3. [Sec. III-C] Sec. III-C contains 'strateg' which should be 'strategy'.
  4. [Sec. V-D] Sec. V-D refers to 'our XVK' near the end of the section; it should be 'XKV'.
  5. [Algorithm 1, line 13] Algorithm 1, line 13 uses the slice 'wi[-(pos+1):pos]' with a negative start and a positive stop; this is ambiguous and in standard Python slicing would typically yield an empty result. The pseudocode should be rewritten to select the next token by contribution in a clear and correct way.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: XKV's allocation is derived from input attention statistics and validated on independent LongBench accuracy, so no prediction reduces to its own inputs.

full rationale

The paper's central optimization objective (Eq. 17) maximizes the average importance retention ratio R_avg, which is defined directly from per-layer attention score distributions w_i (Eqs. 3-4). These w_i are computed from the input's attention matrices during mini-prefill; no downstream accuracy labels or task-specific scores enter the objective or the allocation algorithm. The resulting allocation is then evaluated against held-out LongBench accuracy under fixed compression ratios, and the reported memory reduction (61.6%) and throughput gain (5.2x) are measured outcomes rather than restatements of the optimized quantity. The most serious weakness is the theoretical bridge from R_avg to end-to-end accuracy: Eqs. (7)-(8) assert a proportionality between the scalar retention ratio and the vector-valued attention output, and the layer-independence assumption is stated but not proven. However, this is an unsupported proxy or correctness risk, not a circular reduction: the paper does not define accuracy in terms of R, nor does it fit R to accuracy labels, nor does it rely on a load-bearing self-citation to justify the proxy. Figure 5 gives only an empirical correlation, but using an imperfect proxy is not the same as making the prediction equivalent to its input. No specific equation or passage exhibits a construction in which the claimed prediction is forced by definition or by a self-citation chain, so the appropriate circularity finding is none.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central claim rests mainly on attention-persistence and observation-window assumptions from prior work, plus the paper's own assertion that R_avg drives accuracy. The greedy allocation is parameter-free except for hand-set ows, pooling size, sampling ratio, and the undefined δ threshold. No new physical or algorithmic entities are invented; DDID is an observed pattern and R_i is a defined metric.

free parameters (4)
  • observation window size (ows) = 8
    Hand-set default in Sec. V-A; controls which tokens are used as the reference for importance scoring. No sensitivity analysis is provided.
  • pooling size (ps) = 7
    Hand-set default in Sec. V-A; determines the granularity of the attention distribution vector after averaging. No ablation is provided.
  • sampling ratio = 10%
    Hand-set fraction of tasks used for offline sampling in Sec. IV-C; trades overhead against adaptation quality. No ablation is provided.
  • ISR threshold δ = unspecified
    Eq. (10)-(16) define f(n_i)=δ but no concrete value or algorithm use is given, making the ISR-based argument qualitative.
assumptions (7)
  • domain assumption Attention persistence hypothesis: tokens important during prefill remain important during generation.
    Invoked in Sec. III-B to extend prefill eviction choices to later decoding; if false, early eviction can remove tokens needed later.
  • domain assumption The last few tokens in the observation window are a valid reference for first-token prediction during prefill.
    Used in Eq. (3) to define w_i; based on SnapKV and related work, but assumed true for all tasks and layers.
  • domain assumption End-to-end inference accuracy is monotonically related to R_avg, the average per-layer importance retention ratio.
    Stated as a hypothesis in Sec. III-B and supported only by Fig. 5; it is the bridge between the optimization objective in Eqs. (17)-(18) and downstream quality.
  • ad hoc to paper The post-eviction attention output at a layer is proportional to the scalar R_i (Eq. (8), a2 ∝ R2).
    This proportionality is asserted rather than derived; it underlies the theoretical validation of DDID and the claimed equivalence between R and accuracy.
  • ad hoc to paper The approximation log2(n2-1)/log2(n2) ≈ 1 for cache sizes n2 is valid.
    Used in Eq. (13) to simplify the ISR difference argument; no error bound is given.
  • domain assumption Inference tasks of the same type have similar cross-layer allocation distributions, so a sampled average allocation generalizes.
    Basis for the sampling strategy in Sec. IV-C, supported by Fig. 10 on QA tasks only.
  • domain assumption Mini-prefill without caching has acceptable overhead.
    Sec. IV-A calls mini-prefill lightweight, but no overhead measurements for this phase are reported in the efficiency experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference." pith.science (2026). https://pith.science/paper/6GLTI54Z

@misc{pith2026241205896,
  author       = {Pith},
  title        = {Pith review of: XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GLTI54Z}},
  note         = {Machine review of arXiv:2412.05896}
}
read the original abstract

Recently the generative Large Language Model (LLM) has achieved remarkable success in numerous applications. Notably its inference generates output tokens one-by-one, leading to many redundant computations. The widely-used KV-Cache framework makes a compromise between time and space complexities. However, caching data generates the increasingly growing memory demand, that can quickly exhaust the limited memory capacity of the modern accelerator like GPUs, particularly in long-context inference tasks. Existing studies reduce memory consumption by evicting some of cached data that have less important impact on inference accuracy. But the benefit in practice is far from ideal due to the static cache allocation across different LLM network layers. This paper observes that the layer-specific cached data have very different impacts on accuracy. We quantify this difference, and give experimental and theoretical validation. We accordingly make a formal analysis and shows that customizing the cache size for each layer in a personalized manner can yield a significant memory reduction, while still providing comparable accuracy. We simulate the cache allocation as a combinatorial optimization problem and give a global optimal solution. In particular, we devise a mini- and sampling-based inference over a lightweight variant of the LLM model, so as to quickly capture the difference and then feed it into the personalized algorithms. Extensive experiments on real-world datasets demonstrate that our proposals can reduce KV cache memory consumption by 61.6% on average, improve computational efficiency by 2.1x and then increase the throughput by up to 5.5x.

Figures

Figures reproduced from arXiv: 2412.05896 by the authors.

Figure 1
Figure 1. Comparison on memory usage respectively for model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The heatmap of the attention score in the original input sequence, instead of the generated output token Consequently, it is not reasonable to directly utilize the Wattn-based importance of tokens in the input sequence to infer the first new token. However, several studies [18], [19], [23], [36] have demonstrated that the last few tokens in a long-context input sequence are critical for inferring the first new token… view at source ↗
Figure 3
Figure 3. The variation of R across layers under different KV cache sizes x 2 t+1 = F (a2), (6) a2 = f  sof tmax  X2 owsW2 QW2 K T X2 t−ows T  X2 t−ows T W2 V W2 O X1 ows ∈ R ows×d denotes the set of observation window tokens in the first layer, ows is the size of the observation window and d is the dimension of the token vector. Accordingly, X1 t−ows ∈ R (t−ows)×d represents the set of tokens excluding the obser￾vation w… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The variation of KV cache size across layers when the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The relationship between Ravg and inference results R1+R2, R1 (n1 + 1)−R1 (n1)+R2 (n2 − 1)−R2 (n2) > 0, which indicates that there must exist an allocation {n1 + 1, n2 − 1} that is superior to the original allocation{n1, n2} for prefill phase. Combined with the proven …
Figure 6
Figure 6. Figure 6: The inference procedures under XKV C. DDID-driven Memory Reduction For some layers where Ri is highly sensitive to ni , even with a small ni , we can still obtain a considerably large Ri , leading to a significant increase in Ravg. For such layers, it is worthwhile to …
Figure 7
Figure 7. Figure 7: The architecture of mini-prefill pooling size select merge pooling 𝑊𝑎𝑡𝑡𝑛 𝑤 𝑜𝑤𝑠 𝑜𝑤𝑠 [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The attention processing procedure Mini-prefill is a computation-intensive yet lightweight mod￾ule, with its architecture illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Illustration of procedures in a single iteration [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: The KV cache size of each layer in QA tasks [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Comparison on memory reduction under different accuracy loss bounds [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Comparison on inference accuracy under different memory constraints [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , vol. 2, no. 3, p. 6, 2023

    2023

  3. [3]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    arXiv 2023

  4. [4]

    Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model,

    A. Fabbri, I. Li, T. She, S. Li, and D. Radev, “Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , Jan 2019. [Online]. Available: http://dx.doi.org/10.18653/v1/p19-1102

    doi:10.18653/v1/p19-1102 2019

  5. [5]

    Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages,

    W. Yang, C. Li, J. Zhang, and C. Zong, “Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages,” arXiv preprint arXiv:2305.18098 , 2023

    arXiv 2023

  6. [6]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  7. [7]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  8. [8]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

Show all 37 references
  1. [9]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: US...

    2024

  2. [10]

    Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee, “Taming {Throughput-Latency} tradeoff in {LLM} inference with {Sarathi-Serve},” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , 2024, pp. 117–134

    2024

  3. [11]

    Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,

    B. Lin, T. Peng, C. Zhang, M. Sun, L. Li, H. Zhao, W. Xiao, Q. Xu, X. Qiu, S. Li et al. , “Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache,” arXiv preprint arXiv:2401.02669, 2024

    2024 arXiv

  4. [12]

    Cam: Cache merging for memory-efficient LLMs inference,

    Y . Zhang, Y . Du, G. Luo, Y . Zhong, Z. Zhang, S. Liu, and R. Ji, “Cam: Cache merging for memory-efficient LLMs inference,” in Forty-first International Conference on Machine Learning , 2024. [Online]. Available: https://openreview.net/forum?id=LCTmppB165

    2024

  5. [13]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks,

    Z. Wang, B. Jin, Z. Yu, and M. Zhang, “Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks,”arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  6. [14]

    Gear: An efficient kv cache compression recipefor near- lossless generative inference of llm,

    H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao, “Gear: An efficient kv cache compression recipefor near- lossless generative inference of llm,” arXiv preprint arXiv:2403.05527 , 2024

    2024 arXiv

  7. [15]

    Pqcache: Product quantization-based kvcache for long context llm inference,

    H. Zhang, X. Ji, Y . Chen, F. Fu, X. Miao, X. Nie, W. Chen, and B. Cui, “Pqcache: Product quantization-based kvcache for long context llm inference,” arXiv preprint arXiv:2407.12820 , 2024

    2024 arXiv

  8. [16]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=NG7sS51zVF

    2024

  9. [17]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. R´e, C. Barrett et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  10. [18]

    Snapkv: Llm knows what you are looking for before generation,

    Y . Li, Y . Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen, “Snapkv: Llm knows what you are looking for before generation,” arXiv preprint arXiv:2404.14469 , 2024

    2024 arXiv

  11. [19]

    PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference,

    D. Yang, X. Han, Y . Gao, Y . Hu, S. Zhang, and H. Zhao, “PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference,” in Findings of the Association for Computational Linguistics ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, Eds., Aug. 2024

    2024

  12. [20]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,

    Z. Liu, A. Desai, F. Liao, W. Wang, V . Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava, “Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Gl...

    2023

  13. [21]

    Model tells you what to discard: Adaptive KV cache compression for LLMs,

    S. Ge, Y . Zhang, L. Liu, M. Zhang, J. Han, and J. Gao, “Model tells you what to discard: Adaptive KV cache compression for LLMs,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=uNrFpDPMyo

    2024

  14. [22]

    Loogle: Can long- context language models understand long contexts?

    J. Li, M. Wang, Z. Zheng, and M. Zhang, “Loogle: Can long- context language models understand long contexts?” arXiv preprint arXiv:2311.04939, 2023

    2023 arXiv

  15. [23]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling,

    Y . Zhang, B. Gao, T. Liu, K. Lu, W. Xiong, Y . Dong, B. Chang, J. Hu, W. Xiao et al. , “Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling,” arXiv preprint arXiv:2406.02069, 2024

    2024 arXiv

  16. [24]

    Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting,

    X. Nie, X. Miao, Z. Yang, and B. Cui, “Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting,” in Proc. of ICDE, 2022, pp. 2615–2628

    2022

  17. [25]

    Het-gmp: A graph-based system approach to scaling large embedding model training,

    X. Miao, Y . Shi, H. Zhang, X. Zhang, X. Nie, Z. Yang, and B. Cui, “Het-gmp: A graph-based system approach to scaling large embedding model training,” in Proceedings of the 2022 International Conference on Management of Data , 2022, pp. 470–480

    2022

  18. [26]

    Platod2gl: An efficient dynamic deep graph learning system for graph neural net- work training on billion-scale graphs,

    X. Huang, D. Lin, W. Huang, S. Sun, J. Wen, and C. Chen, “Platod2gl: An efficient dynamic deep graph learning system for graph neural net- work training on billion-scale graphs,” in 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024, pp. 2421–2434

    2024

  19. [27]

    Optimizing tensor programs on flexible storage,

    M. Schleich, A. Shaikhha, and D. Suciu, “Optimizing tensor programs on flexible storage,” Proceedings of the ACM on Management of Data , vol. 1, no. 1, pp. 1–27, 2023

    2023

  20. [28]

    You only cache once: Decoder-decoder architectures for language models,

    Y . Sun, L. Dong, Y . Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei, “You only cache once: Decoder-decoder architectures for language models,” arXiv preprint arXiv:2405.05254 , 2024

    2024 arXiv

  21. [29]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y . S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,” arXiv preprint arXiv:2401.18079, 2024

    2024 arXiv

  22. [30]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache,

    Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V . Braverman, B. Chen, and X. Hu, “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,” arXiv preprint arXiv:2402.02750 , 2024

    2024 arXiv

  23. [31]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning . PMLR, 2023, pp. 31 094–31 116

    2023

  24. [32]

    Efficient sparse attention needs adaptive token release,

    C. Zhang, L. Zou, D. Luo, X. Luo, Z. Li, M. Tang, and C. Li, “Efficient sparse attention needs adaptive token release,” in Findings of the Association for Computational Linguistics ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, Eds., Aug. 2024

    2024

  25. [33]

    Alisa: Accelerating large lan- guage model inference via sparsity-aware kv caching,

    Y . Zhao, D. Wu, and J. Wang, “Alisa: Accelerating large lan- guage model inference via sparsity-aware kv caching,” arXiv preprint arXiv:2403.17312, 2024

    2024 arXiv

  26. [34]

    Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters,

    Z. Guo, H. Kamigaito, and T. Watanabe, “Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters,” arXiv preprint arXiv:2406.12335 , 2024

    2024 arXiv

  27. [35]

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference,

    Y . Feng, J. Lv, Y . Cao, X. Xie, and S. K. Zhou, “Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference,” 2024. [Online]. Available: https://arxiv.org/abs/2407.11550

    2024 arXiv

  28. [36]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

    2024

  29. [37]

    LongBench: A bilingual, multitask benchmark for long context understanding,

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li, “LongBench: A bilingual, multitask benchmark for long context understanding,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.