Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LeoAM claims that long-context LLM inference on a single commodity GPU can be accelerated 3.46x on average by adaptive, importance-aware KV cache management that ranks disk-resident chunks from tiny abstracts instead of loading full…

desk verdict Useful systems paper with a genuinely new adaptive chunking and KV-abstract design, but the core bound on chunk importance is unproven and the speedup claim rests on unaudited baselines. read the letter →

arxiv 2506.20187 v2 pith:3GSK2LC3 submitted 2025-06-25 cs.OS cs.CR

He Sun , Li Li , Mingjun Xiao , Chengzhong Xu This is my paper
classification cs.OScs.CR
keywords long-contextLLMinferenceKVcachemanagementcommodityGPUimportance-awaresparsitydiskoffloadingadaptivechunkingcompressionpipelined
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LeoAM claims to make long-context LLM inference practical on a single commodity GPU by treating the key-value cache as a three-tier resource: GPU memory, CPU memory, and disk. The paper argues that token importance is query-dependent and highly skewed, with long contiguous stretches of nearly irrelevant tokens, so evaluating importance on every token wastes both compute and disk bandwidth. LeoAM instead partitions the KV cache into variable-sized chunks, adaptively merging and splitting them, and stores a tiny abstract of each chunk on disk so importance can be judged without loading the full KV data. Across long-context benchmarks the system reports an average 3.46x inference-latency speedup over stronger baselines, up to 5.47x at larger batch sizes, with less than a 1% accuracy drop. If correct, this would remove the need for datacenter GPUs for private or local long-context inference.

What carries the argument

The load-bearing mechanism is the chunk-tree plus the lightweight KV abstract. For each chunk of key-value data stored on disk, LeoAM keeps only the element-wise maximum and minimum key vectors (two vectors per chunk, roughly $2/n'$ of the chunk's data) and treats the query's dot product with those extremes as upper and lower bounds on any token's contribution inside the chunk. The importance-aware adaptive manager maintains chunks in a priority queue ordered by these bounds, splitting high-importance chunks and merging runs of unimportant chunks, so the number of importance evaluations scales with the density of important tokens rather than with context length. The dynamic three-tier pipeline then schedules disk-to-CPU abstract loads, CPU-side importance ranking, and GPU attention computation across layers so that transfer latency is hidden under compute, with a compression ratio $\theta$ chosen so the inequality $T_0 + D(1-\theta) + D\theta\delta/B \le T_c + t(D\theta)$ holds. Together these pieces replace load-all-KV-and-score-it with load-a-tiny-summary, score it, then fetch only the chunks that matter.

What would settle it

Take a long-context prompt from a benchmark such as the paper's evaluation suite, run full-cache inference to record the true per-token attention weights, then run the same prompt through LeoAM and check whether any tokens in the true top 10% of attention lie in chunks the abstract left on disk; if that mismatch is non-negligible on prompts where output quality matters, the central recall claim would fail.

Watch

Extended reading notes

Core claim

LeoAM's central claim is that the disk bottleneck in importance-aware long-context inference can be broken by moving importance evaluation onto compact summaries while making chunk granularity follow the attention landscape. The paper observes an attention desert: at any decoding step, most consecutive chunks of past tokens have near-zero attention weight, and the desert density varies by layer and decoding step. On that basis LeoAM builds tree-structured chunks that split near important tokens and merge across deserts, then stores for each disk-resident chunk only its element-wise maximum and minimum key vectors; with the current query these give upper and lower bounds on the chunk's attention contribution, so the CPU can rank chunks after transferring roughly a fraction $r = \alpha + 2/n'$ of the data instead of all of it. A dynamic three-tier pipeline overlaps disk reads, CPU evaluation, and GPU compute, with compression ratio chosen so transmission hides inside compute time. The reported result is an average 3.46x latency speedup over the better token-level, chunk-level, and prefetch baselines, reaching 5.47x at batch size 8 while keeping accuracy within about 1% of full-cache inference.

Load-bearing premise

Everything rests on the assumption that the per-chunk maximum and minimum key vectors, combined with the current query, give correct upper and lower bounds on the chunk's true attention contribution, so a chunk ranked unimportant by the abstract really is unimportant.

Editorial extensions

If this is right

  • With only abstracts moved from disk, disk bandwidth stops scaling with context length; the cost of importance evaluation becomes nearly independent of how many tokens are stored off-GPU.
  • Because chunk granularity adapts to attention deserts, a single system can use fine chunks where attention is dense and coarse chunks where it is not, which is what lets it beat fixed-chunk and token-level policies.
  • The three-tier pipeline makes larger batches cheaper per token, since transfer and evaluation work is overlapped; at batch size 8 the reported speedup rises to 5.47x.
  • The quality results imply that roughly 10% of KV chunks, chosen query-aware, carry essentially all the information needed for accurate generation on the tested tasks, so long-context inference on a PC is feasible without a datacenter GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The maximum-and-minimum key bound is a heuristic rather than a proven guarantee; one testable extension is to split chunks whose keys have mixed signs before abstracting, which would restore exact bounds at a small memory cost.
  • Attention-desert density could itself be predicted online from the query and recent history, letting the initial chunk size adapt per layer without offline profiling.
  • The same abstract-and-rank pattern should transfer to quantized KV caches or to non-attention score functions such as retrieval-style similarity, which would broaden the method beyond transformer attention.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents LeoAM, an importance-aware long-context LLM inference system for a single commodity GPU that manages the KV cache across GPU, CPU, and disk. The system has three components: an importance-aware adaptive KV manager (IAKM) that partitions KV data into variable-sized chunks organized in a tree and merges/splits chunks according to attention-desert statistics; a lightweight KV abstract (LKA) that stores per-chunk elementwise max/min key vectors so that importance evaluation can be done without loading full KV data from disk; and a dynamic three-tier pipeline (DTP) that overlaps transmission and computation and dynamically compresses transferred KV data. The paper reports an average inference latency speedup of 3.46x over its baselines, up to 5.47x at batch size 8, and less than a 1% accuracy drop on four short-answer tasks. The implementation is built on FlexGen.

Significance. If the claims hold, LeoAM addresses a real and timely bottleneck: making long-context LLM inference practical on personal hardware with limited GPU and CPU memory, where disk I/O becomes the dominant cost. The paper's main strengths are its system-level decomposition of the problem, the adaptive chunk-management idea, the lightweight abstract mechanism for disk-resident KV data, and the explicit latency breakdown showing that each component contributes. However, the central performance and quality claims are measured rather than derived, and the measurement layer has important gaps: no error bars, no code release, baseline implementations that are not independently validated, and quality evaluation only on short-context tasks. The correctness of the LKA bounds is also left unproved. The contribution is therefore plausible and potentially valuable, but the evidence as presented is not yet sufficient to establish the headline numbers.

major comments (4)
  1. [§4.3, Figure 11] The LKA mechanism claims that per-chunk elementwise maximum and minimum key vectors, combined with the current query, yield valid upper and lower bounds on the chunk's attention contribution. This is not generally true without specifying the exact formula. For a raw dot product q·k over a chunk, a valid upper bound requires a sign-aware per-dimension combination: for each dimension j, use q_j * max_k_j when q_j > 0 and q_j * min_k_j when q_j < 0. Simply taking q·max_key, or even max(q·max_key, q·min_key), underestimates the true maximum on mixed-sign queries; for example, with q=[1,-1] and chunk keys [10,-10] and [-10,10], q·max_key = 0 while the true maximum q·k is 20. The paper does not state which formula is used. Moreover, the token importance metric is defined in §4.1 as a softmax attention weight, whose denominator is global across all tokens; a per-chunk bound on q·k does not, by itself, bound the softmax attention weight or the cumulative importance used by IAKM's tree pruning in §4.2. If LKA under-bounds important chunks, IAKM can merge or discard those chunks, and the resulting quality degradation is not captured by the reported experiments. This is a load-bearing correctness gap that needs a precise statement of the bounded quantity and a proof of the bound.
  2. [§6.1, §6.2, Figure 14] The claim of 'maintaining comparable LLM response quality' is supported only by accuracy measurements on COPA, RTE, PIQA, and OpenBookQA, which are short-context tasks. The LongBench and PG-19 datasets are used only for latency/throughput experiments, not for quality evaluation. Since LeoAM's IAKM and LKA mechanisms specifically target disk-resident, long-context KV chunks, the key quality question is whether important KV chunks are correctly identified in long contexts; a <1% accuracy drop on four short-answer tasks does not answer that question. At minimum, the paper should report perplexity on PG-19 and/or accuracy on LongBench tasks under the same offloading configuration, with and without the LKA/IAKM mechanisms, to show that the speedup does not come from unmeasured loss of important KV data.
  3. [§6.2, Figure 15] The central speedup claim of 3.46x average and 5.47x at batch size 8 is based on comparisons to author-implemented baselines (H2O-like, H2O-chunked, Prefetch-based) with no error bars, no variance information, and no code release. Latency measurements on a system with disk I/O and PCIe transfers are inherently noisy, and the figure does not indicate the number of runs or the spread across runs. Without this information, and without an independent or released baseline implementation, the reader cannot assess whether the reported speedups are stable or whether they depend on specific choices in the baseline implementations. The paper should either release the code and baselines, or report mean and standard deviation over multiple runs for each configuration.
  4. [§4.2, §6.1, §6.4] Several key system parameters are chosen via sensitivity analysis on the same benchmark families used for the main evaluation: the default chunk size of 64 tokens (Figure 18), the importance rate of 10%, the early-layer chunk size of 8, and the 50% early-layer retention ratio. Since these parameters directly control how much KV data is transferred from disk, selecting them on the same datasets risks inflating the reported speedups. There is no held-out validation set or a clear separation between parameter exploration and final evaluation. The authors should either report results for a range of these parameters on the final benchmark sets, or justify that the selected defaults are not the result of per-dataset tuning.
minor comments (4)
  1. [Abstract and Conclusion] The abstract contains the sentence 'Experimental results demonstrate that LongInfer achieves...', which appears to refer to the system under a different name; the conclusion also ends with 'commodity CPU' where the context requires 'commodity GPU'.
  2. [§4.2, Eq. (2)] Equation (2) defines f(m) but the text refers to A(m), and the derivation of the optimal chunk count from the difference expression in Eq. (3) is informal; the symbols m, n, and rho(l) should be defined consistently, and the 'approximately determine' step should be made rigorous or explicitly stated as a heuristic.
  3. [§6.3, Figures 16 and 17] The text introduces the ablation order as '+LKA' then '+IAKM', but the figure bars appear to be ordered '+IAKM' then '+LKA'; the caption and legend should be harmonized so that the contribution of each component is unambiguous.
  4. [§6.1] The description of the LongBench sampling says 'a variety of prompts' are sampled, but no details are given about prompt lengths, number of samples per task, or which tasks are used; this information is necessary for reproducibility of the latency experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 3.46x/5.47x speedup and sub-1% accuracy-drop claims are measured against external baselines, not derived from the paper's own definitions; the LKA upper/lower-bound gap is a correctness concern, not a circular step.

full rationale

LeoAM's central claims are empirical. Section 6.2 reports inference latency for LeoAM versus H2O-like, H2O-chunked, and Prefetch-based baselines across LongBench/PG-19 and batch sizes, and Figure 14 reports accuracy on four external tasks against H2O-like and Full Cache. These are measurements, so there is no equation-level reduction of the speedup to the inputs. The LKA abstraction (Sec 4.3, Fig 11) stores elementwise max/min key vectors and claims they yield chunk upper/lower bounds; the paper gives no proof and, for softmax attention with mixed-sign queries, the bound is not guaranteed. That is an unverified soundness assumption that could invalidate the quality claim, but it is not circular: the bound is not defined as the attention output, nor is the quality result derived from it. Similarly, Eq (2)-(3) choose initial chunk size m from an offline density rho(l); the source of rho(l) is not disclosed, and default values (chunk 64, importance 10%, early-layer chunk 8, 50% retention) are selected by sensitivity analysis on the same benchmark families. This creates a tuning/overfitting risk for the absolute speedup, but the speedup itself is a measured comparison at those settings, not a fitted quantity renamed as a prediction. The paper contains no load-bearing self-citations: references [28], [49], [65] etc. are external prior systems, and the authors do not invoke their own prior uniqueness theorem or ansatz. No circular step meets the standard of Eq. X = Eq. Y by construction or parameter-as-prediction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. KV abstracts and tree chunk structures are software representations, not independent entities. The free parameters are design choices or unfitted inputs that the performance and accuracy claims depend on.

free parameters (6)
  • initial chunk size = 64 tokens
    Selected via sensitivity analysis in Section 6.4; default value is tuned to minimize latency on the benchmark models.
  • importance rate = 0.1 (10%)
    Default fraction of KV retained as important; varied in accuracy experiments and not derived.
  • early-layer chunk size = 8 tokens
    Set for the first two layers and early decoding steps based on observed desert rate; Section 6.1.
  • early-layer retention ratio = 0.5 (50%)
    First two transformer layers load 50% of the KV cache by default; Section 6.1.
  • layer-wise importance density rho(l) = not reported
    Input to Eq. (2) for choosing the number of chunks; sourced from 'prior offline knowledge' without stating how it was measured or on which data.
  • KV compression ratio delta = not reported
    The dynamic compression scheme in Section 4.4 requires delta and hardware timings; no calibrated values or measurement procedure are reported.
assumptions (5)
  • domain assumption Attention weight sum is a valid token-importance proxy for KV selection.
    Adopted from H2O [65] and used throughout Section 4.1; not revalidated against output quality beyond aggregate accuracy.
  • domain assumption Long-context attention sparsity forms contiguous deserts that can be merged and split without losing important tokens.
    Insights 1 and 2 in Section 3.5 are demonstrated on LongChat-7B with LongBench samples and generalized to all models and datasets.
  • ad hoc to paper Elementwise max/min keys give valid upper and lower bounds for chunk attention.
    Section 4.3 and Figure 11 assert this bound property without proof; it is not generally true for arbitrary signed query and key vectors.
  • domain assumption Sparsity patterns in the first layers and early decoding steps are distinct enough to justify fixed chunk sizes and 50% retention.
    Insight 2 in Section 3.5; the reported defaults in Section 6.1 depend on this being stable across workloads.
  • domain assumption Offline density knowledge rho(l) is available and representative of the inference workload.
    Section 4.2 uses rho(l) to choose m via Eq. (2), but the paper does not describe how this density is obtained or from which data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU." pith.science (2026). https://pith.science/paper/3GSK2LC3

@misc{pith2026250620187,
  author       = {Pith},
  title        = {Pith review of: Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GSK2LC3}},
  note         = {Machine review of arXiv:2506.20187}
}
read the original abstract

Advanced Large Language Models (LLMs) have achieved impressive performance across a wide range of complex and long-context natural language tasks. However, performing long-context LLM inference locally on a commodity GPU (a PC) with privacy concerns remains challenging due to the increasing memory demands of the key-value (KV) cache. Existing systems typically identify important tokens and selectively offload their KV data to GPU and CPU memory. The KV data needs to be offloaded to disk due to the limited memory on a commodity GPU, but the process is bottlenecked by token importance evaluation overhead and the disk's low bandwidth. In this paper, we present LeoAM, the first efficient importance-aware long-context LLM inference system for a single commodity GPU with adaptive hierarchical GPU-CPU-Disk KV management. Our system employs an adaptive KV management strategy that partitions KV data into variable-sized chunks based on the skewed distribution of attention weights across different layers to reduce computational and additional transmission overheads. Moreover, we propose a lightweight KV abstract method, which minimizes transmission latency by storing and extracting the KV abstract of each chunk on disk instead of the full KV data. LeoAM also leverages the dynamic compression and pipeline techniques to further accelerate inference. Experimental results demonstrate that LongInfer achieves an average inference latency speedup of 3.46x, while maintaining comparable LLM response quality. In scenarios with larger batch sizes, it achieves up to a 5.47x speedup.

Figures

Figures reproduced from arXiv: 2506.20187 by the authors.

Figure 1
Figure 1. The LLM Inference Workflow and KV Caching. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Long-Context LLM Inference. The left figure shows the growth trend of the supported context length of LLM. The right figure shows the memory footprint of different models and context lengths during inference and the memory capacities of common devices. 2 Background 2.1 Large Language Model Inference Basics The majority of Large Language Models (LLMs) are built on the decoder-only transformer architecture and are com… view at source ↗
Figure 4
Figure 4. The Overhead of Token Evaluation in Token Level: Using OPT-2.7b to evaluate the token importance evaluation latency overhead and computation latency on LongBench. 353412 8 0 10 9 2 14 3 2211 1 18172916 6 4 1933283224 7 30132115262520 5 233127 Chunk ID (Sorted by Chunked Attention Weight) 0.00 0.25 0.50 0.75 1.00 Proportion in Top 20% Important tokens identified correctly Important tokens but not identified as import… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Proportion of Tokens in Top 20% per Chunk/Page: The red part represents the important KV identified correctly; Grey part represents the redundant KV but identified as important. RTX 4060, a low-end commodity GPU. When the context length reaches 150K, the memory require…
Figure 7
Figure 7. Figure 7: The attention desert rate across decoding steps. due to resource constraints, LeoAM still faces many chal￾lenges. 3.4 Challenges Challenge 1: Existing token importance evaluation methods for long-context inputs impose additional significant computation and transmission…
Figure 9
Figure 9. Figure 9: Overview of LeoAM System. Insight 2: The attention desert rate varies across different layers of the LLM and different decoding layers. To gain further observations, we conduct a detailed analy￾sis of the attention desert rate across layers with the same experimental s…
Figure 11
Figure 11. Figure 11: Lightweight KV Abstract: Suppose there are 𝑚′ chunks with 𝑛 ′ chunk size. require 32 importance evaluations resulting in significant evaluation overhead. In the tree-structured KV chunk management, we main￾tain a priority queue, where chunks are approximately or￾dered…
Figure 10
Figure 10. Figure 10: Tree-structured KV chunk management: (a) The orange and red chunks mean the identified important chunks without and with this method. (b) Chunk merge and split operations are the tree form of the tree construction process. which leads to unnecessary computation and tr…
Figure 12
Figure 12. Figure 12: KV Management under LKA: The yellow and blue chunks represent the KV data stored in the CPU/GPU and KV abstract on the Disk, respectively. chunk resizing strategy that adjusts the initial chunk size of KV data based on the token sparsity characteristics at each layer,…
Figure 14
Figure 14. Figure 14: The accuracy of different systems across four datasets and three models under different relative KV Cache sizes. inference process, all KV Cache data was stored in FP16 quantization mode and compressed in INT4. Datasets. First, to evaluate the accuracy preservation of…
Figure 15
Figure 15. Figure 15: The inference latency cooperation with baselines of different systems across two datasets and three batch sizes. O: OPT-6.7B, L:LongChat-7B, P:PG-19, L:LongBench. OPT-6.7b longchat-7b PG-19 0 5 10 15 20 25 Latency(s) H2O-like +IAKM +LKA ALL (a) PG-19. OPT-6.7b longcha…
Figure 16
Figure 16. Figure 16: The individual technique breakdown of latency performance. more pronounced. At a batch size of 8, LeoAM achieves a 5.47× speedup. 6.3 Individual Technique Breakdown We evaluate the latency performance contributions of indi￾vidual techniques using OPT-6.7B and LongChat…
Figure 18
Figure 18. Figure 18: The latency performance between different chunk sizes. KV abstracts continues to decrease. Therefore, considering this trade-off, we select 64 as the default chunk size in our experiments. Batch Size [PITH_FULL_IMAGE:figures/full_fig_p012_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAG

    cs.CR 2026-06 unverdicted novelty 7.0 of 10

    SpliceLeak is the first end-to-end side-channel attack on non-prefix KV cache in RAG, using Step-Wave timing leaks to fingerprint private prompt lengths and extract tokens with up to 100% success using 63 requests per...

  2. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference

    cs.CL 2024-07 accept novelty 6.0 of 10

    Ada-KV is the first head-wise adaptive KV cache budget allocator for LLMs, using a theoretical loss upper bound to allocate eviction differently per attention head and yielding higher quality than uniform methods on l...

Reference graph

Works this paper leans on

75 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [8]

    2025.{IMPRESS}: An{Importance-Informed}{ Multi-Tier} Prefix{KV} Storage System for Large Language Model Inference

    Weijian Chen, Shuibing He, Haoyang Qu, Ruidong Zhang, Siling Yang, Ping Chen, Yi Zheng, Baoxing Huai, and Gang Chen. 2025.{IMPRESS}: An{Importance-Informed}{ Multi-Tier} Prefix{KV} Storage System for Large Language Model Inference. In 23rd USENIX Conference on File and Storage Technologies (FAST 25) . 187–201

    work page 2025

  2. [1]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114– 127

    2024

  3. [2]

    Amey Agrawal, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chao- jie Zhang, Alexey Tumanov, and Esha Choukse. 2024. Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Con- text Length LLM Inference Requests Without Approximations. arXiv preprint arXiv:2409.17264 (2024)

    arXiv 2024

  4. [3]

    Anthropic. 2024. Claude 3 Family Announcement. https://www. anthropic.com/news/claude-3-family. Accessed: 2024-10-18

    work page 2024

  5. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al . 2023. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308.14508 (2023)

    arXiv 2023

  6. [5]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

    arXiv 2020

  7. [6]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al . 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems. 1877–1901

    work page 2020

  8. [7]

    CellStrat. 2023. Real-World Use Cases for Large Language Models (LLMs). Medium (2023). https://cellstrat.medium.com/real-world- use-casesfor-large-language-models-llms-d71c3a577bf2 Accessed: 2024-10-08

    work page 2023

Show all 75 references
  1. [9]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. [n. d.]. LongLoRA: Efficient Fine-tuning of Long- Context Large Language Models. In The Twelfth International Confer- ence on Learning Representations

  2. [10]

    Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2019. Few-shot NLG with pre-trained language model. arXiv preprint arXiv:1904.09521 (2019)

    2019 arXiv

  3. [11]

    Minsik Cho, Mohammad Rastegari, and Devang Naik. [n. d.]. KV- Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation. In Forty-first International Conference on Machine Learn- ing

  4. [12]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113

    2023

  5. [13]

    Daivi. 2024. 7 Top Large Language Model Use Cases And Applica- tions. ProjectPro (March 2024). https://www.projectpro.io/article/large- language-model-use-cases-and-applications/887 Accessed: 2024-10- 08

    2024

  6. [14]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. [n. d.]. Get More with LESS: Synthesizing Re- currence with KV Cache Compression for Efficient LLM Inference. In Forty-first International Conference on Machine Learning

  7. [15]

    Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. 2024. QAQ: Quality Adaptive Quantization for LLM KV Cache. arXiv preprint arXiv:2403.04643 (2024)

    2024 arXiv

  8. [16]

    Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. [n. d.]. LazyLLM: Dynamic Token Prun- ing for Efficient Long Context LLM Inference. In Workshop on Efficient Systems for Foundation Models II@ ICML2024

  9. [17]

    2024.{Cost- Efficient} large language model serving for multi-turn conversations with{CachedAttention}

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024.{Cost- Efficient} large language model serving for multi-turn conversations with{CachedAttention}. In 2024 USENIX Annual Technical Conference (USENIX ATC ...

    2024

  10. [18]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. [n. d.]. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (W...

    2023

  11. [19]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. In The Twelfth International Conference on Learning Representations

    2024

  12. [20]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079 (2024)

    2024 arXiv

  13. [21]

    Mingqiang Huang, Ao Shen, Kai Li, Haoxiang Peng, Boyu Li, and Hao Yu. 2024. Edgellm: A highly efficient cpu-fpga heterogeneous edge accelerator for large language models. arXiv preprint arXiv:2407.21325 (2024)

    2024 arXiv

  14. [22]

    Yuxiang Huang, Binhang Yuan, Xu Han, Chaojun Xiao, and Zhiyuan Liu. 2024. Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads. arXiv preprint arXiv:2410.01805 (2024)

    2024 arXiv

  15. [23]

    Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, and Amir Gholami. [n. d.]. Characterizing Prompt Compression Meth- ods for Long Context Inference. In Workshop on Efficient Systems for Foundation Models II@ ICML2024

  16. [24]

    Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. [n. d.]. MInference: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention. In Workshop on Efficient Systems fo...

  17. [25]

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christo- pher Ré, and Azalia Mirhoseini. 2024. Hydragen: High-throughput llm inference with shared prefixes. arXiv preprint arXiv:2402.05099 (2024)

    2024 arXiv

  18. [26]

    Tim Keary. 2024. 12 Practical Large Language Model (LLM) Applica- tions. Techopedia (January 2024). https://www.techopedia.com/12- 13 Preprint, under review He Sun et al. practical-large-language-model-llm-applications Accessed: 2024-10- 08

    2024

  19. [27]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  20. [28]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. {InfiniGen}: Efficient generative inference of large language models with dynamic{KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . 155–172

    2024

  21. [29]

    Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the eighteenth conference on computational natural language learning . 171–180

    2014

  22. [30]

    Dacheng Li. 2025. LongChat: An Open Framework for Long-Context Language Models. https://github.com/DachengLi1/LongChat

    2025

  23. [31]

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guo- hong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459 (2024)

    2024 arXiv

  24. [32]

    Hwijoon Lim, Juncheol Ye, Sangeetha Abdu Jyothi, and Dongsu Han

  25. [33]

    Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, et al . 2024. Infinite-llm: Efficient llm service for long context with distattention and distributed kvcache. arXiv preprint arXiv:2401.02669 (2024)

    2024 arXiv

  26. [34]

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al

  27. [35]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. 2024. CacheGen: KV Cache Compression and Stream- ing for Fast Large Language Model Serving. In Proceedings of the ACM SIGCOMM 2024 Co...

    2024

  28. [36]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information P...

    2024

  29. [37]

    arXiv preprint arXiv:2409.10516 (2024)

    RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval. arXiv preprint arXiv:2409.10516 (2024)

    2024 arXiv

  30. [38]

    Shi Luohe, Hongyi Zhang, Yao Yao, Zuchao Li, et al . [n. d.]. Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption. In First Conference on Language Modeling

  31. [39]

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al . 2024. Mmlongbench-doc: Benchmarking long-context document under- standing with visualizations. arXiv preprint arXiv:2407.01523 (2024)

    2024 arXiv

  32. [40]

    LMSYS and Hugging Face. 2025. LongChat-7B-v1.5-32k. https:// huggingface.co/lmsys/longchat-7b-v1.5-32k

    2025

  33. [41]

    NousResearch and Hugging Face. 2025. Yarn-Llama-2-13b-128k. https: //huggingface.co/NousResearch/Yarn-Llama-2-13b-128k . Accessed: 2025-01-13

    2025

  34. [42]

    Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. 2025. Instinfer: In-storage attention offloading for cost-effective long-context LLM inference. In Proceedings of the IEEE International Symposium on High-Performanc...

    2025 arXiv

  35. [43]

    NousResearch. 2024. Yarn-Llama-2-13B-128K. https://huggingface. co/NousResearch/Yarn-Llama-2-13b-128k

    2024

  36. [44]

    PrivateGPT. 2024. PrivateGPT. https://privategpt.io/. Accessed: 2024-10-18

    2024

  37. [45]

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507 (2019)

    2019 arXiv

  38. [46]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5 (2023), 606–624

    2023

  39. [47]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. Powerinfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP). ACM. arXiv preprint arXiv:2312.12456

    2024 arXiv

  40. [48]

    Yifan Tan, Haoze Wang, Chao Yan, and Yangdong Deng. 2024. AlignedKV: Reducing Memory Access of KV-Cache with Precision- Aligned Quantization. arXiv preprint arXiv:2409.16546 (2024)

    2024 arXiv

  41. [49]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

  42. [50]

    In International Conference on Machine Learning

    Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning. PMLR, 31094–31116

  43. [51]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  44. [52]

    Lukas Tuggener, Pascal Sager, Yassine Taoudi-Benchekroun, Ben- jamin F Grewe, and Thilo Stadelmann. 2024. So you want your private LLM at home?: a survey and benchmark of methods for efficient GPTs. In 11th IEEE Swiss Conference on Data Science (SDS), Zurich, Switzer- land, 30...

    2024

  45. [53]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. [n. d.]. QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference. In Forty-first International Conference on Machine Learning

  46. [54]

    TheBloke. 2024. Yarn-Llama-2-7B-128K-GGML. https://huggingface. co/TheBloke/Yarn-Llama-2-7B-128K-GGML

    2024

  47. [55]

    Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al

  48. [56]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al . 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv. arXiv preprint arXiv:1910.03771 (2019)

    2019 arXiv

  49. [57]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. In Advances in neural information processing systems. 5998–6008. http://arxiv.org/abs/1706.03762

    2017 arXiv

  50. [58]

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024. Look-m: Look-once optimiza- tion in kv cache for efficient multimodal long-context inference. arXiv preprint arXiv:2406.18139 (2024)

    2024 arXiv

  51. [59]

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for High- throughput LLM Inference. arXiv preprint arXiv:2405.12532 (2024). 14 Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Managemen...

    2024 arXiv

  52. [60]

    CoRR (2024)

    Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. CoRR (2024)

    2024

  53. [61]

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 11608–11620

    2024

  54. [62]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism. In Proceedings of the 28th ACM Symposium on Operating Systems Principles (SOSP) . USENIX Asso...

    2024 arXiv

  55. [63]

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Ao- jun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2024. Think: Thinner key cache by query-driven pruning. arXiv preprint arXiv:2407.21018 (2024)

    2024 arXiv

  56. [64]

    Shiquan Zhang, Ying Ma, Le Fang, Hong Jia, Simon D’Alfonso, and Vassilis Kostakos. 2024. Enabling on-device llms personalization with smartphone sensing. In Companion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing . 186–190

    2024

  57. [65]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. You Only Prefill Once: Combining Cached Knowledge for Large Language Model Serv- ing with CacheFuse. In Proceedings of the 20th European Conference on Compu...

    2025 arXiv

  58. [66]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  59. [67]

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2024. Pqcache: Product quantization-based kvcache for long context llm inference. arXiv preprint arXiv:2407.12820 (2024)

    2024 arXiv

  60. [68]

    Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. 2024. Instruction Backdoor Attacks Against Customized{LLMs}. In 33rd USENIX Security Symposium (USENIX Security 24). 1849–1866

    2024

  61. [69]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. 2024. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294 (2024)

    2024 arXiv

  62. [70]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  63. [72]

    Youpeng Zhao, Di Wu, and Jun Wang. 2024. ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching. In Proceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). ACM/IEEE. arXiv preprint arXiv:2403.17312

    2024 arXiv

  64. [73]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Syste...

    2024

  65. [75]

    Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, et al. 2024. Near-Lossless Acceleration of Long Context LLM In- ference with Adaptive Structured Sparse Attention. arXiv preprint arXiv:2406.15486 (2024). 15

    2024 arXiv

  66. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626

  67. [2024]

    In Proceedings of the ACM SIGCOMM 2024 Conference

    Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs. In Proceedings of the ACM SIGCOMM 2024 Conference. 707–720

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.