Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Krul claims that conversation-adaptive KV cache compression, guided by attention similarity across layers, lets a serving system restore a paused multi-turn chat 1.5x–2.68x faster and with 1.33x–2.35x less cache storage than…

desk verdict Per-conversation KV compression layer selection is a real idea, but the accuracy evaluation is underpowered and the protocol needs to rule out future-input leakage. read the letter →

arxiv 2507.08045 v2 pith:GGKTXBSJ submitted 2025-07-10 cs.CL cs.AI

Junyi Wen , Junyuan Liang , Zicong Hong , Wuhui Chen , Ting Cai , Zibin Zheng This is my paper · ORCID
classification cs.CLcs.AI
keywords KVcachecompressionmulti-turnconversationstaterestorationLLMinferenceattentionsimilaritycross-layersharingtime-to-first-tokenrecompute-loadpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that resuming a paused multi-turn conversation by recomputing the full history or reloading an uncompressed key-value (KV) cache is wasteful, and that fixed cross-layer KV compression is the wrong cure because the best layer pairs to share differ from one conversation to another. Krul instead picks a compression strategy per conversation, using the similarity of attention patterns between model layers, then restores the conversation by overlapping compressed-cache loading with recomputation. The paper reports that this cuts time-to-first-token by 1.5x–2.68x and KV cache storage by 1.33x–2.35x against state-of-the-art baselines, with average accuracy loss under 1%. If this holds, serving systems that juggle many inactive conversations could keep far more history resident and restart turns much faster.

What carries the argument

The central object is the Euclidean distance between flattened attention-weight matrices of layer pairs, used as a similarity score for deciding which layers' KV caches can be shared. It is paired with the I-R (initial-recent) layer pattern, defined as layers whose attention concentrates on the first 10% and last 10% of tokens regardless of user input, which acts as a preemptive filter that excludes input-sensitive layers from compression. This filter is implemented in Algorithm 1: for each layer, the average attention weight directed to the initial and recent token subsets is compared against a threshold gamma, and layers below it are marked non-I-R. Among the remaining I-R layers, the algorithm picks the layer pairs with the smallest attention distance up to a compression ratio r_l, and it avoids using any layer in more than one shared pair. The supporting machinery is the token-wise heterogeneous estimator, which splits attention-similarity computation between CPU (prefill, offloaded asynchronously) and GPU (decoding, per token), and the bubble-free restoration scheduler, which balances recomputation and loading so that computation and I/O streams fully overlap.

What would settle it

Take a long multi-turn history with several facts placed in the middle of the context, ask a later turn about one of those middle facts, and compare Krul's compressed-restoration answer against the answer from full recompute on a model where the reported I-R layer ratio is lower than claimed; if the compressed response loses the fact while full recompute keeps it, the safety of I-R compression fails. A cheaper per-model check is to compute the average attention weight directed to the first and last 10% of tokens across many query-history pairs; if any layer that the selector compresses consistently falls below the threshold gamma on those pairs, the preemptive exclusion rule is violated.

Watch

Extended reading notes

Core claim

Krul's central claim is that a per-conversation compression strategy selected from attention-weight distances can safely share KV caches across layer pairs without degrading future answers, because a large fraction of transformer layers are input-insensitive. The paper identifies three attention patterns, calling the deep layers I-R layers: these layers consistently concentrate attention on the first and last 10% of tokens regardless of user input, so their KV caches can be compressed. Krul preemptively excludes non-I-R layers from compression, then greedily picks layer pairs with the smallest Euclidean distance between their attention-weight vectors, subject to a tunable compression ratio. To make this affordable, it offloads prefill-phase attention weights to the CPU for asynchronous similarity computation while decoding-phase attention is processed per token on the GPU, and it uses a bubble-free scheduler that balances recomputation and loading across layers. Empirical evaluation on LongBench and ShareGPT across LLaMA-7B/30B, Qwen1.5-7B, and Qwen2-72B shows TTFT reductions of 1.5x–2.68x and KV storage reductions of 1.33x–2.35x compared with full reload, fixed compression, and recompute-load baselines, with accuracy close to the uncompressed model.

Load-bearing premise

The load-bearing premise is that layers whose attention concentrates on the first and last tenth of the tokens are input-insensitive in every model and conversation, so compressing their KV caches will not degrade answers to future turns; this is demonstrated with five queries on one 7B model and then assumed to hold across other models and long real-world conversations.

Editorial extensions

If this is right

  • Multi-turn conversation serving can restart a paused conversation without full prefill, cutting time-to-first-token by 1.5x–2.68x against state-of-the-art restoration methods.
  • KV cache storage for inactive conversations drops by 1.33x–2.35x, so a server can keep many more conversations resident in CPU memory before spilling to SSD.
  • Compression is not limited to the latter half of the model: the paper finds I-R layers in the former half (roughly layers 9–24 of LLaMA-7B), so dynamically chosen pairs outperform MiniCache's fixed latter-half adjustment-layer compression.
  • Accuracy stays close to the uncompressed baseline (average loss under 1% on LongBench), because input-sensitive layers are excluded from compression.
  • The recompute-vs-load ratio can be tuned per hardware configuration, allowing the restoration pipeline to adapt to different CPU-GPU bandwidth ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's results, the I-R layer property is likely architecture- and training-dependent, not universal; the paper validates it with five queries on Qwen1.5-7B and then extrapolates to other models, so future models with different attention biases may need an online calibration step rather than a fixed I-R assumption.
  • The paper does not claim that Euclidean distance between attention weights is the optimal proxy for compressibility; a testable extension is to compare it against KL-divergence, mutual information, or the actual task-loss increase from compressing each candidate layer pair.
  • Krul's approach is complementary to token-level eviction and KV quantization, so combining it with H2O-style heavy-hitter eviction or low-bit KV quantization could compound storage savings, though the paper does not explore this combination.
  • The CPU offload of prefill attention weights assumes spare CPU bandwidth; on memory-bandwidth-limited servers the asynchronous similarity computation could become a bottleneck, a system-level condition not directly measured in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Krul, an LLM inference system for multi-turn conversations that restores historical KV caches faster and with less memory by dynamically selecting which layer pairs to compress, based on attention-weight similarity. The system has three main components: a preemptive compression strategy selector that identifies input-insensitive I-R layers and customizes compression per conversation, a token-wise heterogeneous attention similarity estimator that offloads prefill attention computation to the CPU while handling decoding attention on the GPU, and a bubble-free restoration scheduler that overlaps recomputation and loading. The authors report 1.5x-2.68x TTFT reduction and 1.33x-2.35x KV storage reduction versus state-of-the-art baselines, with accuracy evaluated on LongBench tasks. The core idea is that I-R layer attention is input-invariant, making those layers safe to compress, while non-I-R layers are preserved to avoid degrading future-turn quality.

Significance. If validated, the paper makes a useful systems contribution: it replaces fixed layer-pair compression with conversation-adaptive compression, addresses the memory cost of attention similarity computation, and introduces a scheduler that hides restoration latency. The preemptive selector is a plausible response to the real challenge that future user inputs are unavailable at compression time, and the multi-model TTFT and storage experiments are broad. However, the central claim that quality is not compromised rests on an accuracy evaluation whose experimental protocol is ambiguous and whose statistical support is thin; the current evidence is not sufficient to establish the headline claim.

major comments (4)
  1. [§8.1 and Algorithm 1] The accuracy evaluation in §8.1 does not state whether the prompt C passed to Algorithm 1 includes the 128 tokens that simulate the user's input or only the historical prefix. If C is the full context including those 128 tokens, then the I-R layer detection (Algorithm 1 lines 3-8) and the layer-pair similarity computation are performed with access to the future user input, which is precisely the information that Challenge 1 (§2.3) says is unavailable in deployment. Table 4 would then validate an oracle compression selector, not the preemptive selector that the central claim depends on. The authors should specify C explicitly and, if C includes the simulated user input, rerun the accuracy evaluation with attention weights obtained from the historical prefix alone.
  2. [§8.5, Table 4] Accuracy is reported only for Qwen1.5-7B, with no variance, error bars, or significance tests. The average is only 0.24 points below the full-KV baseline (26.70 vs. 26.94), and several tasks drop below the baseline (e.g., gov_report 25.3 vs. 23.24, qasper 19.7 vs. 18.46, trec 65.5 vs. 64.0). Thus the abstract's claim of 'without compromising generation quality' is not established. Please report per-task variance and statistical significance, and include accuracy results for at least one additional model used in the TTFT experiments, since the TTFT claims are made for LLaMA-7B, LLaMA-30B, and Qwen2-72B.
  3. [§4.1, Table 3] The I-R layer pattern is validated through a controlled study with five synthetic queries on Qwen1.5-7B (Figure 5 and Table 2), and then extrapolated to DeepSeek-6.7B, LLaMA-13B, and Qwen2-72B solely by counting I-R layers (Table 3). The safety claim that I-R layers can be compressed without future-turn degradation is model-specific and attention-dependent; the absence of accuracy measurements on those models leaves the generalization claim unsupported. Please either add per-model accuracy checks or restrict the safety conclusions to Qwen1.5-7B.
  4. [Abstract, §8.2, §10] The abstract reports a 1.5x-2.68x TTFT reduction, while the conclusion and §8.2 state 1.28x-2.68x. This is an internal inconsistency in the headline quantitative claim and must be resolved by using one consistent range throughout the paper, with the lower bound justified by the reported experiments.
minor comments (5)
  1. [§6.1, Eq. (1)] Eq. (1) begins with 'ere' instead of 'Here', and the recomputation ratio r_c is not clearly defined before it appears in the optimization; please define r_c before using it in Eq. (1).
  2. [Figure 21] The y-axis label in Figure 21 says 'Latency (s)' while the values (tens) and the surrounding text suggest milliseconds; please correct the units.
  3. [§3] The overview contains a duplicated phrase: 'identifies model layers input-sensitive model layers' should be 'identifies input-sensitive model layers'.
  4. [Algorithm 1, line 6] The expression for avg_weight_sum is notationally ambiguous; please clarify which entries of the attention matrix are summed and how heads and positions are averaged.
  5. [References] Reference [3] is cited as LongBench, but the arXiv identifier 2412.15204 corresponds to LongBench v2; please confirm which benchmark was actually used and cite accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the only concern is an evaluation-protocol ambiguity in Section 8.1, not a demonstrated reduction.

full rationale

The paper's derivation chain is: (i) observe that attention-similarity patterns across layer pairs vary by conversation; (ii) select per-conversation compression pairs from Euclidean distances between attention weights (Algorithm 1); (iii) compress the historical prefix and measure LongBench accuracy. None of these steps is defined in terms of the target accuracy result: the attention weights come from a forward pass over the prompt, not from test labels, and the accuracy evaluation is separate from the strategy-selection procedure. The I-R layer classification is operationalized by attention mass on initial and recent tokens, and the safety of compressing those layers is then tested in Table 4; this is a heuristic risk rather than circularity, since the test accuracy does not feed back into the layer or pair selection. There is no self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. One genuine caveat: Section 8.1 says 'we split the last 128 tokens of the context to simulate the user's input. Then we apply KV caches compression to the remaining context,' but it does not explicitly state which prompt C is passed to Algorithm 1 when computing attention weights. If that forward pass included the last 128 tokens, the strategy would be informed by the simulated future input, and Table 4 would not validate the preemptive claim; however, the text's 'remaining context' wording is consistent with the intended historical-prefix use, and there is no equation or sentence that makes the strategy a function of the future input by construction. That ambiguity is an evaluation-cleanliness concern, not a demonstrated circular step, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

Krul's contribution rests on several unvalidated modeling assumptions: attention distance as a proxy for accuracy, the I-R layer classification, CPU/GPU overlap timing, and a simplified hardware model. These are empirically plausible but not directly measured, and they carry most of the risk in the paper.

free parameters (3)
  • gamma (attention threshold for I-R layer detection) = not reported
    Algorithm 1 line 7 compares average attention to initial/recent tokens against gamma; its value is never given, and the sensitivity of accuracy to it is not studied.
  • compression ratio r_l = 0.5 (half the layers)
    Section 8.5: 'we fix the total shared layer number to half of the total layers, which is the same as Minicache'; no sweep over r_l is reported.
  • recomputation ratio r_c = hardware dependent, tuned via calibration dataset
    Section 6.1: the optimal value is found by 'preparing a calibration dataset and computing multiple model forwards with different r_c'; this is performance fitting to hardware, not a derived quantity.
assumptions (5)
  • domain assumption Euclidean distance between flattened attention matrices predicts the safety of compressing a layer pair.
    Sections 4.2 and 5: the entire strategy search minimizes this distance without an accuracy-based loss function.
  • domain assumption I-R layers, whose attention concentrates on the first 10% and last 10% of tokens, are input-insensitive and safe to compress.
    Algorithm 1 lines 3-8 and Table 3; validated only through the five-query controlled study in Section 4.1.
  • domain assumption Prefill attention similarity computed on the CPU completes before the end of decoding, based on the 6.56x output/input length ratio.
    Sections 5 and 8.6: this is a ShareGPT average, not a guarantee for unusually short model outputs.
  • domain assumption Attention weights can be captured during model forward passes without materially altering model behavior or inference latency.
    Sections 4.2 and 5: the implementation hooks attention outputs, but the overhead of capturing, transferring, and managing these tensors during large-model serving is not fully accounted for.
  • standard math Hardware latencies are modeled as T_C = F(N,L,d)/F_peak and T_L = G(N,L,d)/B_peak.
    Equation (1) in Section 6.1 uses a simple performance model; no validation of the linear scaling assumptions is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing." pith.science (2026). https://pith.science/paper/GGKTXBSJ

@misc{pith2026250708045,
  author       = {Pith},
  title        = {Pith review of: Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGKTXBSJ}},
  note         = {Machine review of arXiv:2507.08045}
}
read the original abstract

Efficient state restoration in multi-turn conversations with large language models (LLMs) remains a critical challenge, primarily due to the overhead of recomputing or loading full key-value (KV) caches for all historical tokens. To address this, existing approaches compress KV caches across adjacent layers with highly similar attention patterns. However, these methods often apply a fixed compression scheme across all conversations, selecting the same layer pairs for compression without considering conversation-specific attention dynamics. This static strategy overlooks variability in attention pattern similarity across different conversations, which can lead to noticeable accuracy degradation. We present Krul, a multi-turn LLM inference system that enables accurate and efficient KV cache restoration. Krul dynamically selects compression strategies based on attention similarity across layer pairs and uses a recomputation-loading pipeline to restore the KV cache. It introduces three key innovations: 1) a preemptive compression strategy selector to preserve critical context for future conversation turns and selects a customized strategy for the conversation; 2) a token-wise heterogeneous attention similarity estimator to mitigate the attention similarity computation and storage overhead during model generation; 3) a bubble-free restoration scheduler to reduce potential bubbles brought by the imbalance of recomputing and loading stream due to compressed KV caches. Empirical evaluations on real-world tasks demonstrate that Krul achieves a 1.5x-2.68x reduction in time-to-first-token (TTFT) and a 1.33x-2.35x reduction in KV cache storage compared to state-of-the-art methods without compromising generation quality.

Figures

Figures reproduced from arXiv: 2507.08045 by the authors.

Figure 1
Figure 1. Overview of Krul’s state restoration. Conversations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The analysis of ShareGPT. storage, compared to state-of-the-art baselines, without compromising generation quality. 2 BACKGROUND AND MOTIVATION This section first introduces the fundamentals of generative LLM inference and explores the inefficiency of multi-turn conversation state restoration, and then discusses the design opportunities for dealing with these inefficiencies and the challenges faced in design￾ing suc… view at source ↗
Figure 4
Figure 4. The attention weight cosine similarity. 0 50 100 150 Token Position 0 25 50 75 100 125 150 Token Position Layer 2 Question: Which continent is the Amazon rainforest located i... 0.00 0.02 0.04 0.06 0.08 0.10 (a) layer 0-2 (I-P layer) 0 50 100 150 Token Position 0 25 50 75 100 125 150 Token Position Layer 6 Question: Which continent is the Amazon rainforest located i... 0.00 0.02 0.04 0.06 0.08 0.10 (b) layer 3-8 (I-… view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: An example of information damage due to KV [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Average attention weight of a single token varying different user in￾puts. 500 1000 2000 3000 4000 Prompt Length 0 5 10 15 20 25 30 KV Storage (GB) KV Cache Attention Weight [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 10
Figure 10. Figure 10: The system architecture of Krul [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 12
Figure 12. Figure 12: Example of a bubble-free restoration pipeline. [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: The overall performance comparison of Krul and other SOTA works. [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Time to first token. LLaMA-7B LLaMA-30BQwen1.5-7BQwen2-72B 0 100 200 300 400 500 600 700 800 Throughput (token/s) RE KV FKC PRE Ours [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 19
Figure 19. Figure 19: KV caches storage varying different prompt lengths. slower TTFT increasing speed compared with the other three meth￾ods. This improvement comes from the KV caches compression and the recomputation-loading pipeline of KV caches restoration. 8.4 KV Caches Storage KV cac…
Figure 21
Figure 21. Figure 21: The comparison of latency of CPU attention sim [PITH_FULL_IMAGE:figures/full_fig_p012_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages

  1. [1]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Solov- eychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems 6 (2024), 114–127

    work page 2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  3. [3]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks. arXiv preprint arXiv:2412.15204 (2024)

    arXiv 2024

  4. [4]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15, 3 (2024), 1–45

    2024

  5. [5]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision . Springer, 370– 387

    work page 2024

  6. [6]

    Sanghyun Choo and Wonjoon Kim. 2023. A study on the evaluation of tokenizer performance in natural language processing. Applied Artificial Intelligence 37, 1 (2023), 2175112

    work page 2023

  7. [7]

    Ju Fan, Zihui Gu, Songyue Zhang, Yuxin Zhang, Zui Chen, Lei Cao, Guoliang Li, Samuel Madden, Xiaoyong Du, and Nan Tang. 2024. Combining small language models and large language models for zero-shot NL2SQL. Proceedings of the VLDB Endowment 17, 11 (2024), 2750–2763

    work page 2024

  8. [8]

    Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. 2022. Mmdialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:2211.05719 (2022)

    arXiv 2022

Show all 48 references
  1. [9]

    Zhe Fu, Mo Sha, Yiran Li, Huorong Li, Yubing Ma, Sheng Wang, and Feifei Li

  2. [10]

    2024.{Cost-Efficient} large language model serving for multi-turn conversations with{CachedAttention}

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024.{Cost-Efficient} large language model serving for multi-turn conversations with{CachedAttention}. In 2024 USENIX Annual Technical Conference (USENIX ATC 2...

    2024

  3. [11]

    Shiwei Gao, Youmin Chen, and Jiwu Shu. 2025. Fast state restoration in llm serving with hcache. In Proceedings of the TwentFheieth European Conference on Computer Systems. 128–143

    2025

  4. [12]

    Mozhdeh Gheini, Xiang Ren, and Jonathan May. 2021. Cross-attention is all you need: Adapting pretrained transformers for machine translation. arXiv preprint arXiv:2104.08771 (2021)

    2021 arXiv

  5. [13]

    Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang

  6. [14]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems 37 (2024), 1270–1303

    2024

  7. [15]

    Shuowei Jin, Xueshen Liu, Qingzhao Zhang, and Z Morley Mao. 2024. Compute or load kv cache? why not both? arXiv preprint arXiv:2410.03065 (2024)

    2024 arXiv

  8. [16]

    Katikapalli Subramanyam Kalyan. 2024. A survey of GPT-3 family large language models including ChatGPT and GPT-4. Natural Language Processing Journal 6 (2024), 100048

    2024

  9. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  10. [18]

    2024.{InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024.{InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) . 155–172

    2024

  11. [19]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  12. [20]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. 2024. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems 37 (2024), 139997–140031

    2024

  13. [21]

    Martin Moller. 2024. Efficient training of feed-forward neural networks. In Neural Network Analysis, Architectures and Applications. CRC Press, 136–173

    2024

  14. [22]

    Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Qiaozhi He, Murun Yang, Tong Xiao, and Jingbo Zhu. 2024. Cross-layer attention sharing for large language models. arXiv preprint arXiv:2408.01890 (2024)

    2024

  15. [23]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining ...

    2020

  16. [24]

    Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2020. How good is your tokenizer? on the monolingual performance of multilingual language models. arXiv preprint arXiv:2012.15613 (2020)

    2020 arXiv

  17. [25]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774 (2024)

    2024 arXiv

  18. [26]

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. 2022. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE transactions...

    2022

  19. [27]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  21. [29]

    Xiaoyang Wang, Chen Li, Jianqiao Zhao, and Dong Yu. 2021. Naturalconv: A chinese dialogue dataset towards multi-turn topic-driven conversation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 14006–14014

    2021

  22. [30]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454 (2024)

    2024 arXiv

  23. [31]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

  24. [32]

    Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. 2024. Kvsharer: Efficient inference via layer-wise dissimilar KV cache sharing. arXiv preprint arXiv:2410.18517 (2024)

    2024 arXiv

  25. [33]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. 2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005 (2025)

    2025 arXiv

  26. [34]

    Gokul Yenduri, M Ramalingam, G Chemmalar Selvi, Y Supriya, Gautam Sri- vastava, Praveen Kumar Reddy Maddikunta, G Deepti Raj, Rutvij H Jhaveri, B Prabadevi, Weizheng Wang, et al. 2024. Gpt (generative pre-trained transformer)– a comprehensive review on enabling technologies, p...

    2024

  27. [35]

    Gokul Yenduri, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, Rutvij H Jhaveri, Weizheng Wang, Athanasios V Vasilakos, Thippa Reddy Gadekallu, et al

  28. [36]

    Wangsong Yin, Mengwei Xu, Yuanchun Li, and Xuanzhe Liu. 2024. Llm as a system service on mobile devices. arXiv preprint arXiv:2403.11805 (2024)

    2024 arXiv

  29. [37]

    Bobbi Yogatama, Weiwei Gong, and Xiangyao Yu. 2024. Scaling your hybrid CPU-GPU DBMS to multiple gpus. Proceedings of the VLDB Endowment 17, 13 (2024), 4709–4722

    2024

  30. [38]

    Lingfan Yu, Jinkun Lin, and Jinyang Li. 2025. Stateful large language model serving with pensieve. In Proceedings of the Twentieth European Conference on Computer Systems. 144–158

    2025

  31. [39]

    arXiv preprint arXiv:2305.10435 (2023)

    Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. arXiv preprint arXiv:2305.10435 (2023)

    2023 arXiv

  32. [40]

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. 2024. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems 37 (2024), 3304– 3331

    2024

  33. [41]

    Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. 2024. Cam: Cache merging for memory-efficient llms inference. In Forty-first International Conference on Machine Learning

    2024

  34. [42]

    Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. 2024. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems 6 (2024), 381–394

    2024

  35. [43]

    Meihui Zhang, Zhaoxuan Ji, Zhaojing Luo, Yuncheng Wu, and Chengliang Chai

  36. [44]

    In 2024 IEEE 40th International Conference on Data Engineering (ICDE)

    Applications and challenges for large language models: From data manage- ment perspective. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 5530–5541

    2024

  37. [48]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  38. [2021]

    Advances in neural information processing systems 34 (2021), 15908–15919

    Transformer in transformer. Advances in neural information processing systems 34 (2021), 15908–15919

    2021

  39. [2023]

    arXiv preprint arXiv:2309.17453 (2023)

    Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  40. [2024]

    Proceedings of the VLDB Endowment 17, 12 (2024), 4413–4416

    EncChain: Enhancing Large Language Model Applications with Advanced Privacy Preservation Techniques. Proceedings of the VLDB Endowment 17, 12 (2024), 4413–4416

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.