Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

A Survey of LLM Inference Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims that the full range of LLM inference techniques can be understood through load prediction, adaptive mechanisms, and cost reduction.

desk verdict Useful survey with a shaky organizing thesis: the claim that all techniques rely on load prediction, adaptivity, and cost reduction is contradicted by the paper's own quality-focused sections. read the letter →

arxiv 2506.21901 v1 pith:FITYQVD5 submitted 2025-06-27 cs.DB

James Pan , Guoliang Li This is my paper · ORCID
classification cs.DB
keywords LLMinferencesystemsautoregressivegenerationKVcacheloadpredictionadaptiveschedulingcontinuousbatchingpagedmemoryquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey tries to establish that the techniques used by large language model (LLM) inference systems, including attention variants, kernels, batching, scheduling, memory management, quantization, and cache reuse, belong to one design space rather than isolated tricks. The paper's unifying claim is that these techniques fundamentally rely on load prediction, adaptive mechanisms, and cost reduction, because autoregressive generation makes request cost unpredictable. It maps the techniques onto a complete system stack of request processing, model optimization and execution, and memory management, then shows how single-replica, disaggregated, and serverless systems combine them. If the claim holds, system designers and researchers gain a common vocabulary for comparing existing engines and for locating open problems such as load prediction and elastic scaling.

What carries the argument

The machinery that carries the argument is the three-part analytic scheme itself: load prediction, adaptive mechanisms, and cost reduction. The scheme is applied inside a three-layer system stack, spanning request processing, model optimization and execution, and memory management, so that every surveyed technique has a place. The concrete object that motivates all three categories is the KV cache, the growing store of key and value vectors for tokens generated so far; its unpredictable growth is why prediction is hard, why adaptation is needed, and why reducing cost pays off.

What would settle it

Apply the survey's three categories to a held-out set of inference techniques outside the ones it discusses, using a coding scheme fixed in advance, and count the techniques that fit none of the three categories or fit them only through severe overlap; a substantial unclassifiable residue would refute the claim that the techniques fundamentally rely on these mechanisms. A simpler probe is to identify one production engine whose main speed-up mechanism is neither predictive, adaptive, nor cost-reducing, and to show that it achieves comparable gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that the defining feature of LLM inference, namely autoregressive generation, in which each output token is produced by rerunning the model over all previous tokens, makes request length, memory footprint, and execution time fundamentally non-deterministic, and that the field's diverse techniques are best understood as three types of response. Load prediction tries to estimate request cost in advance through output-length ranking or memory-usage models. Adaptive mechanisms, such as continuous batching, dynamic scheduling, preemption, page-based allocation, and offloading, react to conditions as they change. Cost reduction lowers the fundamental compute or memory burden through fused kernels, efficient attention variants, quantization, speculative decoding, and cache persistence. The survey reads the literature through this three-part lens and concludes that every surveyed class of technique ultimately depends on one or more of these mechanisms.

Load-bearing premise

The load-bearing assumption is that the three-category taxonomy, load prediction, adaptive mechanisms, and cost reduction, is the exhaustive and accurate explanation of why the surveyed techniques work, an editorial framing the paper does not test against data or experiments.

Editorial extensions

If this is right

  • If the three-mechanism framing is correct, then even very different engines can be compared on common axes: what they predict, what they adapt to, and what cost they reduce.
  • The survey's convergence observation implies that paged attention and continuous batching with chunked prefills will continue to be adopted broadly, while multi-level-queue scheduling and asynchronous cache recovery become routine components.
  • Frontend features such as structured outputs and template completion are not merely usability improvements; by constraining output length they make load prediction easier and enable cache reuse, so they affect the runtime as well.
  • For disaggregated and serverless systems, cache transfer and recovery cost, rather than raw compute or memory, become the dominant constraint that the three mechanisms must address.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the taxonomy is to classify the next wave of inference systems under a pre-registered coding scheme; a substantial residue of techniques fitting none of the three categories would show the framework is only a label, not an analysis.
  • The paper's load-prediction examples mostly predict request-level cost; a sharper design principle would separate request-cost prediction from system-load prediction, since the two can fail independently.
  • Applications with a fixed latency budget, such as test-time scaling, may deliberately spend extra compute for quality; if that behavior becomes central to serving, the three-mechanism lens may need a fourth axis or a budget-aware variant.
  • The taxonomy also suggests a practical design recipe: reduce the dominant cost first, predict only what remains uncertain, and add adaptivity where prediction fails.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This survey reviews techniques and systems for LLM inference, organizing the material into request processing (operators, sampling, sequence generation), model optimization and execution (kernels, batching, scheduling), and memory management (paging, eviction/offloading, quantization, cache persistence). It then discusses how these techniques combine into single-replica, multi-replica, disaggregated, and serverless systems, and provides tables comparing system features. The paper's central thesis is that all surveyed techniques fundamentally rely on three mechanisms — load prediction, adaptive mechanisms, and cost reduction — to handle the uncertainties introduced by autoregressive generation.

Significance. The survey is timely and comprehensive, covering many recent systems (vLLM, SGLang, Mooncake, DeepFlow, SplitWise, DistServe) and providing useful structured comparisons in Tables 1–4. The technical descriptions are generally accurate and well-referenced, with standard mathematical presentations of attention and clear system summaries. If the three-mechanism framework were substantiated, it would be a valuable conceptual lens for comparing LLM inference system designs. However, the universal claim is currently asserted rather than demonstrated, and the survey's own content includes techniques that do not fit the framework, so the significance of the contribution depends on the authors revising and validating the claim.

major comments (3)
  1. [Abstract, Section 1, Section 7] The claim that all surveyed techniques 'fundamentally rely on load prediction, adaptive mechanisms, and cost reduction' is not established by the survey and is contradicted by its own content. Section 2.3 states that structured generation approaches (beam search, tree-of-thoughts, graph-of-thoughts, self-consistency) 'multiply the request processing costs' and are motivated by output quality, which is neither load prediction nor adaptivity nor cost reduction. Section 2.2.3's top-k and nucleus sampling similarly target text diversity rather than system load. Section 5.1's frontend features (constrained generation, template completion, DSPy prompt optimization) are driven by user control and prompt optimization, not by the three mechanisms. The authors should either restrict the claim to the techniques that address memory and compute uncertainty or provide an explicit mapping from each technique family to the three mechanisms, acknowledging those that fall outside.
  2. [Section 4.1, Section 3.2] The three categories are not defined as a partition, and several techniques instantiate more than one mechanism simultaneously, which undermines the promised 'unified framework'. Paged memory (Section 4.1) is both an adaptive allocation mechanism and a cost-reduction technique; chunked prefills (Section 3.2) are both adaptive batching and a way to reduce peak memory. The paper should either define the boundaries of the categories more precisely or explicitly discuss how a technique can realize multiple mechanisms; otherwise the framework risks being a set of labels rather than an analysis.
  3. [Section 5.3, Tables 3–4] The survey's comparison of systems is purely feature-based and does not apply the three-mechanism framework. For example, Tables 3 and 4 list load balancing, job prioritization, batch size, memory model, eviction, and persistence, but never indicate how these choices instantiate load prediction, adaptivity, or cost reduction in each system. Since the framework is the paper's central contribution, the authors should use it to analyze the systems, e.g., by identifying which mechanisms each system emphasizes and how that affects trade-offs.
minor comments (7)
  1. [Abstract] The abstract contains spacing errors (e.g., 'rapidLLMadoptionviaserviceslikeChatGPT') that should be corrected.
  2. [Section 1] 'In particuar allow for' should read 'In particular allow for'.
  3. [Section 3.2] 'Reconsituted' is misspelled (two occurrences); it should be 'reconstituted'.
  4. [Section 3.1.2] 'Implemention' should be 'implementation'.
  5. [Section 5.2.2] The system name is spelled inconsistently as 'Splitwise' in the text and 'SplitWise' in Table 3; please standardize.
  6. [Table 2 and Section 5.1] The citation for LMQL is inconsistent: Table 2 cites [9] while Section 5.1 uses [10]; these should be reconciled.
  7. [Section 7] 'In order maintain high performance' should be 'in order to maintain high performance'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey reports and categorizes published systems; its three-mechanism framing is an interpretive summary, not a derivation that reduces to its own inputs.

full rationale

This is a survey paper with no fitted parameters, no predictions, and no formal derivation chain. The central claim that LLM inference techniques 'fundamentally rely on load prediction, adaptive mechanisms, and cost reduction' (Abstract and Section 7) is an editorial generalization over the surveyed literature, not a result derived from those techniques by construction. The paper does not define any technique in terms of the three mechanisms, nor does it cite the authors' own prior work as the sole justification for the framework. Section 2.3 explicitly describes structured generation as multiplying request processing costs, and Section 5.1 describes frontend features motivated by user control and prompt optimization, so the taxonomy is a stated interpretive lens rather than a self-fulfilling definition. Even if the taxonomy is debatable or incomplete, that is a correctness or framing concern, not circularity. No self-citation is load-bearing, no empirical quantity is predicted from a fitted input, and no known result is renamed as a new derivation. The paper is self-contained as a review, and the honest circularity finding is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is a literature review, so there are no free parameters or invented entities. The main unstated premise is the adequacy of the proposed taxonomy, which is an editorial claim rather than a tested result. Additionally, the survey's value depends on the accuracy of its representations of cited systems.

assumptions (2)
  • ad hoc to paper The three-mechanism framework (load prediction, adaptivity, cost reduction) is an adequate and complete taxonomy of LLM inference techniques.
    Introduced by the authors in the Abstract and Section 7; no evidence is provided that the taxonomy is exhaustive or that the mechanisms are independent.
  • domain assumption The cited system and technique descriptions faithfully reflect the referenced papers.
    The survey relies entirely on secondary descriptions from cited works; any misrepresentation would propagate through the survey.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of LLM Inference Systems." pith.science (2026). https://pith.science/paper/FITYQVD5

@misc{pith2026250621901,
  author       = {Pith},
  title        = {Pith review of: A Survey of LLM Inference Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FITYQVD5}},
  note         = {Machine review of arXiv:2506.21901}
}
read the original abstract

The past few years has witnessed specialized large language model (LLM) inference systems, such as vLLM, SGLang, Mooncake, and DeepFlow, alongside rapid LLM adoption via services like ChatGPT. Driving these system design efforts is the unique autoregressive nature of LLM request processing, motivating new techniques for achieving high performance while preserving high inference quality over high-volume and high-velocity workloads. While many of these techniques are discussed across the literature, they have not been analyzed under the framework of a complete inference system, nor have the systems themselves been analyzed and compared. In this survey, we review these techniques, starting from operators and algorithms for request processing, then moving on to techniques for model optimization and execution, including kernel design, batching, and scheduling, before ending with techniques for memory management, including paged memory, eviction and offloading techniques, quantization, and cache persistence. Through these discussions, we show that these techniques fundamentally rely on load prediction, adaptive mechanisms, and cost reduction in order to overcome the challenges introduced by autoregressive generation and achieve the goals of the system. We then discuss how these techniques can be combined to form single-replica and multi-replica inference systems, including disaggregated inference systems that offer more control over resource allocation and serverless systems that can be deployed over shared hardware infrastructure. We end with a discussion of remaining challenges.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cognitive Load-Aware Inference: A Neuro-Symbolic Framework for Optimizing the Token Economy of Large Language Models

    cs.LG 2025-07 reject novelty 3.0 of 10

    The paper claims that framing LLM inference as cognitive load management reduces token consumption without accuracy loss, but the evidence is under-powered and the 'emergent' decomposition is directly trained.

Reference graph

Works this paper leans on

140 extracted references · 79 canonical work pages · cited by 1 Pith paper

  1. [1]

    Adnan, A

    M. Adnan, A. Arunkumar, G. Jain, P. Nair, I. Soloveychik, and P. Kamath. Keyformer: KV cache reduction through 24 http://gemini.google/overview/deep-research key tokens selection for efficient generative inference. In MLSys’24, volume 6, pages 114–127, 2024

    2024

  2. [2]

    Agrawal, N

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwa- tra, B. Gulavani, A. Tumanov, and R. Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi- Serve. InOSDI’24, pages 117–134, 2024

    2024

  3. [3]

    Ainslie, J

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Le- bron, and S. Sanghai. GQA: Training generalized multi- query transformer models from multi-head checkpoints. In EMNLP’23, pages 4895–4901, 2023

    2023

  4. [4]

    Ainslie, S

    J. Ainslie, S. Ontanon, C. Alberti, V. Cvicek, Z. Fisher, P. Pham, A. Ravula, S. Sanghai, Q. Wang, and L. Yang. ETC: Encoding long and structured inputs in transformers. In EMNLP’20, pages 268–284, 2020

    2020

  5. [5]

    R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y. He. DeepSpeed-Inference: Enabling efficient inference of transformer models at unprecedented scale. InSC’22, 2022

    2022

  6. [6]

    A. Asai, S. Min, Z. Zhong, and D. Chen. Retrieval-based language models and applications. InACL’23, 2023

    2023

  7. [7]

    Beltagy, M

    I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer, 2020

    2020

  8. [8]

    Besta, N

    M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, P. Nyczyk, and T. Hoefler. Graph of thoughts: Solving elaborate problems with large language models. AAAI’24, 38(16):17682–17690, 2024

    2024

Show all 140 references
  1. [9]

    Beurer-Kellner, M

    L. Beurer-Kellner, M. Fischer, and M. Vechev. LMQL Chat: Scripted chatbot development, 2023

    2023

  2. [10]

    Beurer-Kellner, M

    L. Beurer-Kellner, M. Fischer, and M. Vechev. Prompting is programming: A query language for large language models. Proc. ACM Program. Lang., 7, 2023

    2023

  3. [11]

    Boehm, M

    M. Boehm, M. Interlandi, and C. Jermaine. Optimizing tensor computations: From applications to compilation and runtime techniques. InSIGMOD’23, pages 53–59, 2023

    2023

  4. [12]

    Understand- ing and overcoming the challenges of efficient transformer quantization

    Y.Bondarenko,M.Nagel,andT.Blankevoort. Understand- ing and overcoming the challenges of efficient transformer quantization. InEMNLP’21, pages 7947–7969, 2021

    2021

  5. [13]

    Z. Cai, Y. Zhang, B. Gao, Y. Liu, T. Liu, K. Lu, W. Xiong, Y. Dong, B. Chang, J. Hu, and W. Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal infor- mation funneling, 2024

    2024

  6. [14]

    S. Cao, Y. Wang, Z. Mao, P.-L. Hsu, L. Yin, T. Xia, D. Li, S.Liu,Y.Zhang,Y.Zhou,Y.Sheng,J.Gonzalez,andI.Sto- ica. Locality-aware fair scheduling in LLM serving, 2025

    2025

  7. [15]

    Chavan, R

    A. Chavan, R. Magazine, S. Kushwaha, M. Debbah, and D. Gupta. Faster and lighter LLMs: A survey on current challenges and way forward. InIJCAI’24, 2024

    2024

  8. [16]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, ...

    2021

  9. [17]

    S. Chen, D. Beeferman, and R. Rosenfeld. Evaluation met- rics for language models. Technical report, Carnegie Mellon University, 1998

    1998

  10. [18]

    Z. Chen, R. Sadhukhan, Z. Ye, Y. Zhou, J. Zhang, N. Nolte, Y. Tian, M. Douze, L. Bottou, Z. Jia, and B. Chen. Mag- icPIG: LSH sampling for efficient LLM generation, 2024. A Survey of LLM Inference Systems 23

    2024

  11. [19]

    Crankshaw, X

    D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica. Clipper: A low-latency online pre- diction serving system. InNSDI’17, pages 613–627, 2017

    2017

  12. [20]

    T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAt- tention: Fast and memory-efficient exact attention with IO- awareness. InNeurIPS’22, 2022

    2022

  13. [21]

    DeepSeek-R1: Incentivizing reasoning capa- bility in LLMs via reinforcement learning, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capa- bility in LLMs via reinforcement learning, 2025

    2025

  14. [22]

    Dettmers, M

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In NeurIPS’22, 2022

    2022

  15. [23]

    Frantar, S

    T.Dettmers,R.Svirschevski,V.Egiazarian,D.Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression, 2023

    2023

  16. [24]

    J. Dong, B. Feng, D. Guessous, Y. Liang, and H. He. Flex Attention: A programming model for generating optimized attention kernels, 2024

    2024

  17. [25]

    J. Fang, Y. Yu, C. Zhao, and J. Zhou. TurboTransformers: An efficient GPU serving system for transformer models. In PPoPP’21, pages 389–402, 2021

    2021

  18. [26]

    Frantar, S

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In ICLR’23, 2023

    2023

  19. [27]

    Freitag and Y

    M. Freitag and Y. Al-Onaizan. Beam search strategies for neural machine translation. In WNMT’17, pages 56–60, 2017

    2017

  20. [28]

    Efficiently serving LLM reasoning programs with Certain- dex, 2024

    Y.Fu,J.Chen,S.Zhu,Z.Fu,Z.Dai,A.Qiao,andH.Zhang. Efficiently serving LLM reasoning programs with Certain- dex, 2024

    2024

  21. [29]

    Y. Fu, S. Zhu, R. Su, A. Qiao, I. Stoica, and H. Zhang. Efficient LLM scheduling by learning to rank, 2024

    2024

  22. [30]

    R. B. Gabrielsson, J. Zhu, O. Bhardwaj, L. Choshen, K. Greenewald, M. Yurochkin, and J. Solomon. Compress then serve: Serving thousands of LoRA adapters with little overhead, 2025

    2025

  23. [31]

    Gholami, S

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference, 2021

    2021

  24. [32]

    I. Gim, G. Chen, S.-s. Lee, N. Sarda, A. Khandelwal, and L. Zhong. Prompt Cache: Modular attention reuse for low- latency inference. In MLSys’24, volume 6, pages 325–338, 2024

    2024

  25. [33]

    R. L. Graham. Bounds on multiprocessing timing anoma- lies. SIAM J. Appl. Math., 17(2):416–429, 1969

    1969

  26. [34]

    A. Graves. Sequence transduction with recurrent neural networks, 2012

    2012

  27. [35]

    Griggs, X

    T. Griggs, X. Liu, J. Yu, D. Kim, W.-L. Chiang, A. Cheung, and I. Stoica. Mélange: Cost efficient large language model serving by exploiting GPU heterogeneity, 2024

    2024

  28. [36]

    Gujarati, R

    A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kauf- mann, Y. Vigfusson, and J. Mace. Serving DNNs like clock- work: Performance predictability from the bottom up. In OSDI’20, pages 443–462, 2020

    2020

  29. [37]

    Holtzman, J

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration, 2020

    2020

  30. [38]

    K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, k. chen, Y. Dong, and Y. Wang. FlashDecoding++: Faster large languagemodelinferencewithasynchronization,flatGeMM optimization, and heuristics. InMLSys’24, volume 6, pages 148–161, 2024

    2024

  31. [39]

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y. Bao, N. Sun, and Y. Shan. Inference without interference: Disaggregate LLM inference for mixed downstream workloads, 2024

    2024

  32. [40]

    J. Hu, W. Huang, H. Wang, W. Wang, T. Hu, Q. Zhang, H. Feng, X. Chen, Y. Shan, and T. Xie. EPIC: Efficient position-independent context caching for serving large lan- guage models, 2025

    2025

  33. [41]

    J. Hu, J. Xu, Z. Liu, Y. He, Y. Chen, H. Xu, J. Liu, B. Zhang, S. Wan, G. Dan, Z. Dong, Z. Ren, J. Meng, C. He, C. Liu, T. Xie, D. Lin, Q. Zhang, Y. Yu, H. Feng, X. Chen, and Y. Shan. DeepFlow: Serverless large language model serving at scale, 2025

    2025

  34. [42]

    Jiang, Y

    H. Jiang, Y. Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. H. Abdi, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu. MInfer- ence 1.0: Accelerating pre-filling for long-context LLMS via dynamic sparse attention, 2024

    2024

  35. [43]

    Jiang, F

    Y. Jiang, F. Fu, X. Yao, G. He, X. Miao, A. Klimovic, B. Cui, B. Yuan, and E. Yoneki. Demystifying cost- efficiency in LLM serving over heterogeneous GPUs, 2025

    2025

  36. [44]

    Jin, C.-F

    Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei. S3: Increas- ing GPU utilization during generative inference for higher throughput. In NeurIPS’23, 2023

    2023

  37. [45]

    Juravsky, B

    J. Juravsky, B. Brown, R. Ehrlich, D. Y. Fu, C. Ré, and A. Mirhoseini. Hydragen: High-throughput LLM inference with shared prefixes, 2024

    2024

  38. [46]

    Khattab, A

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. San- thanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into self- improving pipelines. InICLR’24, 2024

    2024

  39. [47]

    Khoshnoodi, V

    M. Khoshnoodi, V. Jain, M. Gao, M. Srikanth, and A. Chadha. A comprehensive survey of accelerated gen- eration techniques in large language models, 2024

    2024

  40. [48]

    Kim and A

    K. Kim and A. Ailamaki. Trustworthy and efficient LLMs meet databases, 2024

    2024

  41. [49]

    Kossmann, B

    F. Kossmann, B. Fontaine, D. Khudia, M. Cafarella, and S. Madden. Is the GPU half-empty or half-full? Practical scheduling techniques for LLMs, 2025

    2025

  42. [50]

    H. Kwon, K. Koo, J. Kim, W. Lee, M. Lee, H. Lee, Y. Jung, J. Park, Y. Song, B. Yang, H. Choi, G. Kim, J. Won, W. Shin, C. Kim, G. Shin, Y. Kwon, I. Kim, E. Lim, J. Kim, and J. Choi. LoL-PIM: Long-context LLM decoding with scalable DRAM-PIM system, 2025

    2025

  43. [51]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with Paged- Attention. InSOSP’23, pages 611–626, 2023

    2023

  44. [52]

    W. Lee, J. Lee, J. Seo, and J. Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. InOSDI’24, pages 155–172, 2024

    2024

  45. [53]

    Leviathan, M

    Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In ICML’23, 2023

    2023

  46. [54]

    B. Li, Y. Jiang, V. Gadepally, and D. Tiwari. LLM inference serving: Survey of recent advances and opportunities, 2024

    2024

  47. [55]

    H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen. A survey on large language model acceleration based on KV cache management, 2025

    2025

  48. [56]

    Liakopoulos, T

    D. Liakopoulos, T. Hu, P. Sinha, and N. J. Yadwadkar. iServe: An intent-based serving system for LLMs, 2025

    2025

  49. [57]

    C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In ACL’04, pages 74–81, 2004

    2004

  50. [58]

    Y. Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han. QServe: W4A8KV4 quantization and system co- design for efficient LLM serving. InMLSys’25, 2025

    2025

  51. [59]

    N. Ling, G. Chen, and L. Zhong. TimelyLLM: Segmented LLM serving system for time-sensitive robotic applications, 2024

    2024

  52. [60]

    A. Liu, J. Liu, Z. Pan, Y. He, G. Haffari, and B. Zhuang. MiniCache: KV cache compression in depth dimension for large language models, 2024. 24 James Pan, Guoliang Li

    2024

  53. [61]

    D. Liu, M. Chen, B. Lu, H. Jiang, Z. Han, Q. Zhang, Q. Chen, C. Zhang, B. Ding, K. Zhang, C. Chen, F. Yang, Y. Yang, and L. Qiu. RetrievalAttention: Accelerating long- context LLM inference via vector retrieval. InNeurIPS’24, 2024

    2024

  54. [62]

    Liu and P

    H. Liu and P. Abbeel. Blockwise parallel transformers for large context models. In NeurIPS’23, volume 36, pages 8828–8844, 2023

    2023

  55. [63]

    H. Liu, M. Zaharia, and P. Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023

    2023

  56. [64]

    J. Liu, P. Tang, W. Wang, Y. Ren, X. Hou, P.-A. Heng, M. Guo, and C. Li. A survey on inference optimization techniques for mixture of experts models, 2025

    2025

  57. [65]

    Q. Liu, Z. Hong, F. Chen, P. Li, and S. Guo. Mell: Memory- efficient large language model serving via multi-GPU KV cache management, 2025

    2025

  58. [66]

    S. Liu, A. Biswal, A. Cheng, X. Mo, S. Cao, J. E. Gonza- lez, I. Stoica, and M. Zaharia. Optimizing LLM queries in relational workloads, 2024

    2024

  59. [67]

    Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyril- lidis, and A. Shrivastava. Scissorhands: Exploiting the per- sistence of importance hypothesis for LLM KV cache com- pression at test time. InNeurIPS’23, 2023

    2023

  60. [68]

    S. Luo, W. Gao, Z. Wu, J. Peng, C. W. Coley, and J. Ma. Projecting molecules into synthesizable chemical spaces, 2024

    2024

  61. [69]

    Milakov and N

    M. Milakov and N. Gimelshein. Online normalizer calcula- tion for softmax, 2018

    2018

  62. [70]

    Mitzenmacher

    M. Mitzenmacher. The power of two choices in randomized load balancing. IEEE Trans. Parallel and Distrib. Syst., 12(10):1094–1104, 2001

    2001

  63. [71]

    Nagel, M

    M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen, and T. Blankevoort. A white paper on neural network quantization, 2021

    2021

  64. [72]

    Narayan, I

    A. Narayan, I. Chami, L. Orr, and C. Ré. Can foundation modelswrangleyourdata? Proc. VLDB Endow.,16(4):738– 746, 2022

    2022

  65. [73]

    Olston, N

    C. Olston, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V. Rajashekhar, S. Ramesh, and J. Soyke. TensorFlow- Serving: Flexible, high-performance ML serving. In NeurIPS’17 Workshop on ML Systems, 2017

    2017

  66. [74]

    M. Oren, M. Hassid, N. Yarden, Y. Adi, and R. Schwartz. Transformers are multi-state RNNs, 2024

    2024

  67. [75]

    Osama, D

    M. Osama, D. Merrill, C. Cecka, M. Garland, and J. D. Owens. Stream-K: Work-centric parallel decomposition for dense matrix-matrix multiplication on the GPU, 2023

    2023

  68. [76]

    R. Pan, Z. Wang, Z. Jia, C. Karakus, L. Zancato, T. Dao, Y. Wang, and R. Netravali. Marconi: Prefix caching for the era of hybrid LLMs, 2024

    2024

  69. [77]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. In ACL’02, pages 311–318, 2002

    2002

  70. [78]

    Patel, E

    P. Patel, E. Choukse, C. Zhang, A. Shah, I. Goiri, S. Maleki, and R. Bianchini. Splitwise: Efficient generative LLM in- ference using phase splitting. InISCA’24, pages 118–132, 2024

    2024

  71. [79]

    Patke, D

    A. Patke, D. Reddy, S. Jha, C. Narayanaswami, Z. Kalbar- czyk, and R. Iyer. Hierarchical autoscaling for large lan- guage model serving with Chiron, 2025

    2025

  72. [80]

    Peeperkorn, T

    M. Peeperkorn, T. Kouwenhoven, D. Brown, and A. Jor- danous. Is temperature the creativity parameter of large language models?, 2024

    2024

  73. [81]

    Prabhu, A

    R. Prabhu, A. Nayak, J. Mohan, R. Ramjee, and A. Pan- war. vAttention: Dynamic memory management for serving LLMs without PagedAttention, 2025

    2025

  74. [82]

    R. Qin, Z. Li, W. He, M. Zhang, Y. Wu, W. Zheng, and X. Xu. Mooncake: A KVCache-centric disaggregated archi- tecture for LLM serving, 2024

    2024

  75. [83]

    Ramapuram, F

    J. Ramapuram, F. Danieli, E. Dhekane, F. Weers, D. Bus- bridge,P.Ablin,T.Likhomanenko,J.Digani,Z.Gu,A.Shi- dani, and R. Webb. Theory, analysis, and best practices for sigmoid self-attention, 2025

    2025

  76. [84]

    Ren and K

    S. Ren and K. Q. Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference, 2024

    2024

  77. [85]

    Ribar, I

    L. Ribar, I. Chelombiev, L. Hudlass-Galley, C. Blake, C.Luschi,andD.Orr. SparQattention:Bandwidth-efficient LLM inference. InForty-first International Conference on Machine Learning, 2024

    2024

  78. [86]

    Sahoo, A

    P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications, 2025

    2025

  79. [87]

    V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter, 2020

    2020

  80. [88]

    Sanovar, S

    R. Sanovar, S. Bharadwaj, R. S. Amant, V. Rühle, and S. Rajmohan. Lean attention: Hardware-aware scalable at- tention mechanism for the decode-phase of transformers, 2025

    2025

  81. [89]

    Shahout, E

    R. Shahout, E. Malach, C. Liu, W. Jiang, M. Yu, and M. Mitzenmacher. Don’t stop me now: Embedding based scheduling for LLMs, 2024

    2024

  82. [90]

    N. Shazeer. Fast transformer decoding: One write-head is all you need, 2019

    2019

  83. [91]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. In ICLR’17, 2017

    2017

  84. [92]

    Sheng, S

    Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica. S-LoRA: Serving thousands of concurrent LoRA adapters. InMLSys’24, 2023

    2023

  85. [93]

    Sheng, L

    Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang. FlexGen: High- throughput generative inference of large language models with a single GPU. InICML’23, 2023

    2023

  86. [94]

    Srivatsa, Z

    V. Srivatsa, Z. He, R. Abhyankar, D. Li, and Y. Zhang. Pre- ble: Efficient distributed prompt scheduling for LLM serv- ing, 2024

    2024

  87. [95]

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin. Llumnix: Dynamic scheduling for large language model serving. InOSDI’24, pages 173–191, 2024

    2024

  88. [96]

    T. Sun, P. Wang, and F. Lai. HyGen: Efficient LLM serving via elastic online-offline request co-location, 2025

    2025

  89. [97]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. InNeurIPS’17, pages 6000–6010, 2017

    2017

  90. [98]

    Z. Wan, X. Wu, Y. Zhang, Y. Xin, C. Tao, Z. Zhu, X. Wang, S. Luo, J. Xiong, and M. Zhang. D2O: Dynamic discrim- inative operations for efficient generative inference of large language models, 2024

    2024

  91. [99]

    G. Wang, J. Zeng, X. Xiao, S. Wu, J. Yang, L. Zheng, Z. Chen, J. Bian, D. Yu, and H. Wang. FlashMask: Ef- ficient and rich mask extension of FlashAttention, 2024

    2024

  92. [100]

    Wang and G

    J. Wang and G. Li. AOP: Automated and interactive LLM pipeline orchestration for answering complex queries. In CIDR’25, 2025

    2025

  93. [101]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. InICLR’23, 2023

    2023

  94. [102]

    X. Wang, Y. Xiong, Y. Wei, M. Wang, and L. Li. LightSeq: A high performance inference library for transformers. In NAACL’21, pages 113–120, 2021. A Survey of LLM Inference Systems 25

    2021

  95. [103]

    Z. Wang, B. Jin, Z. Yu, and M. Zhang. Model tells you where to merge: Adaptive KV cache merging for LLMs on long-context tasks, 2024

    2024

  96. [104]

    Z. Wang, S. Li, Y. Zhou, X. Li, R. Gu, N. Cam-Tu, C. Tian, and S. Zhong. Revisiting SLO and goodput metrics in LLM serving, 2024

    2024

  97. [105]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS’22, volume 35, pages 24824–24837, 2022

    2022

  98. [106]

    Welleck, I

    S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood train- ing, 2019

    2019

  99. [107]

    B. T. Willard and R. Louf. Efficient guided generation for large language models, 2023

    2023

  100. [108]

    B. Wu, Y. Zhong, Z. Zhang, S. Liu, F. Liu, Y. Sun, G. Huang, X. Liu, and X. Jin. Fast distributed inference serving for large language models, 2024

    2024

  101. [109]

    Y. Wu, Z. Sun, S. Li, S. Welleck, and Y. Yang. Inference scaling laws: An empirical analysis of compute-optimal in- ference for problem-solving with language models, 2024

    2024

  102. [110]

    H. Xia, Z. Yang, Q. Dong, P. Wang, Y. Li, T. Ge, T. Liu, W. Li, and Z. Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative de- coding, 2024

    2024

  103. [111]

    C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, and M. Sun. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory, 2024

    2024

  104. [112]

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. SmoothQuant: Accurate and efficient post-training quanti- zation for large language models. InICML’23, 2023

    2023

  105. [113]

    G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han. DuoAttention: Efficient long-context LLM in- ference with retrieval and streaming heads, 2024

    2024

  106. [114]

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Effi- cient streaming language models with attention sinks. In ICLR’24, 2024

    2024

  107. [115]

    D. Xu, T. Xie, B. Xia, H. Li, Y. Bai, Y. Sun, and W. Wang. Does few-shot learning help LLM performance in code syn- thesis?, 2024

    2024

  108. [116]

    M. Xu, D. Cai, W. Yin, S. Wang, X. Jin, and X. Liu. Resource-efficient algorithms and systems of foundation models: A survey.ACM Comput. Surv., 57(5), 2025

    2025

  109. [117]

    Xuanlei, B

    Z. Xuanlei, B. Jia, H. Zhou, Z. Liu, S. Cheng, and Y. You. HeteGen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices. In ML- Sys’24, volume 6, pages 162–172, 2024

    2024

  110. [118]

    M. Yan, S. Agarwal, and S. Venkataraman. Decoding spec- ulative decoding, 2025

    2025

  111. [119]

    Y. Yan, F. Hu, J. Chen, N. Bhendawade, T. Ye, Y. Gong, N. Duan, D. Cui, B. Chi, and R. Zhang. FastSeq: Make sequence generation faster. In H. Ji, J. C. Park, and R. Xia, editors, ACL-IJCNLP’21, pages 218–226, 2021

    2021

  112. [120]

    D. Yang, X. Han, Y. Gao, Y. Hu, S. Zhang, and H. Zhao. PyramidInfer: Pyramid KV cache compression for high- throughput LLM inference. In ACL’24, pages 3258–3270, 2024

    2024

  113. [121]

    J. Yao, H. Li, Y. Liu, S. Ray, Y. Cheng, Q. Zhang, K. Du, S. Lu, and J. Jiang. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion, 2024

    2024

  114. [122]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InNeurIPS’23, 2023

    2023

  115. [123]

    Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. In NeurIPS’22, 2022

    2022

  116. [124]

    L. Ye, Z. Tao, Y. Huang, and Y. Li. ChunkAttention: Ef- ficient self-attention with prefix-aware KV cache and two- phase partition. InACL’24, pages 11608–11620, 2024

    2024

  117. [125]

    Z. Ye, L. Chen, R.Lai, W. Lin,Y. Zhang, S. Wang, T. Chen, B. Kasikci, V. Grover, A. Krishnamurthy, and L. Ceze. FlashInfer: Efficient and customizable attention engine for LLM inference serving, 2025

    2025

  118. [126]

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. Orca: A distributed serving system for transformer-based generative models. InOSDI’22, pages 521–538, 2022

    2022

  119. [127]

    A. H. Zadeh, I. Edo, O. M. Awad, and A. Moshovos. GOBO: Quantizing attention-based NLP models for low latency and energy efficient inference. In MICRO’20, pages 811–824, 2020

    2020

  120. [128]

    J. Zeng, M. Li, Z. Wu, J. Liu, Y. Liu, D. Yu, and Y. Ma. Boosting distributed training performance of the unpadded BERT model, 2022

    2022

  121. [129]

    Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu. ByteTransformer: A high-performance transformer boosted for variable-length inputs. In IPDPS’23, pages 344–355, 2023

    2023

  122. [130]

    Zhang, X

    H. Zhang, X. Ji, Y. Chen, F. Fu, X. Miao, X. Nie, W. Chen, and B. Cui. PQCache: Product quantization-based KV- Cache for long context LLM inference, 2024

    2024

  123. [131]

    Zhang, J

    J. Zhang, J. wei, P. Zhang, J. Zhu, and J. Chen. SageAt- tention:Accurate8-bitattentionforplug-and-playinference acceleration. In ICML’25, 2025

    2025

  124. [132]

    Zhang, S

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mi- haylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. OPT: Open pre-trained transformer language models, 2022

    2022

  125. [133]

    Zhang, Y

    Y. Zhang, Y. Du, G. Luo, Y. Zhong, Z. Zhang, S. Liu, and R. Ji. CaM: Cache merging for memory-efficient LLMs in- ference. InPMLR’24,volume235,pages58840–58850,2024

    2024

  126. [134]

    Zhang, J

    Y. Zhang, J. Henkel, A. Floratou, J. Cahoon, S. Deep, and J.M.Patel. ReAcTable:EnhancingReActfortablequestion answering. Proc. VLDB Endow., 17(8):1981–4, 2024

    1981

  127. [135]

    Zhang, Y

    Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. InNeurIPS’23, 2023

    2023

  128. [136]

    Zheng, L

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng. SGLang: Efficient execution of structured language model programs, 2024

    2024

  129. [137]

    Zheng, X

    Z. Zheng, X. Ren, F. Xue, Y. Luo, X. Jiang, and Y. You. Response length perception and sequence scheduling: An LLM-empowered LLM inference pipeline. In NeurIPS’23, 2023

    2023

  130. [138]

    Zhong, S

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang. DistServe: Disaggregating prefill and decod- ing for goodput-optimized large language model serving. In OSDI’24, pages 193–210, 2024

    2024

  131. [139]

    Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y. Lou, L. Wang, Z. Yuan, X. Li, S. Yan, G. Dai, X.-P. Zhang, Y. Dong, and Y. Wang. A survey on efficient inference for large language models, 2024

    2024

  132. [140]

    L. Zhu, X. Wang, W. Zhang, and R. W. H. Lau. RelayAt- tention for efficient large language model serving with long system prompts, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.