Pith. sign in

REVIEW 2 major objections 48 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

KV-RM regularizes KV-cache movement beneath a static-graph LLM decoder to absorb irregular request lengths and EOS events while keeping fixed tensor shapes.

2026-07-01 08:03 UTC pith:X2DYZKX6

load-bearing objection KV-RM gives static-graph LLM serving a block-pager plus merge-staged coalescing layer to cut memory waste and latency spikes, but the supporting measurements are missing from the write-up. the 2 major comments →

arxiv 2605.09735 v2 pith:X2DYZKX6 submitted 2026-05-10 cs.AR cs.AIcs.DCcs.OS

KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

Zhiqing Zhong , Zhijing Ye , Jian Zhang , Weijian Zheng , Bolun Sun , Xiaodong Yu This is my paper
classification cs.AR cs.AIcs.DCcs.OS
keywords KV cachestatic graphLLM servinginference runtimememory managementdecode latencythroughputcache movement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that much of the variability in online LLM decoding can be handled by regularizing KV-cache movement rather than by relaxing the fixed decode interface itself. It decouples logical histories from physical storage with a block pager, issues one committed descriptor per step, and uses a merge-staged transport path to gather non-contiguous blocks into large transfers for a fixed-shape attention kernel. This keeps the predictable launches and low submission overhead of static graphs while reducing the need to over-reserve memory. A reader would care because the design directly targets the burst-time latency spikes and wasted reservation that appear when static executors meet production traces. The reported results on a 2-GPU A100 node show gains in mixed-length throughput, tail latency, and memory footprint under both synthetic and replayed workloads.

Core claim

KV-RM decouples logical KV histories from physical storage, tracks active state through a block pager, and materializes each decode step through a single committed descriptor. A merge-staged transport path coalesces non-contiguous KV mappings into a small number of large transfer groups before a fixed-shape attention kernel consumes them. The core design does not depend on optional bounded far-history summaries. On a 2-GPU NVIDIA A100 node the approach improves mixed-length decoding throughput and tail latency relative to a static-graph baseline, reduces reserved KV memory across workload families, and removes severe burst-time latency spikes under production-trace replay.

What carries the argument

The merge-staged transport path that coalesces non-contiguous KV mappings into large transfer groups, together with the block pager and single committed descriptor per decode step.

Load-bearing premise

The merge-staged transport path and single committed descriptor per decode step can coalesce non-contiguous mappings efficiently enough to avoid new bottlenecks while preserving the fixed-shape attention kernel and static-graph predictability.

What would settle it

A production-trace replay on the 2-GPU A100 node in which the added coalescence overhead causes overall throughput or tail latency to fall below the static-graph baseline.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Mixed-length decoding achieves higher throughput than the static-graph baseline on the tested 2-GPU A100 node.
  • Tail latency improves for the same mixed-length workloads.
  • Reserved KV memory is reduced across the workload families examined.
  • Severe burst-time latency spikes disappear when the system replays production traces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same regularization boundary could be applied to other static-graph components such as embedding tables or optimizer states.
  • Hardware schedulers might expose merge-staged primitives to make the coalescence cheaper on future accelerators.
  • The single-descriptor interface may simplify integration with request-level batching policies that currently assume fully dynamic KV management.
  • If the coalescence cost stays low, the approach could reduce the hardware requirement for large on-device KV buffers in multi-tenant serving.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper presents KV-RM, a runtime design that decouples logical KV histories from physical storage using a block pager, materializes each decode step via a single committed descriptor, and employs a merge-staged transport path to coalesce non-contiguous KV mappings before a fixed-shape attention kernel. It claims that this regularizes KV-cache movement beneath a static-graph LLM decoder, yielding improved mixed-length decoding throughput and tail latency on a 2-GPU A100 node, reduced reserved KV memory across workloads, and elimination of severe burst-time latency spikes under production-trace replay, all while preserving static-graph predictability without requiring optional far-history summaries.

Significance. If the empirical results hold and the coalescing mechanism remains efficient, the work shows that variability from irregular KV behavior can be absorbed below the fixed decode interface rather than through dynamic kernels or over-reservation. This could be significant for production static-graph serving systems that prioritize launch predictability and low overhead, offering a practical engineering boundary for flexibility.

major comments (2)
  1. [Abstract] Abstract: The claims of improved throughput, tail latency, reduced reserved KV memory, and removal of burst-time latency spikes under production traces are stated without any quantitative metrics, baseline comparisons, workload definitions, or error analysis. This absence is load-bearing because the central claim rests entirely on external workload replay rather than self-contained derivations.
  2. [Abstract] Abstract: The merge-staged transport path is asserted to coalesce non-contiguous KV mappings into a small number of large transfer groups using a single committed descriptor per decode step, but no bound is supplied on the number of physical blocks touched per step nor any measurement of coalescing overhead under fragmentation. This assumption is central to the latency and predictability claims; if fragmentation produces many segments, the transport path itself could introduce variability or extra launches.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the feedback on the abstract. We agree that strengthening the abstract with quantitative support will improve clarity and address the load-bearing nature of the claims. We respond to each major comment below and will revise the abstract accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The claims of improved throughput, tail latency, reduced reserved KV memory, and removal of burst-time latency spikes under production traces are stated without any quantitative metrics, baseline comparisons, workload definitions, or error analysis. This absence is load-bearing because the central claim rests entirely on external workload replay rather than self-contained derivations.

    Authors: We agree the abstract should include concrete metrics to support the claims. The full manuscript already details these in the evaluation section (throughput and tail latency gains on 2-GPU A100 relative to static-graph baseline, memory reduction across workloads, and elimination of spikes under production traces), including workload definitions and baseline comparisons. In revision we will incorporate representative quantitative results (e.g., percentage improvements and workload families) directly into the abstract while preserving its length. A brief note on observed variability can also be added if it fits. revision: yes

  2. Referee: [Abstract] Abstract: The merge-staged transport path is asserted to coalesce non-contiguous KV mappings into a small number of large transfer groups using a single committed descriptor per decode step, but no bound is supplied on the number of physical blocks touched per step nor any measurement of coalescing overhead under fragmentation. This assumption is central to the latency and predictability claims; if fragmentation produces many segments, the transport path itself could introduce variability or extra launches.

    Authors: The block-pager and merge-staged design are intended to keep the number of transfer groups small by construction, with the single descriptor guaranteeing one kernel launch per decode step. We acknowledge that an explicit worst-case bound on physical blocks per step and direct overhead measurements under extreme fragmentation are not quantified in the current text. If the revision allows, we can add a short analytical bound derived from the pager parameters and note that empirical results under the evaluated traces already demonstrate stable latency; otherwise the point can be addressed in the body discussion of the transport path. revision: partial

Circularity Check

0 steps flagged

No circularity: engineering design evaluated on external workloads

full rationale

The paper describes a runtime system (KV-RM) for regularizing KV-cache movement under a static-graph decoder. It presents a block pager, merge-staged transport, and single-descriptor interface as an engineering artifact, with throughput/latency claims resting on empirical replay of production traces and workload families. No equations, fitted parameters, self-referential definitions, or load-bearing self-citations appear in the provided text. The central claims are not derived from prior author work via uniqueness theorems or ansatzes; they are direct measurements against a static-graph baseline. This is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 2 invented entities

Abstract-only view supplies no explicit free parameters, axioms, or externally validated invented entities; the design itself introduces the block pager and merge-staged path as new mechanisms.

invented entities (2)
  • block pager no independent evidence
    purpose: Decouples logical KV histories from physical storage and tracks active state
    Core component of KV-RM introduced to regularize movement
  • merge-staged transport path no independent evidence
    purpose: Coalesces non-contiguous KV mappings into large transfer groups
    Mechanism that enables fixed-shape attention under irregular cache layouts

pith-pipeline@v0.9.1-grok · 5803 in / 1088 out tokens · 34960 ms · 2026-07-01T08:03:32.205741+00:00 · methodology

0 comments
read the original abstract

Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time. Dynamic runtimes recover flexibility through paged KV management and step-level scheduling, while static-graph executors often over-reserve memory and suffer burst-time latency outliers. This paper studies whether much of this variability can be absorbed below a fixed decode interface. We present KV-RM, a runtime design that regularizes KV-cache movement beneath a static-graph LLM decoder. KV-RM decouples logical KV histories from physical storage, tracks active KV state through a block pager, and materializes each decode step through a single committed descriptor. A merge-staged transport path coalesces non-contiguous KV mappings into a small number of large transfer groups before a fixed-shape attention kernel consumes them. Optional bounded far-history summaries can be enabled under the same interface, but the core design does not depend on them. On a 2-GPU NVIDIA A100 node, KV-RM improves mixed-length decoding throughput and tail latency relative to a static-graph baseline, reduces reserved KV memory across workload families, and removes severe burst-time latency spikes under production-trace replay. These results suggest that KV-cache movement, rather than kernel shape, can be an effective boundary for recovering runtime flexibility in static-graph LLM serving.

Figures

Figures reproduced from arXiv: 2605.09735 by Bolun Sun, Jian Zhang, Weijian Zheng, Xiaodong Yu, Zhijing Ye, Zhiqing Zhong.

Figure 1
Figure 1. Figure 1: GPU-side structural limits of static-graph decoding. (a) Under identical dense-attention semantics, static-graph execution retains a large idle memory floor compared with a paged runtime. In panel (a), after-idle process-resident foot￾print is reported as the aggregate across the two active GPUs of the 2× A100-40GB PCIe node used by the run. (b) A sepa￾rate internal sweep shows the O(T) bandwidth wall unde… view at source ↗
Figure 2
Figure 2. Figure 2: KV-RM architecture and invariants. The con￾trol plane (left) shapes a logical KV view using a fixed￾shape near-window decoder, a KV pager, an optional far￾view policy, and lookahead placement/prefetch. The merge￾staged transport pipeline (right) bridges the control plane and the DMA engine, turning these mappings into coalesced DMA trains that feed the static attention kernel. Across all mechanisms, the ke… view at source ↗
Figure 3
Figure 3. Figure 3: One decode step under the KV-RM contract. Run￾time variability is expressed as mapping edits (Alias, Trim, Reserve) and sealed by a single Frame commit via shadow￾to-active descriptor swap. Descriptor merging then converts fragmented page descriptors into a small constant number of trains, typically a near-window train and, when needed, one far-view train in the main operating regime, that feed the same fi… view at source ↗
Figure 4
Figure 4. Figure 4: GPU main end-to-end behavior on a 2× A100- 40GB PCIe node. (a,b) Under a 60-second high-load Azure replay window, KV-RM approaches the dynamic-runtime baselines while tightening replay-window p99/p99.9 latency relative to the static-graph baseline. (c,d) Under controlled mixed-length serving, KV-RM improves throughput and p99 relative to the static-graph baseline and remains close to the dynamic-runtime ba… view at source ↗
Figure 6
Figure 6. Figure 6: Mechanism audit and bounded-budget qual [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Boundary stress at the main 2× A100-40GB PCIe node operating point. (a–c) Under a concurrency sweep, KV-RM preserves a single committed descriptor per step, bounded control-plane cost, and competitive through￾put/tail behavior as concurrency rises. In panel (c), submit share denotes host submit plus frame commit divided by per-step wall time, and commit cost is measured per commit￾ted step. (d–f) Under har… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages · 13 internal anchors

  1. [1]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. InProceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Associ- ation, Berkeley, CA, USA, 103–12...

    work page 2024

  2. [2]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Min- jia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-inference: enabling efficient inference of transformer models at unprecedented scale. InProceedings of the International Conference on High Perfor- mance Computing, Networki...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/sc41404.2022.00051 2022

  3. [3]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. arXiv preprint arXiv:2406.02069.https://arxiv.org/abs/24 06.02069

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yu Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Berkeley, CA, USA, 578–594.https://www.usenix.org...

    work page 2018

  5. [5]

    Yihua Cheng, Kuntai Du, Jiayi Yao, and Junchen Jiang. 2024. Do Large Language Models Need a Content Delivery Network? arXiv preprint arXiv:2409.13761.https://arxiv.org/abs/2409.13761

    work page arXiv 2024

  6. [6]

    Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, and Junchen Jiang

    work page

  7. [7]

    Lmcache: An efficient kv cache layer for enterprise-scale llm inference,

    LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv preprint arXiv:2510.09665.https://arxiv.org/ab s/2510.09665

    work page arXiv

  8. [8]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Paral- lelism and Work Partitioning. International Conference on Learning Representations (ICLR).https://arxiv.org/abs/2307.08691

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    work page

  10. [10]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 16344– 16359.https://arxiv.org/abs/2205.14135

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    A dataset of information- seeking questions and answers anchored in research papers

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A Dataset of Information-Seeking Ques- tions and Answers Anchored in Research Papers. arXiv preprint arXiv:2105.03011.https://arxiv.org/abs/2105.03011

    work page arXiv 2021

  12. [12]

    Hao Geng, Phitchaya Mangpo Phothilimthana, et al. 2024. vAttention: Dynamic Memory Management for Serving LLMs without PagedAt- tention. arXiv preprint arXiv:2405.04437.https://arxiv.org/abs/2405.0 4437 12

    work page Pith review arXiv 2024

  13. [13]

    Graphcore. 2023. Poplar SDK Documentation.https://docs.graphcore .ai/

    work page 2023

  14. [14]

    Graphcore Research. 2024. SparQ Attention: Speed up LLM inference with top-k Value reads.https://graphcore-research.github.io/posts/s parq/

    work page 2024

  15. [15]

    Arpan Gujarati, Reza Karrell, Seth Ganesan, Avinash Vachharajani, H Ramachandran, et al. 2020. Serving DNNs like Clockwork: Perfor- mance Predictability from the Bottom Up. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX As- sociation, Berkeley, CA, USA, 443–462.https://www.usenix.org/con ference/osdi20/presentation/gujarati

    work page 2020

  16. [16]

    Huanle He, Lin Jin, Yizhe Cai, Jeeyong Kim, Taejoon Kang, Simin Chen, Ran Tian, Yifeng Zhang, Sizhe Zheng, Yi Yang, et al . 2025. WaferLLM: Large Language Model Inference at Wafer Scale. In19th USENIX Symposium on Operating Systems Design and Implementation (OSDI 25). USENIX Association, Berkeley, CA, USA, 41–57.https: //www.usenix.org/conference/osdi25/p...

    work page 2025

  17. [17]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv preprint arXiv:2404.06654.https://arxiv.org/abs/2404.06654

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Ziyu Hu, Zhiqing Zhong, Weijian Zheng, Zhijing Ye, Xuwei Tan, Xueru Zhang, Zheng Xie, Rajkumar Kettimuthu, and Xiaodong Yu. 2025. DABench-LLM: Standardized and In-Depth Benchmarking of Post- Moore Dataflow AI Accelerators for LLMs . In2025 IEEE International Symposium on Workload Characterization (IISWC). IEEE Computer Society, Los Alamitos, CA, USA, 127–...

    work page doi:10.1109/iiswc66894.2 2025

  19. [19]

    Zhe Jia, Blake Tillman, Marco Maggioni, and Daniele P Scarpazza. 2019. Dissecting the Graphcore IPU architecture via microbenchmarking. arXiv preprint arXiv:1912.03413.https://arxiv.org/abs/1912.03413

    work page arXiv 2019

  20. [20]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. 2023. TPU v4: An optically reconfigurable supercom- puter for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA). Ass...

    work page doi:10.1145/3579371.3589350 2023

  21. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica

    work page

  22. [22]

    InProceedings of the 29th Symposium on Operating Systems Principles (SOSP)

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP). Association for Computing Machinery, New York, NY, USA, 611–626.https://arxiv.org/abs/2309 .06180

    work page

  23. [23]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen

    work page

  24. [24]

    SnapKV: LLM Knows What You are Looking for Before Generation

    SnapKV: LLM knows what you are looking for before gener- ation. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 722, 24 pages. https://arxiv.org/abs/2404.14469

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI).https: //www.usenix.org/conference/osdi...

    work page 2023

  26. [26]

    Haotian Liu, Hanchen Li, Qing Yang, and Yongqiang Wang. 2023. Scissorhands: Exploiting the Persistence of Importance Hypothe- sis for LLM KV Cache Compression at Test Time. arXiv preprint arXiv:2305.17118.https://arxiv.org/abs/2305.17118

    work page Pith review arXiv 2023

  27. [27]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anantha- narayanan, et al. 2024. Cachegen: Kv cache compression and stream- ing for fast large language model serving. InProceedings of the ACM SIGCOMM 2024 Conference. Association for Computing Machinery, New York, NY, USA, 38–56. doi:10.1...

    work page doi:10.1145/3651890.3672274 2024

  28. [28]

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Luis Ceze, Alex Beutel, and Christopher Ré. 2023. Deja Vu: Contextual sparsity for efficient LLMs at inference time. InInternational Conference on Machine Learning (ICML). PMLR, PMLR, Honolulu, HI, USA, 22137–22176.https://arxiv.org/abs/2310.1 7157

    work page 2023

  29. [29]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: a tuning- free asymmetric 2bit quantization for KV cache. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria) (ICML’24). JMLR.org, Cambridge, MA, USA, Article 1311, 13 pages. https://arxiv.org/...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    work page

  31. [31]

    Pointer Sentinel Mixture Models

    Pointer Sentinel Mixture Models. International Conference on Learning Representations (ICLR).https://arxiv.org/abs/1609.07843

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    Microsoft. 2023. AzureLLMInferenceDataset2023.https://github.com /Azure/AzurePublicDataset/blob/master/AzureLLMInferenceDatas et2023.md

    work page 2023

  33. [33]

    NVIDIA. 2023. TensorRT-LLM: A High-Performance Library for LLM Inference.https://github.com/NVIDIA/TensorRT-LLM

    work page 2023

  34. [34]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aagan Shah, Iñigo Goiri Wisniewski, David Koufaty, et al. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. InProceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, Los Alamitos, CA, USA, 211–224. doi:10.1109/ISCA59077.2024.00019

    work page doi:10.1109/isca59077.2024.00019 2024

  35. [35]

    Shah, Zhengyu Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J

    Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang, Xiaoyan Li, Yongning Sheng, Joshua Brot, Denis Sokolov, Apurv Vivek, Calvin Leung, Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jackson, Mark Luttrell, Manish K. Shah, Zhengyu Chen, Kaizhao Liang, Swayambhoo Jain,...

    work page doi:10.1109/micro61859.2024.00100 2024

  36. [36]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot. In23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, Santa Clara, CA, 155–170.https://www.us...

    work page 2025

  37. [37]

    Amit Sabne. 2020. XLA: Compiling machine learning for peak per- formance. Proceedings of the 3rd Conference on Machine Learning and Systems (MLSys).https://research.google/pubs/xla-compiling- machine-learning-for-peak-performance/

    work page 2020

  38. [38]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI).https://www.usenix.org /conference/osdi24/presentation/sun-biaoUSENIX Association

    work page 2024

  39. [39]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST: query-aware sparsity for efficient long- context LLM inference. InProceedings of the 41st International Con- ference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Cambridge, MA, USA, Article 1955, 11 pages.https://arxiv.org/abs/24 06.10774 13

    work page 2024

  40. [40]

    Bingyang Wu, Yinmin Zhong, Zizheng Gupta, Whan Huang, Christo- pher Ré, DA Ce, and Kai Li. 2023. FastServe: Lean-quanta imple- mentation of preemptive scheduling for distributed LLM serving. In Proceedings of the 17th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI). USENIX Association, Berkeley, CA, USA, 1197–1213.https://arxiv.or...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  41. [41]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks.https://arxiv.org/abs/2309.17453

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).https://arxiv.org/abs/1809.09600

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  43. [43]

    Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. InProceedings of the Twentieth European Conference on Com- puter Systems. Association for Computing Machinery, New York, NY, USA, 94–109. doi:10.1145/36...

    work page doi:10.1145/3689031.3696098 2025

  44. [44]

    Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. arXiv:2402.15220 [cs.LG]https://arxiv.org/abs/2402.15220

    work page arXiv 2024

  45. [45]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Associ- ation, Berkeley, CA, USA, 521–538.https://www.usenix.org/confere nce/osdi22/presentation/yu

    work page 2022

  46. [46]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024. H2O: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems (NeurIPS).https://arxiv.org/abs/2306.14048

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  47. [47]

    SGLang: Efficient Execution of Structured Language Model Programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: efficient execution of structured language model programs. InPro- ceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC,...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI).https://www.usenix.org/conference/osdi24 /presentation/zhong-yinminUSENIX Ass...

    work page 2024