pith. machine review for the scientific record. sign in

arxiv: 2605.05467 · v1 · submitted 2026-05-06 · 💻 cs.DC

Recognition: unknown

Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism

Vikranth Srivatsa , Zijian He , Pu Guo , Dongming Li , Yiying Zhang
Authors on Pith no claims yet

Pith reviewed 2026-05-08 15:42 UTC · model grok-4.3

classification 💻 cs.DC
keywords LLM servingtensor parallelismadaptive parallelismmulti-tenant inferenceSLO goodputKV cache migrationprefill decode splitruntime reconfiguration
0
0 comments X

The pith

Nitsum makes tensor parallelism a runtime choice to serve mixed LLM requests with different latency targets and raise goodput up to 5.3 times.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Nitsum as a distributed serving system for large language models that must handle both latency-critical interactive requests and background workloads sharing the same GPUs. It shows that keeping tensor parallelism fixed wastes capacity when the mix of request lengths, intensities, and service level objectives changes over time. Instead, Nitsum adjusts the degree of tensor parallelism, the split of GPUs between prefill and decode phases, and the request schedule together at runtime. Two supporting techniques keep the cost of these changes low: reuse of model weights across different parallelism configurations and rapid migration of key-value caches. Experiments on real workload traces demonstrate that these adaptations produce substantially more requests that meet both time-to-first-token and time-per-output-token targets than prior static approaches.

Core claim

Nitsum treats tensor parallelism as a first-class runtime control surface rather than a static deployment choice, jointly optimizing TP level, prefill/decode GPU split, and request scheduling while introducing TP-aware weight reuse and fast KV migration to make frequent adaptations practical.

What carries the argument

Adaptive tensor parallelism with TP-aware weight reuse and fast KV migration, which together enable low-overhead reconfiguration of parallelism degree and GPU allocation during serving.

If this is right

  • More requests satisfy both latency and throughput SLOs under a fixed GPU budget when the system can reconfigure parallelism to match the current mix of interactive and background work.
  • Static configurations become suboptimal as soon as workload intensity or request length distribution varies, creating headroom that dynamic TP can reclaim.
  • Multi-tenant LLM clusters can operate closer to full utilization without separate deployments for each service tier.
  • The same adaptation surface can be applied to other resource decisions such as batch size or decoding strategy once the reconfiguration cost is controlled.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same weight-reuse and KV-migration techniques could be applied to pipeline parallelism or hybrid parallelism schemes to reduce reconfiguration cost in larger clusters.
  • Production deployments might combine this runtime adaptation with offline profiling of common workload mixes to pre-compute safe reconfiguration points.
  • Extending the approach to heterogeneous GPU fleets would require generalizing the weight reuse mechanism across different hardware generations.
  • If the overhead remains low at larger model scales, similar ideas could improve serving efficiency for mixture-of-experts models where expert routing already changes dynamically.

Load-bearing premise

The overhead of changing tensor parallelism degrees often enough to track workload shifts stays small enough that the gains in SLO-compliant requests exceed the reconfiguration costs.

What would settle it

Measure end-to-end goodput on the same real traces when the system is forced to use a single fixed tensor parallelism level with no weight reuse or KV migration, and compare the fraction of requests meeting both TTFT and TPOT targets.

Figures

Figures reproduced from arXiv: 2605.05467 by Dongming Li, Pu Guo, Vikranth Srivatsa, Yiying Zhang, Zijian He.

Figure 1
Figure 1. Figure 1: Tiered SLO Workload and Cluster Dynamism. ServeGen [44] conversation and coding workloads running on 8 H100 GPUs and Llama 3.1-8B. Top row: request pattern and cluster reconfiguration. Middle: optimal cluster configuration of “request group | Prefill/Decode stage | TP level”. Bottom: Nitsum and static TP configuration goodput (higher is better) . substantially view at source ↗
Figure 2
Figure 2. Figure 2: Properties of Tensor parallelism Effect of Tensor Parallelism on 14B and 70B models across A100, H100, and B200 architectures via TTFT, Decode throughput, L2 Cache hit rate, and communication cost. often within milliseconds [1]. Agentic workflows, includ￾ing computer-use assistants (e.g., Claude Computer Use and OpenClaw), execute multi-step tasks with moderate but still user-visible latency expectations [… view at source ↗
Figure 3
Figure 3. Figure 3: Static vs. Dynamic TP Goodput on the ServeGen Workload Per-second goodput for three static TP baselines (All TP1, All TP2, TP1-Prefill + TP2-Decode) on A100, with incoming RPS overlaid (dashed, secondary axis). The black line (“Optimal”) is an oracle upper bound that selects the best configuration at each time step. No single static configuration dominates across time The bar chart reports aggregate goodpu… view at source ↗
Figure 4
Figure 4. Figure 4: Nitsum Architecture optimal dynamic TP setup and the fixed TP configurations, with the former being 23% to 29% higher. 3 Nitsum Design This section first presents an overview of Nitsum’s design. We then introduce our mechanisms for efficient TP switching, discuss our GPU cluster configuration and request schedul￾ing policy, and finally present our global and local scheduler designs. 3.1 System Overview Nit… view at source ↗
Figure 6
Figure 6. Figure 6: KV Conversion in Nitsum When changing from TP 1 to TP 2 then to TP 4 on a cluster of four GPUs and 8 KV heads. Our solution is to keep one full copy of model weights on each GPU and select the TP-specific shard at execution time using customized kernels ( view at source ↗
Figure 7
Figure 7. Figure 7: KV-Migration Latency Comparison. Transfer latency (log scale) across payload sizes for three strategies on fully fragmented and contiguous per-request KV layouts. though benefitial for runtime model forwarding, small pages result in high KV fragmentation within a single request. The KV context for a request can end up reside in a lot of discon￾tigous memory space. Standard memory-movement opera￾tions (e.g.… view at source ↗
Figure 8
Figure 8. Figure 8: Nitsum Request Scheduling and Dynamic Serving Configuration. above problem is combinatorial: each tier can take multiple TP levels and GPU allocations, and the feasible configurations grow exponentially with the number of tiers and cluster scale, rendering it infeasible to solve exactly per control window. We therefore adopt a greedy approximation that iteratively assigns GPUs to the configuration with the… view at source ↗
Figure 9
Figure 9. Figure 9: Goodput Results with Two Production Traces RPS shows incoming request per second. Goodput measured as requests meeting both TTFT and TPOT SLOs per second (higher is better). Results shown across two types of GPUs (A100, H100), two size of GPU cluster (4 and 8), two traces (ServeGen and Azure), and two model sizes (8B and 14B) view at source ↗
Figure 10
Figure 10. Figure 10: TTFT TPOT Raw Traces Median TTFT/TPOT collected from 8B A100 4 H100 across the code and conversation tiers on ServeGen workload. Among the baseline systems, Split performs better than Llumnix and Chiron, because it is able to isolate impact of the two SLOs. Default SGlang performs better in certain settings because it does not perform unnecessary reactions to the multi tier SLOs. At low system load, all t… view at source ↗
Figure 11
Figure 11. Figure 11: TTFT TPOT p90/p99 Raw Traces Tail TTFT and TPOT under the ServeGen workload on 8B models with 4 A100 GPUs. 11b. Observations. Across both coding and conversation work￾loads, Nitsum consistently achieves the lowest or comparable TTFT and TPOT at both p90 and p99. In contrast, baseline systems degrade significantly as load increases. Systems with static execution configurations experience growing queueing d… view at source ↗
Figure 13
Figure 13. Figure 13: Strict-Tier SLO vs. Goodput X axis shows a scale fac￾tor of SLOs in view at source ↗
Figure 16
Figure 16. Figure 16: Sensitivity to reconfigura￾tion interval 0.5 1 2 3 Window Size (s) 0 25 50 75 100 125 Goodput (req/s) view at source ↗
read the original abstract

LLM serving is increasingly multi-tenant: the same deployment must handle latency-critical interactive requests and more relaxed background workloads under a fixed GPU budget. This creates a tiered-SLO setting where maximizing overall goodput (requests that satisfy both TTFT and TPOT targets) is challenging because workload mix, request lengths, and load intensity vary over time. Existing systems mainly optimize request-level controls (e.g., queuing and batching) while keeping execution configuration largely static, which limits adaptation under multi-tier contention. We present Nitsum, a distributed LLM serving system that treats tensor parallelism (TP) as a first-class runtime control surface rather than a static deployment choice. Nitsum jointly optimizes TP level, prefill/decode GPU split, and request scheduling. To make frequent TP adaptation practical, Nitsum introduces TP-aware weight reuse and fast KV migration. Experiments on real traces and targeted microbenchmarks show that Nitsum improves SLO-compliant goodput over SoTA by up to 5.3 times.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces Nitsum, a distributed LLM serving system that elevates tensor parallelism (TP) to a dynamic runtime control, jointly optimizing TP degree, prefill/decode GPU partitioning, and request scheduling for multi-tenant workloads with tiered SLOs. It proposes TP-aware weight reuse and fast KV migration to enable low-overhead adaptations, and reports up to 5.3× higher SLO-compliant goodput than state-of-the-art systems on real traces and microbenchmarks.

Significance. If the empirical gains hold after overhead quantification, the work would meaningfully advance LLM serving by demonstrating that adaptive TP can outperform static configurations under varying request mixes and load intensities. The focus on concrete goodput metrics for tiered SLOs addresses a practical deployment gap; reproducible microbenchmarks on weight reuse and KV migration would further strengthen the contribution.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (Experiments): The central claim of up to 5.3× SLO-compliant goodput improvement is presented without specifying the exact SoTA baseline implementations, the characteristics of the real traces (e.g., request-length distributions, load intensity, adaptation frequency), or measured per-adaptation overheads, leaving the net benefit dependent on unshown assumptions about reconfiguration cost.
  2. [§3] §3 (Design, KV migration subsection): The fast KV migration mechanism is described as keeping overhead low, but no quantitative analysis is given of how migration latency scales with KV cache size or TP degree changes; if migration cost grows linearly with cache occupancy, the benefit could be erased under high-variance workloads that trigger frequent reconfigurations.
  3. [§4] §4 (Microbenchmarks): The targeted microbenchmarks on weight reuse and KV migration are referenced as supporting low overhead, yet the paper does not report the fraction of total inference time spent on adaptations across the evaluated traces, nor does it include sensitivity analysis for cases where request-length variance forces TP changes every few seconds.
minor comments (2)
  1. [§2] Notation for prefill/decode GPU split ratios is introduced without a clear equation or diagram in the early sections, making it harder to follow the joint optimization.
  2. [§4] The paper would benefit from an explicit table listing the SoTA systems compared, their static TP settings, and the precise SLO targets (TTFT/TPOT) used in the goodput metric.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments highlight areas where additional detail will strengthen the paper. We address each major comment below and will incorporate the requested clarifications and analyses in the revised manuscript.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): The central claim of up to 5.3× SLO-compliant goodput improvement is presented without specifying the exact SoTA baseline implementations, the characteristics of the real traces (e.g., request-length distributions, load intensity, adaptation frequency), or measured per-adaptation overheads, leaving the net benefit dependent on unshown assumptions about reconfiguration cost.

    Authors: We agree that more explicit information is required. In the revision we will name the precise SoTA baselines (including versions and static TP configurations), provide summary statistics for the real traces (request-length distributions, load intensities, and observed adaptation frequencies), and add a table of measured per-adaptation overheads so that the 5.3× net goodput gain is clearly shown after reconfiguration costs. revision: yes

  2. Referee: [§3] §3 (Design, KV migration subsection): The fast KV migration mechanism is described as keeping overhead low, but no quantitative analysis is given of how migration latency scales with KV cache size or TP degree changes; if migration cost grows linearly with cache occupancy, the benefit could be erased under high-variance workloads that trigger frequent reconfigurations.

    Authors: We will add the requested quantitative scaling data to §3. A new microbenchmark will report migration latency versus KV cache size for multiple TP degree transitions, together with a short discussion of why the observed sub-linear cost does not negate the benefit even when high-variance workloads trigger frequent reconfigurations. revision: yes

  3. Referee: [§4] §4 (Microbenchmarks): The targeted microbenchmarks on weight reuse and KV migration are referenced as supporting low overhead, yet the paper does not report the fraction of total inference time spent on adaptations across the evaluated traces, nor does it include sensitivity analysis for cases where request-length variance forces TP changes every few seconds.

    Authors: We accept this observation. The revised §4 will include the fraction of total inference time consumed by adaptations for each evaluated trace and a sensitivity analysis examining performance when request-length variance forces TP changes every few seconds, confirming that overhead remains negligible under those conditions. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical system evaluation with no derivations or fitted predictions

full rationale

The paper describes a distributed LLM serving system (Nitsum) that dynamically adjusts tensor parallelism, prefill/decode splits, and scheduling, supported by TP-aware weight reuse and KV migration techniques. All central claims, including up to 5.3× SLO-compliant goodput improvement, rest exclusively on experimental results from real traces and microbenchmarks. No equations, first-principles derivations, parameter fitting, or self-referential predictions appear in the manuscript. The work is self-contained as an empirical systems paper; performance gains are measured directly rather than derived from inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The system introduces new runtime mechanisms whose implementation details and any hidden tuning knobs are not visible.

pith-pipeline@v0.9.0 · 5490 in / 1099 out tokens · 32428 ms · 2026-05-08T15:42:16.119672+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    Taming throughput-latency tradeoff in llm inference with sarathi.18th USENIX Symposium on Operating Systems Design and Imple- mentation, 2024

    Animesh Agrawal et al. Taming throughput-latency tradeoff in llm inference with sarathi.18th USENIX Symposium on Operating Systems Design and Imple- mentation, 2024

    2024

  2. [2]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    work page internal anchor Pith review arXiv 2023

  3. [3]

    Cloudwatch application sig- nals now supports request based service level objectives (slos)

    Amazon Web Services. Cloudwatch application sig- nals now supports request based service level objectives (slos). AWS What’s New, September 2024. Accessed: 2026-03-31

    2024

  4. [4]

    Service level objectives (slos) - amazon cloudwatch

    Amazon Web Services. Service level objectives (slos) - amazon cloudwatch. AWS Documentation, 2026. Ac- cessed: 2026-03-31. 13

    2026

  5. [5]

    Computer use

    Anthropic. Computer use. https://docs.anthropic. com/en/docs/build-with-claude/computer-use ,

  6. [6]

    Accessed: 2026-04-02

    2026

  7. [7]

    Api overview

    Anthropic. Api overview. Claude API Docs, 2026. Accessed: 2026-03-31

    2026

  8. [8]

    Claude code overview

    Anthropic. Claude code overview. Claude Code Docs,

  9. [9]

    Accessed: 2026-03-31

    2026

  10. [10]

    Claude cowork by anthropic

    Anthropic. Claude cowork by anthropic. Anthropic Product, 2026. Accessed: 2026-03-31

    2026

  11. [11]

    Introducing claude opus 4.6

    Anthropic. Introducing claude opus 4.6. Anthropic News, February 2026. Accessed: 2026-03-31

    2026

  12. [12]

    Anthropic. Pricing. Claude Docs, 2026. Includes batch/asynchronous pricing details; Accessed: 2026-03- 31

    2026

  13. [13]

    Anthropic. Pricing. Claude Docs, 2026. Accessed: 2026-03-31

    2026

  14. [14]

    Rate limits

    Anthropic. Rate limits. Claude Docs, 2026. Accessed: 2026-03-31

    2026

  15. [15]

    What interfaces can i use to access claude? Claude Help Center, 2026

    Anthropic. What interfaces can i use to access claude? Claude Help Center, 2026. Accessed: 2026-03-31

    2026

  16. [16]

    Borg, omega, and kubernetes

    Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes. Borg, omega, and kubernetes. Communications of the ACM, 59(5):50–57, April 2016

    2016

  17. [17]

    SCOOT: SLO-Oriented Perfor- mance Tuning for LLM Inference Engines

    Ke Cheng, Zhi Wang, Wen Hu, Tiannuo Yang, Jianguo Li, and Sheng Zhang. SCOOT: SLO-Oriented Perfor- mance Tuning for LLM Inference Engines. InProceed- ings of the ACM Web Conference (WWW), 2025. To appear

    2025

  18. [18]

    Live Migration of Virtual Ma- chines

    Christopher Clark, Keir Fraser, Steven Hand, Ja- cob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. Live Migration of Virtual Ma- chines. In2nd Symposium on Networked Systems De- sign & Implementation (NSDI 05), Boston, MA, May 2005

    2005

  19. [19]

    Optimizing and Characterizing High-Throughput Low-Latency LLM Inference in ML- CEngine

    MLC Community. Optimizing and Characterizing High-Throughput Low-Latency LLM Inference in ML- CEngine. 2024

    2024

  20. [20]

    Openclaw: Open-source imple- mentation of computer-use agents

    OpenClaw Contributors. Openclaw: Open-source imple- mentation of computer-use agents. https://github. com/openclaw/openclaw, 2025. GitHub repository, Accessed: 2026-04-02

    2025

  21. [21]

    Sglang: An llm serving frame- work with high throughput and flexible multi-turn programming

    SGLang contributors. Sglang: An llm serving frame- work with high throughput and flexible multi-turn programming. https://github.com/InternLM/ InternLM/tree/main/serving/SGLang, 2023. GitHub repository

    2023

  22. [22]

    Introducing Simple, Fast, and Scalable Batch LLM Inference on Mosaic AI Model Serving, 2024

    Databricks. Introducing Simple, Fast, and Scalable Batch LLM Inference on Mosaic AI Model Serving, 2024

    2024

  23. [23]

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    work page internal anchor Pith review arXiv 2024

  24. [24]

    Google ai plans and pricing

    Google. Google ai plans and pricing. Google One, 2026. Accessed: 2026-03-31

    2026

  25. [25]

    Rate limits

    Google. Rate limits. Gemini API Docs, 2026. Accessed: 2026-03-31

    2026

  26. [26]

    Silo: Predictable message latency in the cloud

    Keon Jang, Justine Sherry, Hitesh Ballani, and Toby Moncaster. Silo: Predictable message latency in the cloud. InProceedings of the ACM SIGCOMM 2015 Conference, pages 435–448, August 2015

    2015

  27. [27]

    Efficient memory man- agement for large language model serving with page- dattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. InProceedings of the 29th Symposium on Operating Systems Principles, Koblenz, Germany, Oc- tober 2023

    2023

  28. [28]

    Accessed: 2026-04-02

    Linux Manual Pages Project.crontab(5) — Linux Man- ual Page, 2024. Accessed: 2026-04-02

    2024

  29. [29]

    Azurepub- licdataset: Microsoft azure traces

    Microsoft Azure and Microsoft Research. Azurepub- licdataset: Microsoft azure traces. GitHub Repository,

  30. [30]

    Includes Azure LLM inference traces; Accessed: 2026-03-31

    2026

  31. [31]

    Batch processing with the batch api, 2023

    OpenAI. Batch processing with the batch api, 2023

    2023

  32. [32]

    Api pricing

    OpenAI. Api pricing. OpenAI, 2026. Accessed: 2026- 03-31

    2026

  33. [33]

    Rate limits

    OpenAI. Rate limits. OpenAI Platform Docs, 2026. Accessed: 2026-03-31

    2026

  34. [34]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. InISCA, June 2024

    2024

  35. [35]

    HIERARCHICAL AUTOSCALING FOR LARGE LANGUAGE MODEL SERVING WITH CHI- RON.arXiv preprint arXiv:2501.08090, 2025

    Archit Patke, Dhemath Reddy, Saurabh Jha, Chan- dra Narayanaswami, Zbigniew Kalbarczyk, and Ravis- hankar Iyer. HIERARCHICAL AUTOSCALING FOR LARGE LANGUAGE MODEL SERVING WITH CHI- RON.arXiv preprint arXiv:2501.08090, 2025

    work page arXiv 2025

  36. [36]

    arXiv preprint arXiv:2402.12345 , year=

    Archit Patke, Dhemath Reddy, Saurabh Jha, Hao- ran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravis- hankar Iyer. One Queue Is All You Need: Resolving Head-of-Line Blocking in Large Language Model Serv- ing.arXiv preprint arXiv:2402.12345, 2024. 14

    work page arXiv 2024

  37. [37]

    Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. In23rd USENIX Confer- ence on File and Storage Technologies (FAST 25), pages 155–170, Santa Clara, CA, February 2025. USENIX Association

    2025

  38. [38]

    Runpod: Cloud gpu platform for ai and ma- chine learning

    RunPod. Runpod: Cloud gpu platform for ai and ma- chine learning. https://www.runpod.io, 2026. Ac- cessed: 2026-04-23

    2026

  39. [39]

    Sglang: Efficient execution of struc- tured language model programs

    SGLang Team. Sglang: Efficient execution of struc- tured language model programs. https://github. com/sgl-project/sglang, 2024. GitHub repository

    2024

  40. [40]

    Tanya Stivers, N. J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoy- mann, Federico Rossano, Jan Peter de Ruiter, Kyung- Eun Yoon, and Stephen C. Levinson. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26):10587–10592, 2009

    2009

  41. [41]

    Dynamollm: Designing llm inference clusters for performance and energy efficiency

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Tor- rellas, and Esha Choukse. Dynamollm: Designing llm inference clusters for performance and energy efficiency. In2025 IEEE International Symposium on High Per- formance Computer Architecture (HPCA), pages 1348– 1362, 2025

    2025

  42. [42]

    Llumnix: Dynamic Scheduling for Large Language Model Serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic Scheduling for Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 100–118. USENIX As- sociation, 2024

    2024

  43. [43]

    Large- scale cluster management at Google with Borg

    Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large- scale cluster management at Google with Borg. InPro- ceedings of the Tenth European Conference on Com- puter Systems (EuroSys’15), 2015

    2015

  44. [44]

    vllm: Easy, fast, and cheap llm serv- ing with pagedattention

    vLLM Team. vllm: Easy, fast, and cheap llm serv- ing with pagedattention. https://github.com/ vllm-project/vllm, 2023. GitHub repository

    2023

  45. [45]

    BurstGPT: A real-world workload dataset to optimize llm serving systems

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. BurstGPT: A real-world workload dataset to optimize llm serving systems. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V .2 (KDD ’25), Toronto, ON, ...

    2025

  46. [46]

    Servegen: Workload characterization and generation of large lan- guage model serving in production

    Yuxing Xiang, Xue Li, Kun Qian, Yan Zhang, Wenyuan Yu, Ennan Zhai, Xin Jin, and Jingren Zhou. Servegen: Workload characterization and generation of large lan- guage model serving in production. In23rd USENIX Symposium on Networked Systems Design and Imple- mentation (NSDI), Santa Clara, CA, USA, 2026

    2026

  47. [47]

    Dis- tllm: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dis- tllm: Disaggregating prefill and decoding for goodput- optimized large language model serving. InProceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’24), Santa Clara, CA, July 2024. 15

    2024