Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Decode tokens per second, not GPU utilization, is the robust autoscaling signal for disaggregated LLM serving; the production system built on it raises average GPU utilization by 26.6 percentage points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A plausible production autoscaling design with a genuinely useful metric comparison, but the headline utilization gain rests on uncontrolled comparisons and should be read as anecdote until better evidence appears. the 4 major comments →

arxiv 2508.19559 v1 pith:D6KX2MO3 submitted 2025-08-27 cs.DC cs.AI

Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

Rongzhi Li , Ruogu Du , Zefang Chu , Sida Zhao , Chunlei Han , Zuocheng Shi , Yiwen Shao , Huanle Han
show 3 more authors
Long Huang Zherui Liu Shufan Liu
This is my paper
classification cs.DC cs.AI
keywords LLM inference servingautoscalingprefill-decode disaggregationdecode TPSGPU utilizationnetwork-aware schedulingheterogeneous GPUsproduction deployment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that autoscaling for prefill-decode disaggregated LLM serving fails when each phase is scaled independently on hardware metrics, because decode GPU utilization stays high even when idle due to KV-cache memory pressure. It presents HeteroScale, a production autoscaling system that uses decode tokens-per-second as a single scaling signal to resize both prefill and decode pools together while enforcing a fixed prefill-to-decode ratio, and places instances with network topology in mind. Deployed on tens of thousands of GPUs, the system reports a 26.6-percentage-point gain in average GPU utilization and daily savings of hundreds of thousands of GPU-hours while meeting latency SLOs. The significance is a template: one workload-level throughput signal plus coordinated scaling can keep heterogeneous, disaggregated inference balanced and efficient.

Core claim

The central claim is that decode TPS is the robust autoscaling signal for prefill-decode disaggregated serving. The paper argues that decode-phase hardware metrics are misleading—KV-cache memory pressure keeps decode GPU utilization and SM activity high even under low load—while latency metrics react nonlinearly with a cliff-like threshold, and prefill TPS is confounded by KV-cache hit rates. Decode TPS, by contrast, tracks load with high signal-to-noise ratio and, the paper states, distributes uniformly across decode instances. Using decode TPS, HeteroScale's proportional-control policy computes the desired total capacity from a target per-instance throughput and then applies the fixed P/D

What carries the argument

The load-bearing mechanism is the coordinated scaling policy built on decode TPS (tokens generated per second across the decode pool), used as a single proportional-control signal: it computes the needed total capacity and splits it by a fixed prefill-to-decode ratio, so the two pools are always scaled together. Its supporting mechanical counterpart is the Deployment Group abstraction—a logical container that pins a service's prefill and decode replicas to a shared network domain—plus RDMA Subgroups that rank those domains by hardware priority. Together they translate one aggregate workload signal into balanced, network-aware placement decisions.

Load-bearing premise

The whole coordinated policy assumes decode TPS is spread evenly across decode instances, so a pool-wide average tells the autoscaler what every instance is feeling; if real traffic concentrates tokens on some instances, the average hides imbalance and the single signal can leave some GPUs saturated while others idle.

What would settle it

Collect per-instance decode TPS across a production decode pool during a traffic peak and compare the highest and lowest instances. If the spread is wide—say several instances at more than 1.5 times the pool mean while others sit near idle—and the pool-level decode TPS still drives scale-in/out decisions that let SLOs degrade, the uniform-distribution premise is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Autoscalers for P/D-disaggregated serving should not scale decode pools on GPU utilization or SM activity; those metrics report high values regardless of actual decode load.
  • Decode TPS-based coordinated scaling maintains the prefill-to-decode ratio under diurnal traffic, avoiding the architectural imbalance that occurs when pools are scaled independently.
  • Network affinity-aware placement of prefill and decode instances is needed to preserve the bandwidth for KV-cache transfer that disaggregated serving depends on.
  • The TPS-driven policy outperforms a time-based periodic policy in production, delivering 10.0 percentage points higher GPU utilization and 11.1 points higher SM activity.
  • The fixed P/D ratio can protect both TTFT and TBT SLOs even during rapid scaling events, as long as prefill and decode instances register in a coordinated way.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single-signal design presumes decode TPS is balanced across instances; a testable extension is to monitor the per-instance distribution of decode TPS and trigger additional rebalancing if some decode replicas saturate while others idle.
  • The fixed P/D ratio is an operator-set constant derived from pressure tests; the paper's own future-work section implies it can be adapted online, and one concrete extension would be to adjust the ratio from queueing or latency signals during workload drift.
  • The same coordinated single-metric pattern may generalize to other memory-bound disaggregated pipelines, such as mixture-of-experts serving, where one stage's hardware utilization is decoupled from its true load.
  • The reported 26.6-point utilization gain is a fleet-level average across heterogeneous services; a sharper evaluation would separate services by workload type and hardware tier to see where the coordinated policy helps most.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents HeteroScale, a production autoscaling framework for prefill-decode (P/D) disaggregated LLM serving at ByteDance. The system combines a topology-aware scheduler that uses Deployment Groups and RDMA Subgroups to enforce network affinity and hardware-priority constraints, with a metrics-driven scaling policy. Based on an empirical comparison of eight autoscaling signals, the authors select decode TPS as the primary signal and scale prefill and decode pools jointly under a fixed P/D ratio, supplemented by a latency-based negative-feedback mechanism. The claimed contributions are a coordinated scaling policy that preserves architectural balance, a set of scheduling abstractions for heterogeneous hardware and network topology, and a large-scale production evaluation reporting a 26.6 percentage-point increase in average GPU utilization, a 9.2 percentage-point increase in SM activity, and hundreds of thousands of GPU-hours saved daily while preserving SLOs.

Significance. If the production-effectiveness claims were convincingly supported, this would be a significant systems contribution: it addresses real operational challenges of P/D-disaggregated LLM serving at a scale rarely reported in the literature. The paper's strengths include the use of production workload traces, the controlled P/D-ratio experiments on two services (Section 4.1), the replay comparison of eight scaling metrics (Section 4.2), and the concrete scheduling abstractions and heuristics in Section 3.4. The P/D coordination idea—using one robust signal to scale both pools with a fixed ratio—is plausible and well motivated. However, the headline quantitative claims in Section 4.3 rest on uncontrolled comparisons, and the metric-selection evidence in Section 4.2 is presented without quantitative scoring. The paper does not release code, data, or configurations, so the reported operational gains are not independently reproducible. The central contribution is credible but currently under-supported by the evidence as presented.

major comments (4)
  1. [Section 4.3 (Production Deployment Analysis)] The headline claim of a 26.6 percentage-point increase in GPU utilization rests on comparing services with HeteroScale enabled to services without autoscaling on 'a representative day,' plus a before/after comparison. No controls are reported for service mix, traffic volume, prompt/generation length distribution, hardware pool, or concurrent platform changes. The per-service numbers in the same section (prefill GPU utilization 46.8%→76.2%, decode GPU utilization 86.0%→82.2%) show that the aggregate gain could be dominated by a shift in service mix toward prefill-heavy workloads. Please report matched cohorts or difference-in-differences estimates, with confidence intervals, and clarify which confounds were controlled.
  2. [Section 4.2.2 (Experiment Results)] The metric-replay evaluation is currently qualitative: eight replays are 'recorded and visualized' in Figure 6, and the text asserts that TPS-based autoscaling responds effectively while decode utilization is ineffective. No quantitative scoring is provided—no SLO violation counts, overshoot/oscillation metrics, time-to-scale, resource waste, or comparison against a no-scaling baseline. Since the selection of decode TPS as the primary signal is load-bearing for the coordinated policy, the replay experiments should be scored on objective criteria and a table of these metrics provided.
  3. [Section 4.2.2 and Section 3.3.2 (Metrics-driven Scaling Policy)] The coordinated policy assumes decode TPS is 'uniformly distributed across instances,' so that a pool-level average can serve as the single scaling signal for both prefill and decode pools. The paper provides no per-instance decode TPS distribution or load-imbalance measurements. If decode TPS is heterogeneous across instances, the single-signal proportional rule could scale the pool based on an average while some instances saturate and others idle, undermining the claimed balance. Please report per-instance decode TPS variance or provide evidence that instance-level imbalance is controlled by the scheduler.
  4. [Section 3.3.3 (Workload-centric Policy Curation, Algorithm 1)] Algorithm 1 obtains the optimal P/D ratio and target per-instance metric from a pressure test on service S and workload W, then selects the policy by simulating on the same W. This configuration appears to be the same one used in the production evaluation of Section 4.3. There is no held-out workload or time period, so the reported gains may partly reflect tuning to the evaluation conditions. Please add a validation split (e.g., a different time window or service) to demonstrate that the chosen ratio and thresholds generalize rather than overfit the calibration data.
minor comments (4)
  1. [Throughout] There are several typos and formatting issues: 'T aming' in the title, 'Geployment Groups' in Section 3.1, 'motivat' in Section 2.2, 'simutaneously' and 'address this this issue' in Section 3.4, and inconsistent variable names such as 'P odDelta'/'P odsDelta' and 'If inal' in the algorithms.
  2. [Figure 6 and Appendix C] The replay figures show scaling events but do not explain the axis units, the threshold calibration, or how the overlaid instance-count curves are computed. A legend and a description of the replay harness would make the comparison interpretable.
  3. [Section 4.3] The statement that the TPS-based policy manages '64% of the total GPU fleet under HeteroScale’s control' needs a denominator and a time window. Also, 'a representative day' and 'a recent date with a date prior to the scaled deployment' should be specified (dates, duration, and what changed between them).
  4. [Figure 7d] The text says the large latency fluctuations 'are an artifact of normalization' but does not explain the normalization or show the unnormalized curves. Please clarify or replot.

Circularity Check

0 steps flagged

No significant circularity: the central claims rest on measured production comparisons and empirical metric selection, not on inputs defined in terms of outputs.

full rationale

The paper's derivation chain is empirical rather than definitional. The workload-centric policy curation (Algorithm 1) uses pressure tests to select a P/D ratio and target metric, and candidate policies are simulated; this is a configuration/tuning step, not a prediction, and the headline utilization gains (26.6 pp, 9.2 pp, GPU-hours saved) are reported from production before/after and cross-service comparisons, not computed from the tuned parameters by construction. The coordinated scaling policy enforces a preset P/D ratio by scaling prefill and decode pools simultaneously, so observing that the ratio is maintained (Figure 7a) is a mechanism sanity check rather than evidence that the policy's benefits are derived from its own definition. No load-bearing self-citations appear; the cited prior work (e.g., MegaScale-Infer) is external and not used to forbid alternatives or as a uniqueness theorem. The assumption that decode TPS is uniformly distributed across instances is an unverified empirical assumption, but an unsupported assumption is a correctness risk, not circularity. The paper does not rename a known result or fit a parameter and then relabel it as a prediction; its evaluation is a deployed-system measurement. Therefore no circular step meeting the stated evidentiary bar can be identified.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on per-service fitted parameters (P/D ratio, target TPS, thresholds) and on assumptions about workload stability and metric uniformity. These assumptions are plausible for the specific ByteDance environment but are not validated with released data. The paper's contribution is a systems integration, so the free parameter and assumption load is moderate and mostly tied to tuning, not to a mathematical derivation.

free parameters (4)
  • target P/D ratio per service = varies, e.g., 1P/5D to 9P/1D (Section 4.1)
    Derived from per-service pressure tests and historical data; used in all scaling decisions.
  • target metric per instance (M_target) = not specified numerically
    Input to Algorithm 2, obtained from pressure test; determines how many instances are provisioned at a given decode TPS.
  • scaling thresholds and cooling periods (θ_out, θ_in, C_out, C_in) = not specified numerically
    Tuning constants in Algorithm 2 that determine when scale-out and scale-in trigger.
  • latency feedback thresholds and step sizes (α_out, β_out, γ_in, 1.2, 1.1, 0.95) = 1.2, 1.1, 0.95 as step multipliers
    Hyperparameters in Algorithm 3; the paper notes these are hard to tune in production.
axioms (4)
  • domain assumption The optimal P/D ratio derived from offline pressure tests remains valid for online scaling with a fixed ratio.
    Section 3.4 states the online system uses a fixed P/D ratio from pressure tests and historical data; Section 6 admits that workload drift may require dynamic adaptation.
  • domain assumption decode TPS is uniformly distributed across decode instances, so a pool-level signal is representative.
    Section 4.2.2: 'decode TPS is preferred, as it can be uniformly distributed across instances.' No per-instance variance data are given.
  • domain assumption The RDMA subgroup hierarchy (S0/S1/S2) is a valid proxy for KV cache transfer cost, and co-location under the same switch is necessary and sufficient for performance.
    Section 3.4 builds the scheduler on this hierarchy; no direct KV transfer latency measurements are presented.
  • domain assumption Workload input/output length distributions are stable enough that TPS-based scaling preserves SLOs without dynamic ratio adjustment.
    Section 4.2.2 argues I/O distribution remains stable within a service; Section 6 lists dynamic ratio adaptation as future work to handle drift.
invented entities (2)
  • Deployment Group no independent evidence
    purpose: Logical container binding prefill and decode roles of a service under a common network affinity constraint.
    Software abstraction introduced by the paper. Its effectiveness is only demonstrated through the system's own production claims, with no external falsifiable handle.
  • RDMA Subgroup no independent evidence
    purpose: Priority tier for S1/S2 switch groups that reserves scarce heterogeneous hardware for demanding services.
    Internal priority-ordering abstraction used by the scheduler. No independent measurement shows it improves placement outcomes.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference." pith.science (2026). https://pith.science/paper/D6KX2MO3

@misc{pith2026250819559,
  author       = {Pith},
  title        = {Pith review of: Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6KX2MO3}},
  note         = {Machine review of arXiv:2508.19559}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short, particularly for modern Prefill-Decode (P/D) disaggregated architectures. This architectural shift, while powerful, introduces significant operational challenges, including inefficient use of heterogeneous hardware, network bottlenecks, and critical imbalances between prefill and decode stages. We introduce HeteroScale, a coordinated autoscaling framework that addresses the core challenges of P/D disaggregated serving. HeteroScale combines a topology-aware scheduler that adapts to heterogeneous hardware and network constraints with a novel metric-driven policy derived from the first large-scale empirical study of autoscaling signals in production. By leveraging a single, robust metric to jointly scale prefill and decode pools, HeteroScale maintains architectural balance while ensuring efficient, adaptive resource management. Deployed in a massive production environment on tens of thousands of GPUs, HeteroScale has proven its effectiveness, increasing average GPU utilization by a significant 26.6 percentage points and saving hundreds of thousands of GPU-hours daily, all while upholding stringent service level objectives.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TurboServe: Serving Streaming Video Generation Efficiently and Economically

    cs.DC 2026-06 unverdicted novelty 7.0

    TurboServe introduces the first serving system for streaming video generation workloads, using migration-aware placement and load-driven autoscaling to cut worst-case latency by 37.5% and GPU cost by 37.2%.

  2. Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda

    cs.DC 2026-04 unverdicted novelty 2.0

    This research agenda argues that cloud-native architectures, microservices, autoscaling, and emerging trends like serverless inference and federated learning are required to make large language models efficient and scalable.

Reference graph

Works this paper leans on

60 extracted references · 40 canonical work pages · cited by 2 Pith papers

  1. [1]

    Accessed 2025-7-24

    Struct GPUInfo—definition of gpu utilization.https: //docs.nvidia.com/holoscan/sdk-user-guide/ api/cpp/structholoscan_1_1GPUInfo.html, . Accessed 2025-7-24

    work page 2025

  2. [2]

    https:// developer.nvidia.com/docs,

    Sm activity metric in nsight compute. https:// developer.nvidia.com/docs, . See Nsight discus- sion of sm__cycles_active

    work page

  3. [3]

    Megha et al. Agarwal. Llm inference per- formance engineering: Best practices, 2023. URL https://www.databricks.com/blog/ llm-inference-performance-engineering-best-practices

    work page 2023

  4. [4]

    Gula- vani, Alexey Tumanov, and Ramachandran Ram- jee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gula- vani, Alexey Tumanov, and Ramachandran Ram- jee. Taming throughput-latency tradeoff in llm inference with sarathi-serve, 2024. URL https: //arxiv.org/abs/2403.02310

    Pith/arXiv arXiv 2024

  5. [5]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, page...

    work page 2022

  6. [6]

    Gpu utilization is a misleading metric, 2025

    Roanak Baviskar. Gpu utilization is a misleading metric, 2025. URL https://www.trainy.ai/blog/ gpu-utilization-misleading

    work page 2025

  7. [7]

    Kvdirect: Dis- tributed disaggregated llm inference.arXiv preprint arXiv:2501.14743, 2024

    Shiyang Chen, Rain Jiang, Dezhi Yu, Jinlai Xu, Mengyuan Chao, Fanlong Meng, Chenyu Jiang, Wei Xu, and Hang Liu. Kvdirect: Dis- tributed disaggregated llm inference.arXiv preprint arXiv:2501.14743, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Leveraging endpoint flexibility in data- intensive clusters

    Mosharaf Chowdhury, Srikanth Kandula, and Ion Stoica. Leveraging endpoint flexibility in data- intensive clusters. ACM SIGCOMM Computer Communication Review, 43(4):231–242, 2013

    work page 2013

  9. [9]

    Efficient coflow scheduling with varys

    Mosharaf Chowdhury, Yuan Zhong, and Ion Sto- ica. Efficient coflow scheduling with varys. In Proceedings of the 2014 ACM conference on SIGCOMM, pages 443–454, 2014

    work page 2014

  10. [10]

    Resource central: Understanding and predict- ing workloads for improved resource management in large cloud platforms

    Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bian- chini. Resource central: Understanding and predict- ing workloads for improved resource management in large cloud platforms. In Proceedings of the 26th Symposium on Operating Systems Principles, pages 153–167, 2017

    work page 2017

  11. [11]

    A complete survey on llm- based ai chatbots.arXiv preprint arXiv:2406.16937, 2024

    Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A complete survey on llm- based ai chatbots.arXiv preprint arXiv:2406.16937, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Paragon: Qos-aware scheduling for heterogeneous datacenters

    Christina Delimitrou and Christos Kozyrakis. Paragon: Qos-aware scheduling for heterogeneous datacenters. Acm SIGPLAN Notices, 48(4):77–88, 2013

    work page 2013

  13. [13]

    Deploy DeepSeek-V3/R1 671b on 8 × H100 and throughput bench- marks

    dzhsurf. Deploy DeepSeek-V3/R1 671b on 8 × H100 and throughput bench- marks. https://github.com/dzhsurf/ deepseek-v3-r1-deploy-and-benchmarks , 2025. Reports ≈ 33 t/s single-user and ≈ 620 output t/s (≈ 3,000 total t/s) at 100-way concurrency on one 8 × H100 node

    work page 2025

  14. [14]

    Autoscale: Dynamic, robust capacity management for multi-tier data centers

    Anshul Gandhi, Mor Harchol-Balter, Ram Raghu- nathan, and Michael A Kozuch. Autoscale: Dynamic, robust capacity management for multi-tier data centers. ACM Transactions on Computer Systems (TOCS), 30(4):1–26, 2012

    work page 2012

  15. [15]

    Firmament: Fast, centralized cluster scheduling at scale

    Ionel Gog, Malte Schwarzkopf, Adam Gleave, Robert NM Watson, and Steven Hand. Firmament: Fast, centralized cluster scheduling at scale. In12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 99–115, 2016

    work page 2016

  16. [16]

    M\’elange: Cost efficient large language model serv- ing by exploiting gpu heterogeneity.arXiv preprint arXiv:2404.14527, 2024

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M\’elange: Cost efficient large language model serv- ing by exploiting gpu heterogeneity.arXiv preprint arXiv:2404.14527, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Tiresias: A {GPU} cluster manager for distributed deep learning

    Juncheng Gu, Mosharaf Chowdhury, Kang G Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A {GPU} cluster manager for distributed deep learning. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 485– 500, 2019

    work page 2019

  18. [18]

    Deepseek-r1: Incen- tivizingreasoningcapabilityinllmsviareinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incen- tivizingreasoningcapabilityinllmsviareinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Mem- serve: Flexible mem pool for building disaggre- gated llm serving with caching, 2024

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Mem- serve: Flexible mem pool for building disaggre- gated llm serving with caching, 2024. URLhttps: //arxiv.org/abs/2406.17565

    Pith/arXiv arXiv 2024

  20. [20]

    Infer- ence without interference: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Infer- ence without interference: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024. 15

    Pith/arXiv arXiv 2024

  21. [21]

    Quincy: fair scheduling for distributed computing clusters

    Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg. Quincy: fair scheduling for distributed computing clusters. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pages 261–276, 2009

    work page 2009

  22. [22]

    Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan

    Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Re- nee St. Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, and Saravan Rajmohan. Serving models, fast and slow:optimizing heteroge- neous llm inferencing workloads at scale, 2025. URL https://arxiv.org/abs/2502.14617

    arXiv 2025

  23. [23]

    Hexgen: Generative inference of large language model over heterogeneous environment

    Youhe Jiang, Ran Yan, Xiaozhe Yao, Yang Zhou, Beidi Chen, and Binhang Yuan. Hexgen: Generative inference of large language model over heterogeneous environment. arXiv preprint arXiv:2311.11514, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Netcache: Balancing key-value stores with fast in-network caching

    Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate Foster, Changhoon Kim, and Ion Stoica. Netcache: Balancing key-value stores with fast in-network caching. InProceedings of the 26th symposium on operating systems principles, pages 121–136, 2017

    work page 2017

  25. [25]

    P/d- serve: Serving disaggregated large language model at scale, 2024

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, Tiandeng Wu, Xing Chu, Ruizhi Huan, Li Ma, Xiao You, Wenting Zhou, Yunpeng Ye, Wen Liu, Xiangkun Xu, Yongsheng Zhang, Tiantian Dong, Jiawei Zhu, Zhe Wang, Xijian Ju, Jianxun Song, Haoliang Cheng, Xiaojing Li, Jian- dong Ding, Hefei Gu...

    work page 2024

  26. [26]

    Morpheus: Towards automated {SLOs} for en- terprise clusters

    Sangeetha Abdu Jyothi, Carlo Curino, Ishai Men- ache, Shravan Matthur Narayanamurthy, Alexey Tu- manov, Jonathan Yaniv, Ruslan Mavlyutov, Inigo Goiri, Subru Krishnan, Janardhan Kulkarni, et al. Morpheus: Towards automated {SLOs} for en- terprise clusters. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 117–134, 2016

    work page 2016

  27. [27]

    Pod-attention: Unlocking full prefill-decode overlap for faster llm inference

    AdityaKKamath, RamyaPrabhu, JayashreeMohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-attention: Unlocking full prefill-decode overlap for faster llm inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 897–912, 2025

    work page 2025

  28. [28]

    Keda: Kubernetes event-driven autoscaling

    KEDA. Keda: Kubernetes event-driven autoscaling. https://keda.sh/, 2023. Accessed: 2025-07-24

    work page 2023

  29. [29]

    Kubernetes horizontal pod au- toscaler

    Kubernetes. Kubernetes horizontal pod au- toscaler. https://kubernetes.io/docs/tasks/ run-application/horizontal-pod-autoscale/,

    work page

  30. [30]

    Kubernetes vertical pod autoscaler

    Kubernetes. Kubernetes vertical pod autoscaler. https://github.com/kubernetes/autoscaler/ tree/master/vertical-pod-autoscaler, 2023. Accessed: 2025-07-24

    work page 2023

  31. [31]

    Kubernetes: Production-grade container orchestration

    Kubernetes Authors. Kubernetes: Production-grade container orchestration. https://kubernetes.io/,

    work page

  32. [32]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient mem- ory management for large language model serving with pagedattention, 2023. URL https://arxiv. org/abs/2309.06180

    Pith/arXiv arXiv 2023

  33. [33]

    Accessed: 2025-07-24

    work page 2025

  34. [34]

    Themis: Fair and efficient {GPU} cluster scheduling

    Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. Themis: Fair and efficient {GPU} cluster scheduling. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 289– 304, 2020

    work page 2020

  35. [35]

    Alpaserve: Statistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vin- cent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. Alpaserve: Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663– 679, 2023

    work page 2023

  36. [36]

    Mastering llm techniques: Inference optimization, 2023

    NVIDIA. Mastering llm techniques: Inference optimization, 2023. URL https://developer.nvidia.com/blog/ mastering-llm-techniques-inference-optimization

    work page 2023

  37. [37]

    {Heterogeneity-Aware} cluster scheduling policies for deep learning workloads

    Deepak Narayanan, Keshav Santhanam, Fiodar Kazhamiaka, Amar Phanishayee, and Matei Za- haria. {Heterogeneity-Aware} cluster scheduling policies for deep learning workloads. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 481–498, 2020

    work page 2020

  38. [38]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai. com/blog/chatgpt, 2022

    work page 2022

  39. [39]

    Tensorrt-llm: A deep learning compiler for large language models

    NVIDIA. Tensorrt-llm: A deep learning compiler for large language models. https://github.com/ NVIDIA/TensorRT-LLM, 2023. Accessed: 2025-07-24

    work page 2023

  40. [40]

    Splitwise: Efficient generative llm inference using phase splitting, 2024

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting, 2024. URL https://arxiv.org/abs/2311.18677

    Pith/arXiv arXiv 2024

  41. [41]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In2024 ACM/IEEE 51st AnnualInternationalSymposium on Computer Architecture (ISCA), pages 118–132. IEEE, 2024. 16

    work page 2024

  42. [42]

    Deepspeed- moe: Advancing mixture-of-experts inference and trainingtopowernext-generationaiscale, 2022

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He. Deepspeed- moe: Advancing mixture-of-experts inference and trainingtopowernext-generationaiscale, 2022. URL https://arxiv.org/abs/2201.05596

    Pith/arXiv arXiv 2022

  43. [43]

    Mooncake: A kvcache-centric disaggre- gated architecture for llm serving.arXiv preprint arXiv:2407.00079, 2024

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xin- ran Xu. Mooncake: A kvcache-centric disaggre- gated architecture for llm serving.arXiv preprint arXiv:2407.00079, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Déjàvu: Kv- cache streaming for fast, fault-tolerant generative llm serving, 2024

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjàvu: Kv- cache streaming for fast, fault-tolerant generative llm serving, 2024. URL https://arxiv.org/abs/ 2403.01876

    Pith/arXiv arXiv 2024

  45. [45]

    Dynamollm: Designing llm inference clusters for performance and energy ef- ficiency, 2024

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas, and Esha Choukse. Dynamollm: Designing llm inference clusters for performance and energy ef- ficiency, 2024. URLhttps://arxiv.org/abs/2408. 00741

    work page 2024

  46. [46]

    Burstgpt: A real-world workload dataset to optimize llm serving systems, 2025

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Burstgpt: A real-world workload dataset to optimize llm serving systems, 2025. URL https://arxiv.org/abs/2401. 17644

    work page 2025

  47. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  48. [48]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xu- anzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Deepscaling: mi- croservices autoscaling for stable cpu utilization in large scale cloud systems

    Ziliang Wang, Shiyi Zhu, Jianguo Li, Wei Jiang, KK Ramakrishnan, Yangfei Zheng, Meng Yan, Xi- aohong Zhang, and Alex X Liu. Deepscaling: mi- croservices autoscaling for stable cpu utilization in large scale cloud systems. InProceedings of the 13th symposium on cloud computing, pages 16–30, 2022

    work page 2022

  50. [50]

    Gandiva: Introspec- tive cluster scheduling for deep learning

    Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhen- hua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, et al. Gandiva: Introspec- tive cluster scheduling for deep learning. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 595–610, 2018

    work page 2018

  51. [51]

    Skylb: A locality-aware cross-region load balancer for llm inference

    Tian Xia, Ziming Mao, Jamison Kerney, Ethan J Jackson, Zhifei Li, Jiarong Xing, Scott Shenker, and Ion Stoica. Skylb: A locality-aware cross-region load balancer for llm inference. arXiv preprint arXiv:2505.24095, 2025

    arXiv 2025

  52. [52]

    When search engine services meet large language models: Visions and challenges, 2024

    Haoyi Xiong, Jiang Bian, Yuchen Li, Xuhong Li, Mengnan Du, Shuaiqiang Wang, Dawei Yin, and Sumi Helal. When search engine services meet large language models: Visions and challenges, 2024. URL https://arxiv.org/abs/2407.00128

    Pith/arXiv arXiv 2024

  53. [53]

    {AntMan}: Dynamic scaling on {GPU} clusters for deep learning

    Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. {AntMan}: Dynamic scaling on {GPU} clusters for deep learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 533–548, 2020

    work page 2020

  54. [54]

    Orca: A distributed serving system for transformer-based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    work page 2022

  55. [55]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557– 62583, 2024

    work page 2024

  57. [57]

    Day zero benchmarks for qwen 3 with sglang on baseten

    Yineng Zhang, Michael Feil, and Philip Kiely. Day zero benchmarks for qwen 3 with sglang on baseten. https://www.baseten.co/blog/ day-zero-benchmarks-for-qwen-3-with-sglang-on-baseten ,

    work page

  58. [58]

    Megascale- infer: Serving mixture-of-experts at scale with disaggregated expert parallelism

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Ce- sar A Stuardo, Dongyang Wang, Xinlei Zhang, Huap- ing Zhou, Haoran Wei, Yang Cheng, et al. Megascale- infer: Serving mixture-of-experts at scale with disaggregated expert parallelism. arXiv preprint arXiv:2504.02263, 2025. 18 Appendix 19 A Extra Analysis of Service Metrics (a) Normalized TPS (b) Normalized...

    Pith/arXiv arXiv 2025

  59. [60]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193– 210, 2024. 17

    work page 2024

  60. [2025]

    Benchmarks show ≈ 45 t/s per-user and ≈ 1,400 total t/s on 4 × H100 GPUs; extrapolates to low-thousands t/s on8 × H100 GPUs

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.