Pith. sign in

REVIEW 4 major objections 7 minor 44 references

CoCoScale scales hot LLM layers onto idle GPUs instead of spinning up full replicas, cutting scale-up time by ~98–99% and hitting 100% SLO under production traces.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-11 21:07 UTC pith:KBZRN2DR

load-bearing objection Clean layer-wise DP for LLM serving that really does cut scale-up from minutes to seconds under skew; the 100% SLO numbers only hold when cold devices exist, which the 4-GPU setup guarantees. the 4 major comments →

arxiv 2607.04181 v1 pith:KBZRN2DR submitted 2026-07-05 cs.DC

CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

Jingfeng Wu , Yiyuan He , Minxian Xu , Xitong Gao , Chong Ma , Le Chen , Min Shen , Lin Qu
show 2 more authors
Kejiang Ye CHengzhong Xu
This is my paper
classification cs.DC
keywords LLM inference servinglayer-wise scalingelastic data parallelismcold-start reductionworkload skewnessSLO attainmentresource scavenging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Online LLM serving routinely suffers from extreme hot/cold skew: a few popular model instances absorb most traffic while others sit underutilized. Traditional autoscalers can only add or remove whole model replicas, so every scale-up pays a multi-minute cold start and every scale-down leaves the system exposed to the next spike. CoCoScale treats Transformer layers as the unit of scaling. It reclaims free memory and cycles from cold devices, replicates only the hot layers onto those devices, and runs them as layer-wise data-parallel workers. Because only a fraction of the model moves and communication is confined to segment boundaries, the system can raise or lower capacity in seconds rather than minutes. On Alibaba and Azure production traces the method delivers 20.7–28.1 % lower average latency and perfect SLO compliance while leaving model architecture and hardware unchanged.

Core claim

The layered structure of Transformer models supplies a natural, non-integer scaling unit: by selectively replicating consecutive layers of a hot instance onto reclaimed cold devices and forming layer-wise data parallelism, CoCoScale expands capacity without full-model cold starts, reducing scale-up latency by 97.9–99.3 % and achieving 100 % SLO attainment under production traces.

What carries the argument

Layer-wise data parallelism (controlled by consecutive layer count N and parallelism degree P), together with a ring-based scatter-then-exchange transfer that keeps communication cost independent of N and a distance-aware controller that chooses the nearest feasible (N, P) configuration.

Load-bearing premise

There must be underutilized devices whose free memory can safely host the chosen layer replicas without starving their original tenants.

What would settle it

Run the same Alibaba/Azure traces on a cluster whose utilization never drops below the 70 % idle threshold; if CoCoScale can no longer meet SLOs without falling back to full-instance cold starts, the scavenging premise fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Autoscalers can treat capacity as a continuous rather than integer variable, matching demand curves more tightly and reducing over-provisioning.
  • Cold-start latency ceases to be the dominant barrier to elastic LLM serving when only layers, not entire models, must be moved.
  • Existing static tensor- or pipeline-parallel deployments can be left intact; layer-wise replication sits on an orthogonal axis and can be added at runtime.
  • The same reclaimed-resource pool can be shared across heterogeneous model instances without violating user-model affinity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same layer-scavenging idea could be applied inside disaggregated prefill/decode clusters, treating only the decode-side layers as elastic units.
  • If interconnect bandwidth falls well below NVLink (e.g., PCIe 4.0 or Ethernet), the transfer-time advantage shrinks and the method’s agility claim would need re-measurement.
  • A multi-tenant scheduler that jointly optimizes which cold devices donate layers could turn the current greedy heuristic into a cluster-wide packing problem.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. CoCoScale addresses workload skewness in online LLM serving by replacing coarse instance-level scale-up/down with layer-wise data parallelism: selected consecutive Transformer layers of a hot instance are replicated onto idle capacity reclaimed from cold devices, with scatter–gather only at segment boundaries. The paper contributes (i) a replication and ring-based chunked transfer design, (ii) a cost/speedup model (Eqs. 1–10) used to rank (N,P) configurations, (iii) a priority-greedy initialization (Alg. 1) and a unified distance-aware scaling loop (Alg. 2) with fallback to full-instance scale, and (iv) a Monitor–Controller–Scheduler prototype on Nano-vLLM. On 4 H20 GPUs with Qwen3-8B/14B/32B under Alibaba and Azure production traces, the authors report 97.9%–99.3% lower scale-up latency than cold full-instance start, 20.7%–28.1% lower average E2E latency versus an Alibaba autoscaler, and 100% SLO attainment (Table 1, Figs. 8–9).

Significance. If the results generalize beyond the evaluated regime, layer-wise scavenging is a practically useful middle ground between static over-provisioning and full-replica cold starts, and it is a natural fit to Transformer structure. Strengths that should be credited include the ablation surfaces (Figs. 5–6), the closed-form ranking model that makes the controller tractable, the ring-multicast and CUDA-Graph compatibility engineering, and evaluation on two real production traces across three model sizes. The work is timely for multi-tenant LLM clusters where hot/cold skew is common. The main open question is how much of the headline SLO and E2E gains survive when scavengable idle capacity is scarce or absent—the regime the current 4-GPU skewed setup does not stress.

major comments (4)
  1. §6.1 Multi-Instance Configuration, Alg. 1 (D_pool with τ_idle=70%), and Alg. 2 lines 18–19: the evaluation deploys 4 concurrent instances on 4 GPUs under deliberately skewed traces, which by construction keeps a non-empty cold pool for scavenging. Table 1’s 100% SLO attainment and the 20.7%–28.1% average-latency gains in Fig. 8 are therefore demonstrated only when residual idle capacity exists. The paper never reports a uniform-load / fully-hot run in which every device exceeds τ_idle and the controller must take the TriggerInstanceScale fallback. That fallback reintroduces the 74–241 s cold-start path of Fig. 9b—the exact cost the abstract claims to eliminate. Either evaluate and quantify the fallback path (frequency and SLO impact under uniform or near-uniform load), or explicitly qualify the SLO/E2E claims as conditional on residual idle capacity. As written, the strongest production
  2. §6.1 Baselines and Fig. 9b: the Alibaba Autoscaler baseline is described as using a warm-pool strategy (scaled-in instances retained), yet the 97.9%–99.3% scale-up reduction is computed against cold from-scratch startup (74–241 s, dominated by model loading and CUDA-graph capture). If the production comparator already warms instances, the fair comparison is warm-pool scale-out latency, not cold start. Please report the actual scale-out latency of the warm-pool Alibaba policy under the same traces, and restate the percentage reduction against that number (or clearly separate “vs cold start” from “vs production autoscaler”).
  3. Table 1 and Fig. 8: no error bars, confidence intervals, number of replay runs, or variance across seeds are reported for Avg/P99 latency or SLO attainment. Production-trace replay and GPU scheduling are stochastic; single-run curves make it hard to judge whether the reported 20.7%–28.1% gains and perfect SLO compliance are robust. At minimum, report multi-run means with intervals (or bootstrap CIs) for the headline metrics.
  4. §4.2 Eqs. (6)–(7) and Alg. 2: the speedup model is presented as an approximation used for relative ranking, yet Alg. 2 filters candidates by the absolute predicate S(N,P) ≥ α_req derived from the current latency/SLO ratio. Ranking preservation alone does not guarantee that the first feasible candidate actually meets the required speedup. Please report quantitative model fidelity on the ablation grid (e.g., rank correlation and MAE/MAPE of predicted vs measured speedup for held-out (N,P) points), and discuss mis-prediction cases that could select an under-powered configuration.
minor comments (7)
  1. Section 6 title: “Perforamnce Evaluations” → “Performance Evaluations”.
  2. Eq. (5): T(P) is written as independent of N and linear in (P−1); briefly state how this maps to the bidirectional ring Scatter-then-Exchange of §3.2 (full-duplex assumption, number of hops, and whether the constant 2 already absorbs ring diameter).
  3. §3.1 / §5.2: discuss interaction with continuous batching and paged attention when sub-batches have unequal sequence lengths or when KV-cache redistribution occurs on a P change mid-decode.
  4. Free parameters τ_idle=70%, δ, and P_max are stated empirically; a short sensitivity sweep (even offline) would strengthen confidence that conclusions are not brittle to these choices.
  5. §6.3 already flags interconnect sensitivity as future work; a single PCIe-only or multi-node Ethernet data point would make that limitation concrete rather than purely qualitative.
  6. Fig. 1 narrative (states ①–④) is hard to follow from text alone; ensure the published figure labels match the caption’s instance-wise vs layer-wise timeline.
  7. Related work (§7): tighten the contrast with LoongServe, FlexPipe, and Llumnix on what is uniquely enabled by layer-granularity DP versus instance migration or pipeline refactoring.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose speedup model is an approximate ranking heuristic fitted to ablations, not a claimed absolute first-principles prediction of the headline latency/SLO numbers.

full rationale

The paper's strongest claims (97.9–99.3% scale-up latency reduction, 20.7–28.1% average E2E latency cut, 100% SLO attainment) are measured end-to-end on production traces against baselines (Table 1, Figure 8, Figure 9); they do not reduce by construction to any fitted constant or self-defined quantity. Section 4.2 presents an Amdahl-style analytical model S(N,P) that is explicitly described as 'an analytical approximation fitted from empirical observations' whose 'value lies in preserving the relative ordering of candidate configurations' rather than cycle-accurate absolute prediction; the model is used only inside Algorithms 1–2 to rank feasible (N,P) tuples. Ablation surfaces (Figures 5–6) are independent empirical measurements that the model is checked against for monotonicity, not inverted to produce the main results. Thresholds (τ_idle=70%, δ) are stated as empirical safety valves with no derivation that equates them to the reported speedups. Self-citations appear only in related-work positioning (e.g., UELLM) and are not load-bearing for any uniqueness claim or uniqueness theorem. No self-definitional loop, no fitted-input-called-prediction of the headline metrics, and no ansatz smuggled via overlapping-author uniqueness results. The evaluation setup (4 instances on 4 GPUs under deliberately skewed traces) is a validity concern for generalizability, not circularity of the derivation chain. Score 1 reflects only the minor, non-load-bearing use of a fitted ranking model.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

Systems paper whose claims rest on a small set of engineering thresholds, a homogeneous-cluster modeling assumption, and the empirical availability of cold devices. No new physical entities are postulated; free parameters are the usual controller knobs.

free parameters (3)
  • τ_idle (idle utilization threshold) = 70%
    Devices admitted to the cold pool only if Load(d) < τ_idle; set empirically to 70% (Section 4.3).
  • δ (latency deviation tolerance)
    Triggers reconfiguration when |latency − SLO|/SLO > δ (Algorithm 2); value left as a free controller parameter.
  • P_max (maximum parallelism degree) = 4
    Search-space bound used in both initialization and online algorithms; example value 4 for Qwen-32B.
axioms (4)
  • domain assumption Cluster devices are homogeneous in compute capacity t_comp and interconnect bandwidth B (Section 4.2).
    Simplifies the closed-form speedup model; later generalized but the main algorithms still rely on the homogeneous case.
  • domain assumption Sub-batches can be equally (or nearly equally) partitioned across replicas without straggler effects dominating (Eqs. 1–2, 8).
    Required for the claimed linear amortization of activation memory and compute.
  • ad hoc to paper Communication occurs only at segment boundaries of consecutive replicated layers, so T(P) is independent of N (Eq. 5).
    Design choice that makes the model tractable; holds only because the system treats N consecutive layers as a monolithic unit.
  • domain assumption Interference on co-located tenants during weight/KV migration stays below 3% (Section 5.2).
    Measured on the authors’ testbed; treated as a general isolation guarantee.
invented entities (1)
  • layer-wise data-parallel replica set (N consecutive layers at parallelism P) no independent evidence
    purpose: Atomic unit of elastic capacity that can be placed on cold devices without full-model redeployment.
    Core abstraction introduced by the paper; not present as a first-class scaling primitive in the cited literature.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving." pith.science (2026). https://pith.science/paper/KBZRN2DR

@misc{pith2026260704181,
  author       = {Pith},
  title        = {Pith review of: CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KBZRN2DR}},
  note         = {Machine review of arXiv:2607.04181}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Online large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for fine-grained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%-99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7\%--28.1\% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.

Figures

Figures reproduced from arXiv: 2607.04181 by Chengzhong Xu, Chong Ma, Jingfeng Wu, Kejiang Ye, Le Chen, Lin Qu, Min Shen, Minxian Xu, Xitong Gao, Yiyuan He.

Figure 1
Figure 1. Figure 1: Comparison between instance-wise and layer-wise scaling. (1) Instance-level scaling suffers from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: , these coarse-grained strategies face significant hurdles [9, 40]. A primary obstacle is the high provisioning latency caused by the inherent cold-start problem. Because initializing a new model instance involves time-consuming operations like container setup and loading massive weights, the system exhibits a pronounced temporal lag. This is visible in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the replication for decoder layers. Yellow blocks represent the layers deployed on the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of data transfer strategies. (a) Single-source multicast causes linear delay and “Resend” [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance analysis of different number of continuous replicas and parallelism under varying request [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: 3D surface plots of end-to-end latency as a function of layer count [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The system overview of CoCoScale, illustrating the closed-loop interaction between the Monitor, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Temporal latency comparison across production traces. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Scaling overhead comparison. (a) CoCoScale agility. (b) Traditional scaling bottlenecks. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 4 canonical work pages

  1. [1]

    2025. GPT-5.2. https://openai.com/index/introducing-gpt-5-2/. Accessed: Dec. 20, 2025

    2025

  2. [2]

    Ollama - Get up and running with large language models

    2025. Ollama - Get up and running with large language models. https://ollama.com/. Accessed: Apr. 23, 2025

    2025

  3. [3]

    Alibaba. 2026. Alibaba Cloud. Online. https://www.alibabacloud.com Alibaba Cloud free service product catalog and introduction page. Accessed: 2026-01-23

    2026

  4. [4]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks.arXiv preprint arXiv:2412.15204(2024). ACM Trans. Arch. Code Optim., Vol. 1, No. 1, Article 1. Publication date: J...

    Pith/arXiv arXiv 2024

  5. [5]

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving heterogeneous machine learning models on Multi-GPU servers with Spatio-Temporal sharing. In2022 USENIX Annual Technical Conference (USENIX ATC 22). 199–216

    2022

  6. [6]

    Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters.Commun. ACM51, 1 (2008), 107–113

    2008

  7. [7]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and Bing-Li Wang et al. 2024. DeepSeek-V3 Technical Report.ArXiv abs/2412.19437 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving.arXiv preprint arXiv:2404.02015(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 135–153

    2024

  10. [10]

    GeeeekExplorer. 2025. Nano-vLLM: A Lightweight vLLM Implementation Built from Scratch. https://github.com/ GeeeekExplorer/nano-vllm. GitHub repository

    2025

  11. [11]

    Google Cloud. 2024. Vertex AI. https://cloud.google.com/vertex-ai. Accessed: 2024-05-20

    2024

  12. [12]

    Yiyuan He, Minxian Xu, Jingfeng Wu, Wanyi Zheng, Kejiang Ye, and Chengzhong Xu. 2024. UELLM: A Unified and Efficient Approach for Large Language Model Inference Serving. InService-Oriented Computing: 22nd International Conference, ICSOC 2024, Tunis, Tunisia, December 3–6, 2024, Proceedings, Part I(Tunis, Tunisia). Springer-Verlag, Berlin, Heidelberg, 218–...

    work page doi:10.1007/978-981-96-0805-8_16 2024

  13. [13]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. arXiv:2401.11181 [cs.DC]

    Pith/arXiv arXiv 2024

  14. [14]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, and Lianmin et al. Zheng. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)(SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611–626. doi:10.1145/3600006.3613165

    doi:10.1145/3600006.3613165 2023

  15. [15]

    Hao Lan, Ziang Zhou, Qi Zhu, Wei Yan, Qinfen Hao, Xiaochun Ye, Yong Liu, and Ninghui Sun. 2026. Heterogeneous Confidential Computing System for Large Language Models: A Survey.ACM Trans. Archit. Code Optim.23, 1, Article 4 (March 2026), 26 pages. doi:10.1145/3779307

    work page doi:10.1145/3779307 2026

  16. [16]

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. 2022. xFormers: A modular and hackable Transformer modelling library. https://github.com/facebookresearch/ xformers

    2022

  17. [17]

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, and Yanping Huang et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA, 663–679

    2023

  18. [18]

    Yanying Lin, Shijie Peng, Chengzhi Lu, Chengzhong Xu, and Kejiang Ye. 2025. FlexPipe: Adapting Dynamic LLM Serving Through Inflight Pipeline Refactoring in Fragmented Serverless Clusters.arXiv preprint arXiv:2510.11938 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Chiheng Lou, Sheng Qi, Chao Jin, Dapeng Nie, Haoran Yang, Yu Ding, Xuanzhe Liu, and Xin Jin. 2025. HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds.arXiv preprint arXiv:2502.15524(2025)

    arXiv 2025

  20. [20]

    Microsoft. 2024. Azure Machine Learning. https://azure.microsoft.com/en-us/products/machine-learning. Accessed: 2024-05-20

    2024

  21. [21]

    Microsoft Azure. 2024. Microsoft Azure Public Dataset. https://github.com/Azure/AzurePublicDataset. Accessed: 2026-01-27

    2024

  22. [22]

    OpenRouter. 2024. OpenRouter: A unified interface for LLMs. https://openrouter.ai/. Accessed: 2024-05-20

    2024

  23. [23]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Inigo Goiri, Aashaka Shah, Saeed Maleki, and Ricardo Bianchini

  24. [24]

    Splitwise: Efficient Generative LLM Inference Using Phase Splitting.2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)(2023), 118–132

    2024

  25. [25]

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2024. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079 [cs.DC]

    Pith/arXiv arXiv 2024

  26. [26]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron- LM: Training Multi-billion Parameter Language Models Using Model Parallelism.arXiv preprint arXiv:1909.08053 (2019)

    Pith/arXiv arXiv 2019

  27. [27]

    Sudipta Saha Shubha, Haiying Shen, and Anand Iyer. 2024. USHER: Holistic Interference Avoidance for Resource Optimized ML Inference. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 947–964. ACM Trans. Arch. Code Optim., Vol. 1, No. 1, Article 1. Publication date: January 2026. 1:22 Wu et al

    2024

  28. [28]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dy- namic Scheduling for Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 173–191

    2024

  29. [29]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, and Percy Liang et al. 2023. Stanford Alpaca: An Instruction-following LLaMA Model. https://github.com/tatsu-lab/stanford_alpaca. GitHub repository, Accessed: 2025-05-04

    2023

  30. [30]

    Xinyi Wan, Penghui Qi, Guangxing Huang, Min Lin, and Jialin Li. 2025. Pipeoffload: Improving scalability of pipeline parallelism with memory optimization.arXiv preprint arXiv:2503.01328(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems. arXiv:2401.17644 [cs.DC] https://arxiv.org/abs/2401.17644

    Pith/arXiv arXiv 2025

  32. [32]

    Jinhui Wei, Shenggan Cheng, Wei Zhu, Jiazhi Jiang, Dan Huang, Zhiguang Chen, Jiangsu Du, and Yutong Lu. 2026. Dynamic Latency-Throughput Balancing in Distributed Large Model Inference with Interleaved Parallelism.ACM Trans. Archit. Code Optim.23, 1, Article 27 (March 2026), 26 pages. doi:10.1145/3797040

    work page doi:10.1145/3797040 2026

  33. [33]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-t...

    2020

  34. [34]

    Bingya Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism.Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles(2024)

    2024

  35. [35]

    Yuxing Xiang, Xue Li, Kun Qian, Yufan Yang, Diwen Zhu, Wenyuan Yu, Ennan Zhai, Xuanzhe Liu, Xin Jin, and Jingren Zhou. 2025. Aegaeon: Effective GPU pooling for concurrent LLM serving on the market. InProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles. 1030–1045

    2025

  36. [36]

    Minxian Xu, Jingfeng Wu, Shengye Song, Satish Narayana Srirama, Bahman Javad, Rajiv Ranjan, Devki Nandan Jha, Sa Wang, Wenhong Tian, Huanle Xu, Li Li, Zizhao Mo, Shuo Ren, Thomas Kunz, Petar Kochovski, Vlado Stankovski, Kejiang Ye, Chengzhong Xu, and Rajkumar Buyya. 2026. Cloud-native and Distributed Systems for Efficient and Scalable Large Language Model...

    Pith/arXiv arXiv 2026

  37. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Yanan Yang, Laiping Zhao, Yiming Li, Huanyu Zhang, Jie Li, Mingyang Zhao, Xingzhen Chen, and Keqiu Li. 2022. Infless: a native serverless system for low-latency, high-throughput inference. InProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 768–781

    2022

  39. [39]

    Minchen Yu, Ao Wang, Bohui Wu, Yuxuan Liu, Dong Chen, Haoxuan Yu, Wei Wang, Ruichuan Chen, Dapeng Nie, Haoran Yang, and Yu Ding. 2026. Enabling Low-Latency, GPU-Efficient Serverless Inference with Model Swapping. ACM Trans. Archit. Code Optim.(April 2026). doi:10.1145/3800690

    work page doi:10.1145/3800690 2026

  40. [40]

    Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, and Youyou Lu. 2025. Medusa: Accelerating serverless LLM inference with materialization. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 653–668

    2025

  41. [41]

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. 2025. BLITZSCALE: fast and live large model autoscaling with O(1) host caching. InProceedings of the 19th USENIX Conference on Operating Systems Design and Implementation(Boston, MA, USA)(OSDI ’25). USENIX Association, USA, Article 16, 19 pages

    2025

  42. [42]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, and Yanping Huang et al. 2022. Alpa: Automating Inter and Intra-Operator Parallelism for Distributed Deep Learning. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 559–578

    2022

  43. [43]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs.Advances in neural information processing systems37, 62557–62583

    2024

  44. [44]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, and Xuanzhe Liu et al. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, Santa Clara, CA, 193–210. ACM Trans. Arch. Code Optim., Vol. 1, No....

    2024

This paper was first reviewed by grok-4.5 on July 11, 2026.