pith. sign in

arxiv: 2512.13525 · v3 · submitted 2025-12-15 · 💻 cs.DC

Janus: Disaggregating Attention and Experts for Scalable MoE Inference

Zhexiang Zhang , Ye Wang , Yumiao Zhao , Jiayu Xiao , Qianjing Yang , Xiangyu Wang , Jingzhe Jiang , Qizhen Weng
show 5 more authors
Ruichuan Chen Shaohuai Shi Adel N. Toosi Yin Chen Minchen Yu
This is my paper

Pith reviewed 2026-05-16 22:06 UTC · model grok-4.3

classification 💻 cs.DC
keywords mixture of expertsMoE servingdisaggregationGPU schedulingattention layersexpert balancingSLO complianceinference throughput
0
0 comments X

The pith

Disaggregating attention and MoE layers onto separate GPU pools improves per-GPU throughput by up to 4.7 times while meeting latency requirements.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that monolithic deployment of MoE models forces attention and expert layers to share GPU resources despite their differing demands. JANUS separates them into distinct worker pools with independent scaling. An adaptive communication scheme and fast scheduler then balance expert loads and meet latency goals at lower total cost. This setup delivers substantially higher throughput on the same hardware.

Core claim

JANUS disaggregates attention and MoE layers onto separate GPU worker pools, uses an adaptive two-phase communication mechanism, introduces a lightweight microsecond-scale activation scheduler to balance per-layer activated experts, and applies a fine-grained SLO-aware resource scaling scheme to minimize GPU cost under token-level SLOs, achieving up to 4.7x higher per-GPU throughput.

What carries the argument

Disaggregation of attention and MoE layers onto separate GPU worker pools combined with adaptive two-phase communication and a microsecond-scale expert activation scheduler.

Load-bearing premise

That the added communication between separate pools and the scheduler introduce negligible overhead and that workloads show enough expert imbalance to benefit from balancing.

What would settle it

A workload with uniform expert activation across all experts and similar resource profiles for attention and MoE layers would show little or no throughput gain if the disaggregation premise is correct.

Figures

Figures reproduced from arXiv: 2512.13525 by Adel N. Toosi, Jiayu Xiao, Jingzhe Jiang, Minchen Yu, Qianjing Yang, Qizhen Weng, Ruichuan Chen, Shaohuai Shi, Xiangyu Wang, Ye Wang, Yin Chen, Yumiao Zhao, Zhexiang Zhang.

Figure 1
Figure 1. Figure 1: Two distributions of expert activation (left) and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Architecture overview of JANUS. must carefully determine how many resources to allocate to attention and MoE sub-clusters, and how to replicate and place experts across MoE instances. 3 System Design In this section, we introduce the system design of JANUS and elaborate how it addresses the three main challenges. 3.1 Overview [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison between a strawman solution (left) and adaptive two-phase communication (middle and right). [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scheduling workflow of JANUS. expert-activation scheduling across MoE instances at every MoE layer. However, MoE layer execution typically com￾pletes within only a few hundred microseconds according to our measurements ( [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Latency of an attention (top) and MoE layer (bot [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Normalized TPOT under various model variants [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Simulation of scaling decisions for JANUS and SGLang under real-world workloads. 4 16 64 256 512 Batch Size 0.8 0.9 1 Norm. TPOT Base Base+2PC Base+2PC +LB (Janus) [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: presents results across varying batch sizes. First, the two-phase communication (+2PC) substantially alleviates the communication bottleneck inherent in disaggregated ar￾chitectures under heavy workloads. At a batch size of 512, Base+2PC reduces TPOT latency by 18% relative to Base, indicating that optimizing the cross-sub-cluster data transfers is critical for scalability at high data volumes. Second, th… view at source ↗
Figure 12
Figure 12. Figure 12: Overhead of JANUS’s scheduling. able growth trend. These results demonstrate that JANUS ’s scheduling mechanism incurs negligible overhead and does not become a bottleneck even at large batch sizes or larger deployments. 6 Discussion and Related Work Support for heterogeneous hardware. Modern data centers increasingly comprise heterogeneous accelerators, mixing dif￾ferent GPU generations or types [13, 15]… view at source ↗
read the original abstract

Serving large Mixture-of-Experts (MoE) models is challenging because of their large memory footprints, heterogeneous resource demands, and highly dynamic inference workloads. Most existing MoE inference systems deploy the entire model as a monolithic unit, forcing attention and MoE layers to share the same resource configuration despite their different scaling behaviors and resource bottlenecks. Such coarse-grained provisioning leads to resource inefficiency and suboptimal performance. We present JANUS, a scalable and resource-efficient MoE inference system built around three key principles. First, JANUS disaggregates attention and MoE layers onto separate GPU worker pools, enabling independent resource provisioning for the two layer types, and uses an adaptive two-phase communication mechanism for low-latency data exchange. Second, because MoE-layer execution is often memory-bound and highly sensitive to activated-expert imbalance, JANUS introduces a lightweight, microsecond-scale activation scheduler that balances per-layer activated experts across MoE instances to reduce inference latency. Third, JANUS employs a fine-grained, SLO-aware resource scaling scheme that jointly selects attention resources, MoE resources, and expert placement to minimize GPU cost under token-level SLOs. Evaluation shows that JANUS improves per-GPU throughput by up to 4.7x over state-of-the-art MoE inference baselines while satisfying token-level latency SLOs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper presents JANUS, a MoE inference system that disaggregates attention and MoE layers onto separate GPU worker pools with an adaptive two-phase communication mechanism, introduces a microsecond-scale activation scheduler to balance activated experts, and uses an SLO-aware resource scaling scheme to jointly provision attention, MoE, and expert placement. It claims up to 4.7x per-GPU throughput improvement over state-of-the-art baselines while satisfying token-level latency SLOs.

Significance. If the empirical gains prove robust, JANUS would represent a meaningful advance in scalable MoE serving by exploiting the differing resource profiles of attention and expert layers, potentially lowering GPU costs in production inference clusters. The empirical nature of the work (no fitted parameters or closed-form derivations) makes reproducibility of the 4.7x result the key determinant of impact.

major comments (3)
  1. [§5] §5 (Evaluation): The 4.7x per-GPU throughput claim is presented without explicit enumeration of baseline configurations, workload traces, token concurrency levels, or interconnect parameters (PCIe vs. NVLink), which is load-bearing because the central disaggregation benefit rests on the two-phase communication overhead remaining negligible.
  2. [§3.2] §3.2 (Adaptive two-phase communication): No micro-benchmark or sensitivity analysis quantifies the added latency of the two-phase exchange under realistic token rates and contention; if this overhead exceeds a few microseconds it directly erodes the SLO headroom that the independent scaling is supposed to provide.
  3. [§4.3] §4.3 (SLO-aware scaling): The joint optimization of attention/MoE resources and expert placement is described at a high level but lacks an ablation showing how much of the reported gain comes from disaggregation versus the scheduler versus the scaling policy, preventing isolation of the disaggregation contribution.
minor comments (2)
  1. [§3.2] Notation for the two-phase communication phases is introduced without a diagram or pseudocode, making the adaptive decision logic harder to follow.
  2. [Abstract] The abstract states 'up to 4.7x' but the evaluation section should include the exact configuration (model size, batch size, SLO value) that achieves this peak so readers can assess sensitivity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important aspects of reproducibility and component isolation in our evaluation. We address each major comment below and will revise the manuscript to strengthen these areas while preserving the core claims.

read point-by-point responses

  1. Referee: [§5] §5 (Evaluation): The 4.7x per-GPU throughput claim is presented without explicit enumeration of baseline configurations, workload traces, token concurrency levels, or interconnect parameters (PCIe vs. NVLink), which is load-bearing because the central disaggregation benefit rests on the two-phase communication overhead remaining negligible.

    Authors: We agree that explicit enumeration strengthens reproducibility. The revised manuscript will include a new table in §5 that enumerates all baseline systems with their exact configurations, the specific workload traces (including token arrival rates and concurrency levels from 1–128), and interconnect details (NVLink within nodes and PCIe across nodes). We will also add a brief measurement confirming that two-phase communication overhead remains below 3 µs under the evaluated loads, preserving the claimed benefit. revision: yes

  2. Referee: [§3.2] §3.2 (Adaptive two-phase communication): No micro-benchmark or sensitivity analysis quantifies the added latency of the two-phase exchange under realistic token rates and contention; if this overhead exceeds a few microseconds it directly erodes the SLO headroom that the independent scaling is supposed to provide.

    Authors: We acknowledge the absence of a dedicated micro-benchmark. The revision will add a new subsection (or appendix) in §3.2 with micro-benchmarks measuring two-phase exchange latency across token rates of 1–100 tokens/request and under varying contention. Results show overhead of 1–3 µs, which is negligible relative to typical 100–500 ms token-level SLOs. A sensitivity plot will also be included to demonstrate throughput impact. revision: yes

  3. Referee: [§4.3] §4.3 (SLO-aware scaling): The joint optimization of attention/MoE resources and expert placement is described at a high level but lacks an ablation showing how much of the reported gain comes from disaggregation versus the scheduler versus the scaling policy, preventing isolation of the disaggregation contribution.

    Authors: We agree that an ablation is needed to isolate contributions. The revised §5 will include an ablation study comparing (i) full JANUS, (ii) disaggregation alone with static scheduling, (iii) activation scheduler on a monolithic baseline, and (iv) SLO-aware scaling alone. This will quantify the incremental gains, with disaggregation shown to provide the largest share under high-concurrency workloads. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical system evaluation rests on measurements, not derivations that reduce to inputs

full rationale

The paper describes a systems design for disaggregating attention and MoE layers, with an adaptive scheduler and SLO-aware scaling. Its central claims are supported by empirical throughput and latency measurements against baselines rather than any mathematical derivation chain, fitted parameters renamed as predictions, or self-citation load-bearing steps. No equations, ansatzes, or uniqueness theorems are invoked that collapse to the paper's own inputs by construction. The evaluation is externally falsifiable via replication on hardware, satisfying the criteria for non-circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The system assumes standard distributed-systems primitives (low-latency interconnects, GPU memory management) and that MoE workloads exhibit heterogeneous layer scaling and expert imbalance; no new physical constants or fitted parameters are introduced.

axioms (2)
  • domain assumption GPU interconnects support low-latency data movement between attention and MoE pools
    Required for the two-phase communication to remain fast
  • domain assumption Expert activation patterns vary enough across layers and requests to benefit from dynamic balancing
    Central justification for the microsecond scheduler

pith-pipeline@v0.9.0 · 5577 in / 1321 out tokens · 27460 ms · 2026-05-16T22:06:41.324857+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads

    cs.LG 2026-01 unverdicted novelty 7.0

    A renewal-reward analysis yields a closed-form mean-field rule for the optimal Attention/FFN provisioning ratio in disaggregated LLM serving that accounts for stochastic KV-cache growth and matches simulation optima w...

  2. NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding

    cs.DC 2026-05 unverdicted novelty 6.0

    NanoCP introduces request-level dynamic context parallelism to decouple MoE communication from KV cache placement in hybrid data-expert parallel serving, reporting up to 3.27x higher request rates and 2.12x lower P99 ...

  3. Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

    cs.NI 2026-04 unverdicted novelty 6.0

    Switchless topologies such as 3D full-mesh are 20.6-56.2% more cost-effective than scale-up networks for MoE LLM serving, with current link bandwidths over-provisioned by up to 27%.

  4. Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics

    cs.DC 2026-05 accept novelty 4.0

    LLM serving requires mathematical optimization and algorithms with provable guarantees rather than generic heuristics that fail unpredictably on LLM workloads.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 4 Pith papers · 4 internal anchors

  1. [1]

    Taming 12 Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming 12 Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Oper- ating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July 2024. USENIX Association

    work page 2024

  2. [2]

    Gonzalez, Matei Za- haria, and Ion Stoica

    Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xi- aoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Za- haria, and Ion Stoica. Moe-lightning: High-throughput moe inference on memory-constrained gpus. InPro- ceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS ’2...

    work page 2025

  3. [3]

    Efficient and economic large language model inference with attention offloading

    Shaoyuan Chen, Wencong Xiao, Yutong Lin, Mingxing Zhang, Yingdi Shan, Jinlei Jiang, Kang Chen, and Yong- wei Wu. Efficient heterogeneous large language model decoding with model-attention disaggregation.arXiv preprint arXiv:2405.01814, 2025

    work page arXiv 2025

  4. [4]

    DeepSeek-AI. DeepEP. https://github.com/ deepseek-ai/DeepEP, 2025

    work page 2025

  5. [5]

    ServerlessLLM: Low-Latency serverless inference for large language models

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. ServerlessLLM: Low-Latency serverless inference for large language models. In18th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 24), pages 135–153, Santa Clara, CA, July 2024. USENIX Association

    work page 2024

  6. [6]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Pre- gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre- gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1018–1031, 2024

    work page 2024

  8. [8]

    Shmoys, and Eva Tardos

    Jan Karel Lenstra, David B. Shmoys, and Eva Tardos. Approximation algorithms for scheduling unrelated par- allel machines. In28th Annual Symposium on Founda- tions of Computer Science (sfcs 1987), pages 217–224, 1987

    work page 1987

  9. [9]

    Accelerating distributed MoE training and inference with lina

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 945–959, Boston, MA, July 2023. USENIX Association

    work page 2023

  10. [10]

    DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, econom- ical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 techni- cal report.arXiv preprint arXiv:2412.19437, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    2025.Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving

    Ziming Liu, Boyu Tian, Guoteng Wang, Zhen Jiang, Peng Sun, Zhenhua Han, Tian Tang, Xiaohe Hu, Yanmin Jia, Yan Zhang, et al. Expert-as-a-service: Towards efficient, scalable, and robust large-scale moe serving. arXiv preprint arXiv:2509.17863, 2025

    work page arXiv 2025

  13. [13]

    Helix: Serving large language models over heterogeneous gpus and net- work via max-flow

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. Helix: Serving large language models over heterogeneous gpus and net- work via max-flow. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASP- LOS ’25, pages 586–602, 2025

    work page 2025

  14. [14]

    Spotserve: Serv- ing generative large language models on preemptible instances

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. Spotserve: Serv- ing generative large language models on preemptible instances. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS ’24, pages 1112–1127, New York, NY , USA, 20...

    work page 2024

  15. [15]

    Hetis: Serving llms in heterogeneous gpu clusters with fine-grained and dynamic parallelism

    Zizhao Mo, Jianxiong Liao, Huanle Xu, Zhi Zhou, and Chengzhong Xu. Hetis: Serving llms in heterogeneous gpu clusters with fine-grained and dynamic parallelism. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Anal- ysis (SC ’25), pages 1710–1724, New York, NY , USA,

    work page

  16. [16]

    Association for Computing Machinery

    work page

  17. [17]

    Nvidia collective communications library (nccl).https://github.com/NVIDIA/nccl, 2025

    NVIDIA. Nvidia collective communications library (nccl).https://github.com/NVIDIA/nccl, 2025

    work page 2025

  18. [18]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. InProceedings of the 51st Annual Interna- tional Symposium on Computer Architecture, ISCA ’24, pages 118–132. IEEE Press, 2025

    work page 2025

  19. [19]

    https://github.com/sgl-project/ sglang, 2025

    SGLang. https://github.com/sgl-project/ sglang, 2025

    work page 2025

  20. [20]

    13 Msccl++: Rethinking gpu communication abstrac- tions for cutting-edge ai applications.arXiv preprint arXiv:2504.09014, 2025

    Aashaka Shah, Abhinav Jangda, Binyang Li, Caio Rocha, Changho Hwang, Jithin Jose, Madan Musu- vathi, Olli Saarikivi, Peng Cheng, Qinghua Zhou, et al. 13 Msccl++: Rethinking gpu communication abstrac- tions for cutting-edge ai applications.arXiv preprint arXiv:2504.09014, 2025

    work page arXiv 2025

  21. [21]

    Ucx: an open source framework for hpc network apis and beyond

    Pavel Shamis, Manjunath Gorentla Venkata, M Graham Lopez, Matthew B Baker, Oscar Hernandez, Yossi Itigin, Mike Dubman, Gilad Shainer, Richard L Graham, Liran Liss, et al. Ucx: an open source framework for hpc network apis and beyond. In2015 IEEE 23rd Annual Symposium on High-Performance Interconnects, pages 40–43. IEEE, 2015

    work page 2015

  22. [22]

    ShareGPT Teams.https://sharegpt.com/, 2023

    work page 2023

  23. [23]

    DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

    Jovan Stojkovic, Chaojie Zhang, Inigo Goiri, Josep Tor- rellas, and Esha Choukse. DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency. In2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1348–1362, Los Alamitos, CA, USA, March 2025. IEEE Computer Society

    work page 2025

  24. [24]

    Llumnix: Dynamic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 173–191, Santa Clara, CA, July 2024. USENIX Association

    work page 2024

  25. [25]

    https://github.com/vllm-project/vllm, 2025

    vLLM. https://github.com/vllm-project/vllm, 2025

    work page 2025

  26. [26]

    Step-3 is large yet affordable: Model-system co-design for cost-effective decoding

    Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507.19427, 2025

    work page arXiv 2025

  27. [27]

    Burstgpt: A real-world workload dataset to optimize llm serving systems

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Burstgpt: A real-world workload dataset to optimize llm serving systems. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’25), New York, NY , USA...

    work page 2025

  28. [28]

    Roofline: an insightful visual performance model for multicore architectures.Commun

    Samuel Williams, Andrew Waterman, and David Patter- son. Roofline: an insightful visual performance model for multicore architectures.Commun. ACM, 52(4):65– 76, April 2009

    work page 2009

  29. [29]

    xAI.https://x.ai/blog/grok-os, 2024

    work page 2024

  30. [30]

    xDeepServe: Model-as-a-service on Huawei CloudMa- trix384, 2025

    Ao Xiao, Bangzheng He, Baoquan Zhang, Baoxing Huai, Bingji Wang, Bo Wang, Bo Xu, Boyi Hou, et al. xDeepServe: Model-as-a-service on Huawei CloudMa- trix384, 2025

    work page 2025

  31. [31]

    Moe-infinity: Efficient moe inference on per- sonal machines with sparsity-aware expert cache, 2024

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe-infinity: Efficient moe inference on per- sonal machines with sparsity-aware expert cache, 2024

    work page 2024

  32. [32]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  33. [33]

    Torpor: Gpu-enabled serverless computing for low-latency, resource-efficient inference

    Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xi- aonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, Haoran Yang, et al. Torpor: Gpu-enabled serverless computing for low-latency, resource-efficient inference. InProceedings of the USENIX Annual Tech- nical Conference, 2025

    work page 2025

  34. [34]

    Lambdas- cale: Enabling fast scaling for serverless large language model inference,

    Minchen Yu, Rui Yang, Chaobo Jia, Zhaoyuan Su, Sheng Yao, Tingfeng Lan, Yuchen Yang, Yue Cheng, Wei Wang, Ao Wang, and Ruichuan Chen. λScale: Enabling fast scaling for serverless large language model inference. arXiv preprint arXiv:2502.09922, 2025

    work page arXiv 2025

  35. [35]

    Zhong, Y ., Liu, S., Chen, J., Hu, J., Zhu, Y ., Liu, X., Jin, X., and Zhang, H

    Sungmin Yun, Seonyong Park, Hwayong Nam, Youn- joo Lee, Gunjun Lee, Kwanhee Kyung, Sangpyo Kim, Nam Sung Kim, et al. The new llm bottleneck: A systems perspective on latent attention and mixture-of- experts.arXiv preprint arXiv:2507.15465, 2025

    work page arXiv 2025

  36. [36]

    Blitzscale: fast and live large model autoscaling with o(1) host caching

    Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, and Haibo Chen. Blitzscale: fast and live large model autoscaling with o(1) host caching. InProceedings of the 19th USENIX Confer- ence on Operating Systems Design and Implementation, OSDI ’25, USA, 2025. USENIX Association

    work page 2025

  37. [37]

    Dist- serve: disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: disaggregating prefill and decoding for goodput- optimized large language model serving. InProceed- ings of the 18th USENIX Conference on Operating Sys- tems Design and Implementation, OSDI’24, USA, 2024. USENIX Association

    work page 2024

  38. [38]

    Stuardo, Dongyang Wang, Xinlei Zhang, Huap- ing Zhou, et al

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Ce- sar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huap- ing Zhou, et al. Megascale-infer: Efficient mixture-of- experts model serving with disaggregated expert paral- lelism. InProceedings of the ACM SIGCOMM 2025 Conference, SIGCOMM ’25, pages 592–608, New York, NY , USA, 2025. Association for Computing Machinery. 14

    work page 2025