Pith. sign in

REVIEW 4 cited by

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.11665 v2 pith:XVAUXXEF submitted 2023-02-22 cs.LG cs.DCcs.NI

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Zhuohan Li , Lianmin Zheng , Yinmin Zhong , Vincent Liu , Ying Sheng , Xin Jin , Yanping Huang , Zhifeng Chen
show 3 more authors
Hao Zhang Joseph E. Gonzalez Ion Stoica
This is my paper
classification cs.LG cs.DCcs.NI
keywords modelparallelismservingsinglealpaservedeeplearningmultiplexing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving

    cs.DC 2026-05 unverdicted novelty 7.0

    ViBE co-optimizes expert placement with measured GPU performance variability in MoE inference to cut execution-time imbalance, delivering 14% better SLO attainment and up to 45% lower P90 TTFT.

  2. Efficient Memory Management for Large Language Model Serving with PagedAttention

    cs.LG 2023-09 conditional novelty 7.0

    PagedAttention achieves near-zero waste in LLM key-value cache memory and enables 2-4x higher serving throughput than prior systems.

  3. OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters

    cs.DC 2026-07 unverdicted novelty 6.0

    OmniPilot combines conformal quantile regression with OOD detection to rank LLM serving configurations on mixed GPUs, reporting 6.2% MAPE throughput prediction and 95% top-1 accuracy on 460 benchmark runs while abstai...

  4. JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models

    cs.DC 2026-07 conditional novelty 5.0

    A service-oriented multi-tenant architecture with schema-compatible group batching reduces aggregate GPU time for VLA post-training by about 28% in simulation.