Pith. sign in

REVIEW 8 cited by

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.02263 v4 pith:HPCYHLSY submitted 2025-04-03 cs.DC cs.LG

Ruidong Zhu , Ziheng Jiang , Chao Jin , Peng Wu , Cesar A. Stuardo , Dongyang Wang , Xinlei Zhang , Huaping Zhou
show 12 more authors
Haoran Wei Yang Cheng Jianzhe Xiao Xinyi Zhang Lingjun Liu Haibin Lin Li-Wen Chang Jianxi Ye Xiao Yu Xuanzhe Liu Xin Jin Xin Liu
This is my paper · ORCID
classification cs.DCcs.LG
keywords megascale-inferparallelismattentionmodulesoverheadcommunicationdatadisaggregated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely activated architecture shifts feed-forward networks (FFNs) from being compute-intensive to memory-intensive during inference, leading to substantially lower GPU utilization and increased operational costs. We present MegaScale-Infer, an efficient and cost-effective system for serving large-scale MoE models. MegaScale-Infer disaggregates attention and FFN modules within each model layer, enabling independent scaling, tailored parallelism strategies, and heterogeneous deployment for both modules. To fully exploit disaggregation in the presence of MoE's sparsity, MegaScale-Infer introduces ping-pong pipeline parallelism, which partitions a request batch into micro-batches and shuttles them between attention and FFNs for inference. Combined with distinct model parallelism for each module, MegaScale-Infer effectively hides communication overhead and maximizes GPU utilization. To adapt to disaggregated attention and FFN modules and minimize data transmission overhead (e.g., token dispatch), MegaScale-Infer provides a high-performance M2N communication library that eliminates unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization. Experimental results indicate that MegaScale-Infer achieves up to 1.90x higher per-GPU throughput than state-of-the-art solutions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

    cs.DC 2026-08 conditional novelty 7.0 of 10

    Operator-level autoscaling meets LLM latency targets with 20-36% fewer GPUs and 14-28% less power than model-level replica scaling.

  2. PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PagedWeight pages quantized MoE expert weights on and off the GPU at runtime, releasing memory to the KV cache while using sensitivity, routing, and prompt signals to choose which experts to shrink.

  3. UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

    cs.DC 2026-07 conditional novelty 6.0 of 10

    UBEP replaces BSP All-to-All for MoE on multi-tier superpods with dependency-driven kernel decomposition, topology-aware token scheduling, and Data-as-Flag atomics, cutting All-to-All latency up to 52.4% and TPOT up to 11.1%.

  4. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

  5. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

  6. SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

    cs.DC 2025-06 conditional novelty 6.0 of 10

    SiPipe claims 1.6x to 2.1x throughput gains over vLLM for pipeline-parallel LLM inference by using idle CPUs for sampling, double-buffered CUDA graphs, and structure-aware transmission.

  7. Serving Large Language Models on Huawei CloudMatrix384

    cs.DC 2025-06 conditional novelty 6.0 of 10

    CloudMatrix384 with CloudMatrix-Infer reports prefill 6,688 tokens/s per NPU and decode 1,943 tokens/s per NPU on DeepSeek-R1, exceeding published NVIDIA-based efficiencies.

  8. Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking

    cs.PF 2025-08 conditional novelty 5.0 of 10

    FMwork shows that sampling power-of-two batch sizes and capping output length at 128 tokens reproduces a full Llama 3.1 8B benchmark sweep within about 3-5% error at up to 24x lower experimental cost.

Pith tools