REVIEW 11 cited by
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model. Its training cost saving is demonstrated from encoder-decoder models (prior works) to a 5x saving for auto-aggressive language models (this work along with parallel explorations). However, due to the much larger model size and unique architecture, how to provide fast MoE model inference remains challenging and unsolved, limiting its practical usage. To tackle this, we present DeepSpeed-MoE, an end-to-end MoE training and inference solution as part of the DeepSpeed library, including novel MoE architecture designs and model compression techniques that reduce MoE model size by up to 3.7x, and a highly optimized inference system that provides 7.3x better latency and cost compared to existing MoE inference solutions. DeepSpeed-MoE offers an unprecedented scale and efficiency to serve massive MoE models with up to 4.5x faster and 9x cheaper inference compared to quality-equivalent dense models. We hope our innovations and systems help open a promising path to new directions in the large model landscape, a shift from dense to sparse MoE models, where training and deploying higher-quality models with fewer resources becomes more widely possible.
Forward citations
Cited by 11 Pith papers
-
Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.
-
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.
-
MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
MindSpeed RL uses a distributed transfer dock and an allgather-swap method to raise RL training throughput on Ascend NPUs by 1.42 to 3.97 times over two open baselines.
-
Lilith: Developmental Modular LLMs with Chemical Signaling
A conceptual framework in which untrained modular LLMs are developed through simulated life and token-based chemical signaling, with the goal of enabling empirical study of consciousness emergence via Integrated Infor...
-
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation
A hierarchical routing method that combines predicted task labels with context-aware token routing improves BLEU and reduces activated experts in translation MoE models.
-
MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators
MixGCN combines feature-level and node-level parallelism with a dedicated sparse accelerator and node reordering to speed up full-graph GCN training.
-
Optimizing ML Concurrent Computation and Communication with GPU DMA Engines
Using DMA engines for collectives (ConCCL) lifts concurrent compute-communication speedup on MI300X from 21% to 72% of ideal.
-
MoE-Beyond: Learning-Based Expert Activation Prediction on Edge Devices
An abstract-only MoE paper claiming 97.5% activation prediction accuracy and a 17% to 72% cache hit rate gain, whose full text is a different paper on functional equations, making the results unverifiable.
-
ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing
ViFusion combines dynamic tensor fusion with hierarchical AllReduce to speed up distributed video feature indexing, but the 8-22x throughput claim is an overstatement of bandwidth gains over a self-defined baseline.
-
Taming the Titans: A Survey of Efficient LLM Inference Serving
A survey that classifies LLM inference serving techniques into instance, cluster, emerging-scenario, and miscellaneous categories, with a companion GitHub index and four research directions.
-
Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
A lightweight heterogeneous MoE with a GRU and an FFNN expert trails homogeneous baselines, and its claimed reasoning-type specialization is confounded by unequal expert inputs.
Discussion (0). Continue with ORCID to comment.