Pith. sign in

REVIEW 18 cited by

FastMoE: A Fast Mixture-of-Expert Training System

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.13262 v1 pith:HCIUEKOC submitted 2021-03-24 cs.LG cs.CLcs.DC

Jiaao He , Jiezhong Qiu , Aohan Zeng , Zhilin Yang , Jidong Zhai , Jie Tang This is my paper · ORCID
classification cs.LGcs.CLcs.DC
keywords systemfastmoetrainingdifferentpytorchavailabledistributedenlarging
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture-of-Expert (MoE) presents a strong potential in enlarging the size of language model to trillions of parameters. However, training trillion-scale MoE requires algorithm and system co-design for a well-tuned high performance distributed training system. Unfortunately, the only existing platform that meets the requirements strongly depends on Google's hardware (TPU) and software (Mesh Tensorflow) stack, and is not open and available to the public, especially GPU and PyTorch communities. In this paper, we present FastMoE, a distributed MoE training system based on PyTorch with common accelerators. The system provides a hierarchical interface for both flexible model design and easy adaption to different applications, such as Transformer-XL and Megatron-LM. Different from direct implementation of MoE models using PyTorch, the training speed is highly optimized in FastMoE by sophisticated high-performance acceleration skills. The system supports placing different experts on multiple GPUs across multiple nodes, enabling enlarging the number of experts linearly against the number of GPUs. The source of FastMoE is available at https://github.com/laekov/fastmoe under Apache-2 license.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure

    cs.CV 2026-07 unverdicted novelty 7.0 of 10

    BrainFIBRE pretrains a five-expert Mixture-of-Experts model on NODDI-derived microstructural maps and outperforms prior deep models on age, sex, cerebrovascular, neurodegenerative, and cognitive prediction.

  2. FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

    cs.LG 2025-01 conditional novelty 7.0 of 10

    FSMoE co-schedules intra-node and inter-node communication with expert computation and adaptively partitions gradients, achieving 1.18x-1.22x speedups over Tutel and up to 3.01x over DeepSpeed-MoE in MoE training.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

    cs.CV 2026-08 accept novelty 6.0 of 10

    The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.

  5. On the Design of Mixture-of-Experts for Dynamic Gaussian Splatting

    cs.CV 2026-07 accept novelty 6.0 of 10

    Two MoE integration strategies (joint canonical MoDE vs. independent-then-route MoE-GS) improve dynamic Gaussian Splatting by composing complementary deformation priors.

  6. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  7. Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing

    cs.DC 2025-01 reject novelty 6.0 of 10

    A Bayesian optimization framework with epsilon-greedy search selects expert memory, replicas, and scatter-gather communication mode to reduce serverless MoE inference cost.

  8. Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-execution

    cs.DC 2024-11 conditional novelty 6.0 of 10

    DHelix interleaves two micro-batches at operator granularity and folds the pipeline into a W-shape, hiding communication and improving LLM training throughput by 12-40% on tested clusters.

  9. Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation

    cs.DC 2024-11 conditional novelty 6.0 of 10

    Luffy speeds up distributed Mixture-of-Experts training by migrating sequences toward the GPUs that hold their tokens and condensing similar tokens before dispatch.

  10. HarMoEny: Efficient Multi-GPU Inference of MoE Models

    cs.DC 2025-06 conditional novelty 5.0 of 10

    HarMoEny uses dynamic token redistribution and asynchronous expert prefetching to achieve near-perfect GPU load balance in multi-GPU MoE inference.

  11. Two Is Better Than One: Rotations Scale LoRAs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    RadarGate rotates LoRA outputs before mixing them, expanding the set of reachable outputs beyond plain weighted sums.

  12. MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing

    cs.LG 2025-02 conditional novelty 5.0 of 10

    MoETuner formulates MoE expert placement as two ILPs to balance token loads and minimize inter-GPU communication, reporting 9.3% and 17.5% end-to-end inference speedups on Mixtral-8x7B.

  13. Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

    cs.DC 2024-11 conditional novelty 5.0 of 10

    Pro-Prophet balances MoE training load by moving each expert to only the devices that need it and by pre-scheduling those moves to hide communication behind computation.

  14. UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learning

    cs.LG 2025-08 conditional novelty 4.0 of 10

    A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.

  15. Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edge

    cs.CV 2025-07 reject novelty 4.0 of 10

    EMC2 routes each driving scene to one of three expert 3D detectors using distance and confidence thresholds, reporting higher accuracy and 2.6x faster inference on Jetson, though thresholds are tuned on the validation set.

  16. MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing

    cs.LG 2025-06 conditional novelty 4.0 of 10

    MoE-GPS shows that predicting only the aggregate token-to-expert distribution, instead of exact token routing, can improve MoE inference latency by more than 23% in simulation, with guidelines for when each strategy wins.

  17. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

  18. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

Pith tools