X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms

Ahan Gupta; Feiyi Wang; Jianping Li; Minjia Zhang; Sajal Dash; Yueming Yuan

REVIEW 3 major objections 5 minor 40 references

X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read X-MoE trains 545B-parameter MoEs on 1024 AMD GPUs, a 10x jump over prior systems on the same hardware.

desk verdict A genuinely useful MoE systems paper with a convincing memory-bottleneck story; treat the 545B 'trainable' claim as a memory-feasibility result until a real training trace is shown. read the letter →

arxiv 2508.13337 v1 pith:ZMPKUMY4 submitted 2025-08-18 cs.LG cs.CLcs.DC

Yueming Yuan , Ahan Gupta , Jianping Li , Sajal Dash , Feiyi Wang , Minjia Zhang This is my paper · ORCID

classification cs.LGcs.CLcs.DC

keywords mixture-of-expertstrainingfine-grainedexpertslargetop-kroutingpadding-freetokenbuffershierarchicalall-to-allsequenceshardingAMDROCmHPC

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

X-MoE is a training system for the new generation of mixture-of-experts language models in which many small experts are activated per token (DeepSeek-style, fine-grained experts with large top-$k$ routing). The paper argues that such models hit a memory bottleneck in the dispatch and combine stages, not in expert weights, and that existing MoE frameworks tuned for a few large experts on NVIDIA hardware waste memory and bandwidth on zero-padding and duplicated token copies. X-MoE removes zero-padding with a sparse token buffer, cuts repeated inter-node traffic by routing only one pilot copy of each token across the network, and shards the sequence inside MoE blocks. On the Frontier supercomputer, the paper reports that X-MoE trains a 545-billion-parameter DeepSeek-style model on 1024 AMD MI250X GPUs, ten times larger than the largest model prior systems could fit under the same hardware budget, at an aggregated 10.44 petaFLOP/s. The implication, if correct, is that the next generation of expert-specialized MoEs can be trained on non-NVIDIA HPC clusters instead of being limited to CUDA-only infrastructure.

What carries the argument

Three mechanisms carry the argument. The first is a Padding-Free Token buffer (PFT): a sparse layout that stores only the tokens actually routed to each expert, plus small arrays recording token ids, expert ids, per-expert token counts, and combine weights, so no zero-padded capacity slots travel through dispatch, expert MLPs, or combine. The second is redundancy-bypassing dispatch (RBD), a hierarchical two-stage all-to-all that sends one pilot token across a node boundary and reconstructs local replica tokens inside the destination node using fast intra-node bandwidth. The third is sequence-sharded MoE blocks (SSMB): because gating, dispatch, expert FFNs, and combine are token-wise operations, the input sequence is split across expert-parallel ranks inside the MoE block and reassembled with an all-gather before the next tensor-parallel block, cutting the memory of dispatch/combine activations by the tensor-parallel group size. A portable, Triton-based kernel set for gather, scatter, and sequential GEMM lets these sparse stages run on AMD ROCm without CUDA-specific code.

What would settle it

Run the reported 545.4B configuration to a fixed iteration budget with a logged loss curve and compare its slope to the 201B model's; if the run fits in memory but the loss diverges or the run cannot complete iterations, the trainability claim collapses. Counting the number of completed optimizer steps and sustained throughput over those steps would also settle whether the 1024-GPU result is training or a single-step benchmark.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two structural changes in emerging MoEs—many more, smaller experts and much larger $k$—move the training bottleneck from parameter memory to activation memory, especially the dispatched-input and combined-output tensors, and that a system built around that shifted bottleneck can make such models trainable where prior systems run out of memory. The evidence is a set of Frontier experiments: X-MoE trains 10.1B, 55.2B, and 201B parameter models on 256 GPUs while the compared baselines run out of memory on all but the smallest, and it reaches 5.15x and 1.42x throughput over the two surviving baselines on the 55.2B model. On 1024 GPUs, X-MoE fits and runs a 545.4B parameter model at 10.44 petaFLOP/s aggregate throughput. The paper also validates correctness by showing that on the 10.1B model X-MoE's loss curve tracks the baseline's over 500 iterations, with a small gap attributed to a more permissive token-dropping rule. The scale claim for the 545B model rests on fitting and throughput rather than a reported loss curve for that run.

Load-bearing premise

The load-bearing premise is that the 545B run is genuine sustained training, not merely a configuration whose parameter buffers fit in memory; the paper validates convergence only at the 10.1B scale and does not show a loss curve or completed iteration count for the 545B run.

Editorial extensions

If this is right

The trainable boundary for DeepSeek-style MoEs moves to over half a trillion parameters on 1024 GPUs, where prior systems cannot fit even the 55.2B model.
Padding-free routing cuts per-layer activation memory substantially, so sequence length, top-$k$, and expert count can be pushed further before memory limits bind.
On the 55.2B model, X-MoE is faster than the strongest baseline that can fit the model, and its advantage grows with larger top-$k$ values.
Because the kernels use a portable backend rather than CUDA, the same system runs on NVIDIA A100s as well, fitting configurations that CUDA-only baselines OOM on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the claimed bottleneck shift is general, the pilot-token dispatch trick should also reduce communication in MoE inference on hierarchical networks, since the redundancy it removes is topology-driven rather than training-specific; the paper does not evaluate inference.
A direct test of RBD's mechanism would compare dispatch time on a cluster with balanced high-bandwidth inter-node links: there the redundancy-bypassing scheme should give near-zero benefit, while on an asymmetric topology it should reproduce the reported dispatch speedup.
The absence of a 545B loss curve leaves open whether that run is sustained training or a memory-fit benchmark; reporting the 545B loss trajectory and completed iteration count would settle the distinction.
The paper's placement analysis suggests a testable prediction: on nodes with even higher intra-node bandwidth than Frontier's, a DP-first placement strategy should widen the gap over the EP-first strategy used by existing systems.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is a real systems paper, not a packaging exercise. X-MoE builds a padding-free MoE pipeline in Triton, adds redundancy-bypassing dispatch and sequence-sharded MoE blocks, and targets exactly the fine-grained high-top-k MoE regime where activations—not parameters—are the bottleneck. That diagnosis in Section 3.2 is the paper's sharpest contribution, and the layer-level breakdowns back it up: gating/dispatch/combine time drops by 5.7x/35.7x/8.1x on the small model, and alltoall time halves on the large model. The A100 portability experiments are a nice bonus, and the code is on GitHub.

The soft spot is the headline. "Enables training up to 545B" currently rests on Figure 9: one aggregated throughput bar (10.44 PFLOPS), OOM markers for baselines, and no loss curve, iteration count, or step-time distribution for the 545B run. Correctness validation is only on the 10.1B model, 500 iterations, against DeepSpeed-MoE, and the authors themselves note a token-dropping rule difference. Appendix D shows >500ms all-to-all outliers at 512/1024 GPUs, so the 10.44 PFLOPS number could be a favorable-window measurement. Also, the "10x larger than the largest trainable model" claim is not tied to a measured 1024-GPU baseline ceiling. These are load-bearing gaps for the scale claim, but they don't invalidate the layer-level engineering, which is measured more carefully.

One fairness note: DeepSpeed-MoE is a baseline and includes a co-author of this paper. That's not disqualifying, but baseline configurations are not fully reported, so I'd ask for the config details and for either a 545B training trace or a reworded abstract. The comparison looks plausible on the smaller models, where both systems run, but the OOM-only comparisons at scale don't tell us how close the baselines were to their own limits.

Bottom line: this deserves a serious referee. The systems contributions are solid, the writeup is honest about the noisy cluster, and the code availability is a plus. I'd send it to peer review with a request for a sustained 545B trace or a softened claim, plus error bars on the Frontier throughput. This is the kind of work that moves MoE training on AMD hardware forward, and the right reviewer will be able to push on the weak point without throwing out the engineering.

Referee Report

3 major / 5 minor

Summary. The paper proposes X-MoE, a training system for expert-specialized Mixture-of-Experts (MoE) models targeting non-NVIDIA HPC platforms, with evaluation on the AMD MI250X-based Frontier supercomputer. The system combines three main techniques: Padding-Free Token buffers (PFT) with Triton-based sparse kernels that eliminate zero-padding in MoE dispatch/combine, a hierarchical Redundancy-Bypassing Dispatch (RBD) that reduces inter-node all-to-all traffic, and Sequence-Sharded MoE Blocks (SSMB) that reduce activation memory under hybrid parallelism. The evaluation reports that X-MoE can train DeepSeek-style MoEs up to 545B parameters on 1024 GPUs with 10.44 PetaFLOPs aggregated throughput, that it outperforms DeepSpeed-MoE, DeepSpeed-TED, and Tutel in several Frontier configurations, and that it shows mixed but functional portability on 8x A100 GPUs. The source code is made available.

Significance. If the central claims are fully substantiated, this is a strong systems contribution: it directly addresses the activation-memory bottleneck of fine-grained, high-top-k MoEs, provides a portable (Triton-based) implementation on a major non-NVIDIA HPC platform, and releases code that should aid reproducibility. The paper's memory-bottleneck analysis in Section 3.2 and the SSMB-vs-TED trade-off analysis in Appendix C are valuable in themselves. The empirical gains over three established baselines on Frontier are meaningful, and the design is clearly described. The main reservation is that the headline '545B trainable model' claim rests on a throughput bar and memory feasibility rather than on evidence of a sustained, numerically valid training run; this needs to be addressed before the strongest claims can be accepted.

major comments (3)

[§5.2, Fig. 9, Appendix D] The headline claim that X-MoE 'enables training' of the 545B Super model on 1024 GPUs is not supported by the evidence shown. Figure 9 reports a single aggregated throughput number (10.44 PFLOPS) and OOM markers for the baselines, but no loss curve, gradient-norm trace, iteration count, or step-time distribution is provided for this configuration. Appendix D is an in-scope limitation that directly matters here: it reports that all-to-all collectives at 512 and 1024 GPUs frequently exceed 500 ms and that beyond 256 GPUs latencies are on average over 10x higher due to cross-rack congestion. A throughput measurement over an unspecified window cannot demonstrate sustained, numerically stable training. Please add a training trace (e.g., loss over at least several hundred iterations, with step-time statistics) for the Super configuration, or explicitly narrow the claim to memory feasibility plus a one-step throughput benchmark.
[§5.2] The '10x larger than the largest trainable model' factor is not tied to a measured baseline ceiling. The paper reports OOM for the baselines at particular configurations in Fig. 9, but it does not sweep baseline configurations on 1024 GPUs to find the largest model each baseline can actually train under the same hardware and memory settings. Baseline memory and throughput depend strongly on EP size and ZeRO stage (which the paper itself sweeps in §5.2), so the 10x factor is not established by the data as presented. Please report, for each baseline, the largest trainable model size at 1024 GPUs under matched parallelism and memory settings, or explicitly state the configuration used to derive the 10x ratio.
[§5.6] The numerical validation covers only the 10.1B model for 500 iterations on 16 GPUs, and the paper itself identifies a difference in token-dropping semantics between X-MoE and DeepSpeed-MoE: X-MoE drops only on capacity, whereas DeepSpeed-MoE also drops tokens with negative routing scores. This is not sufficient to validate the larger configurations, where SSMB, RBD, and higher TP degrees change communication and sharding behavior. In particular, no evidence is shown that the 201B or 545B configurations would produce a stable loss trajectory rather than diverging due to routing or gradient issues at scale. Please provide a correctness check at a multi-node expert-specialized scale (e.g., a loss comparison against a reference implementation, or a gradient-norm check), and either align the token-dropping rules or report explicitly how their difference affects the comparison.

minor comments (5)

[§5.2] The sentence 'existing systems such as DeepSpeed-MoE, DeepSpeed-TED, and Tutel run out of memory on medium and large models' is contradicted by Fig. 9, where Tutel trains the Medium model at 17.0 TFLOPS and DeepSpeed-MoE attains 4.7 TFLOPS; please revise the wording to describe which systems actually OOM on which model.
[§5.4.1, Fig. 11] The x-axis labels of Fig. 11 call the 55.2B model 'Large', but Table 3 designates it as 'Medium'; please make the naming consistent throughout.
[§5.5, Table 5] The abbreviations 'Small-SR' and 'Small-LR' are defined in the body text but not in the table caption; please add the definitions to the caption, and state explicitly whether the same EP size, TP degree, and ZeRO stage were used for all three frameworks on the A100 experiments, since these settings directly affect the throughput comparison.
[§4.1.1, Listing 1] The relationship between the 'max_token_count' parameter in Listing 1 and the capacity factor c=1.25 mentioned in §5.1 is not specified; please state the formula (e.g., max_token_count = floor(c * S * K / E)) so that the token-dropping behavior is reproducible.
[§5.2, Appendix D] The measurement methodology for the 10.44 PFLOPS figure is not given: please state over how many steps the throughput is averaged and whether all-to-all outlier events are included or excluded, since Appendix D shows that outliers are frequent at 1024 GPUs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: X-MoE's claims are supported by measured comparisons against external baselines, not by definitional or self-citation-dependent reasoning.

full rationale

X-MoE is an empirical systems paper. Its central results—memory footprint, throughput, 545B trainability, and scaling behavior—are obtained by measurement on Frontier against external baselines (DeepSpeed-MoE, DeepSpeed-TED, Tutel) and by arithmetic conversion of per-GPU throughput (e.g., 1024 GPUs × ~10.2 TFLOPS ≈ 10.44 PFLOPS). No load-bearing equation is defined in terms of the quantity it is said to predict. The PFT pipeline, RBD, and SSMB are described as constructive designs and evaluated by ablation (§5.4), not derived from the conclusions. The memory-savings analysis in Appendix C derives a threshold r > 2/(c·S) from model hyperparameters; this is a standalone calculation, not a restatement of X-MoE's measured result. The only self-reference is that one author is also a DeepSpeed author and X-MoE is integrated into DeepSpeed [31,32], and validation in §5.6 uses DeepSpeed-MoE as a reference implementation; however, the comparison is external, loss curves are directly compared, and the token-dropping difference is explicitly disclosed rather than assumed away. No uniqueness theorem or prior-work citation is invoked to forbid alternatives. The gap that concerns the reviewer—the absence of a sustained loss/iteration trace for the 545B run, with Appendix D disclosing network outliers—is an evidence/validation issue, not circularity, because the claim is not true by construction and the paper supplies independent benchmark comparisons. Under the scoring rubric, no step reduces to its inputs, so score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 3 invented entities

The central result depends on three new system components (PFT, RBD, SSMB), each of which is validated only by the paper's own experiments. The only free parameters are standard training knobs (capacity factor, batch size, parallelism degrees), and the significant axioms are standard MoE semantics, a token-dropping assumption, and the treatment of Frontier network outliers as environmental noise.

free parameters (3)

expert capacity factor c = 1.25
The GShard capacity factor c=1.25 is a user-specified hyperparameter that directly determines token dropping and effective activation memory. The paper does not sweep or justify this value beyond citing GShard.
micro-batch size = maximum power of 2 within memory
The micro-batch size is chosen per configuration under memory constraints, which is a standard practice but a free knob that affects the throughput numbers.
EP size and TP degree = swept per experiment (e.g., EP 32/64/128/256, TP 1/2/4)
Parallelism configurations are chosen by sweeping for baselines and by memory for X-MoE, which influences the comparison and is a chosen parameter rather than a derived one.

assumptions (3)

domain assumption Standard MoE gating and routing semantics (top-k, combine_weights) are assumed correct and unchanged.
The paper relies on the standard MoE layer definition from DeepSeek-MoE and GShard (Section 2) without re-deriving it.
domain assumption The token-dropping strategy (drop only on capacity, not on negative routing scores) is assumed to preserve training quality.
Section 5.6 reveals a difference from DeepSpeed-MoE's token dropping logic and claims lower loss, but this is only validated for 500 iterations on the 10.1B model.
domain assumption All-to-all latency outliers beyond 256 GPUs are an environmental property of Frontier and are treated as noise.
Appendix D attributes latency spikes to network congestion from concurrent jobs, but this is a hypothesis, not a measured control.

invented entities (3)

PFT (Padding-Free Token buffers)
purpose: Sparse token storage with ERI-arrays to avoid zero-padding in dispatch, MLP, and combine.
This is a data structure introduced by the paper. Its benefit is measured in the paper's own experiments, but there is no external verification yet.
RBD (Redundancy-Bypassing Dispatch)
purpose: Hierarchical two-stage dispatch to avoid sending duplicate tokens over inter-node links.
Algorithmic component introduced by the paper, measured only in the paper's own evaluation.
SSMB (Sequence-Sharded MoE Blocks)
purpose: Sequence sharding inside MoE blocks to cut activation memory from dispatch and combine tensors.
Parallelism scheme introduced by the paper, measured only in the paper's own evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms." pith.science (2026). https://pith.science/paper/ZMPKUMY4

@misc{pith2026250813337,
  author       = {Pith},
  title        = {Pith review of: X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMPKUMY4}},
  note         = {Machine review of arXiv:2508.13337}
}

read the original abstract

Emerging expert-specialized Mixture-of-Experts (MoE) architectures, such as DeepSeek-MoE, deliver strong model quality through fine-grained expert segmentation and large top-k routing. However, their scalability is limited by substantial activation memory overhead and costly all-to-all communication. Furthermore, current MoE training systems - primarily optimized for NVIDIA GPUs - perform suboptimally on non-NVIDIA platforms, leaving significant computational potential untapped. In this work, we present X-MoE, a novel MoE training system designed to deliver scalable training performance for next-generation MoE architectures. X-MoE achieves this via several novel techniques, including efficient padding-free MoE training with cross-platform kernels, redundancy-bypassing dispatch, and hybrid parallelism with sequence-sharded MoE blocks. Our evaluation on the Frontier supercomputer, powered by AMD MI250X GPUs, shows that X-MoE scales DeepSeek-style MoEs up to 545 billion parameters across 1024 GPUs - 10x larger than the largest trainable model with existing methods under the same hardware budget, while maintaining high training throughput. The source code of X-MoE is available at https://github.com/Supercomputing-System-AI-Lab/X-MoE.

Figures

Figures reproduced from arXiv: 2508.13337 by the authors.

**Figure 2.** Conventional gating and dispatching logic with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Redundancy rate of all dispatched tokens. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Figures from the paper (12 more)

**Figure 5.** Figure 5: Overview of X-MoE. At a high level, X-MoE enables efficient and scalable training for expert-specialized MoEs through [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: PFT with sparse structure and ERI-arrays. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Hierarchical Redundancy-Bypassing Dispatch: [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Illustration of X-MoE’s hybrid parallelism with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 10.** Figure 10: Scalability results. (a) Weak scaling results: Train [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

**Figure 9.** Figure 9: Results on training Small, Medium, and Large mod [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 12.** Figure 12: Dispatching time breakdown: With and without [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

**Figure 15.** Figure 15: Loss validation with DeepSpeed-MoE and X-MoE. [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]

**Figure 16.** Figure 16: DeepSpeed-MoE training pipeline. Gating. During the gating stage, a dispatching mask, dispatch_mask of size [S, E, C] is constructed. The entry dispatch_mask[t, e, c] is either 1 or 0 indicating if the t th token is routed to the c th position in expert e’s buffer. A …

**Figure 17.** Figure 17: The memory saving advantage regions of SSMB [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]

**Figure 18.** Figure 18: The all-to-all collective time characterization [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]

**Figure 20.** Figure 20: Scaling efficiency of X-MoE on 256 GPUs with varying model configurations. (Left) Training throughput when [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 27 canonical work pages

[1]

Meta AI. 2024. Introducing Meta LLaMA-3. https://ai.meta.com/blog/meta-llama- 3/

work page 2024
[2]

Meta AI. 2025. Llama 4: Multimodal Intelligence. https://ai.meta.com/blog/llama- 4-multimodal-intelligence/

work page 2025
[3]

Anthropic. 2024. Claude 3 haiku: our fastest model yet. https://www.anthropic. com/news/claude-3-haiku

work page 2024
[4]

Argonne National Laboratory. 2024. Aurora Supercomputer. https://www.alcf. anl.gov/aurora

work page 2024
[5]

Scott Atchley, Christopher Zimmer, John Lange, and et al. 2023. Frontier: Explor- ing Exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023

work page 2023
[6]

Tom Brown, Benjamin Mann, Nick Ryder, and et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS ’20)

work page 2020
[7]

Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. 2024. Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communi- cation Partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating...

work page 2024
[8]

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174 [cs.LG]

arXiv 2016

Show all 40 references

[9]

Weihao Cui, Zhenhua Han, Lingji Ouyang, Yichuan Wang, Ningxin Zheng, Lingxiao Ma, Yuqing Yang, Fan Yang, Jilong Xue, Lili Qiu, Lidong Zhou, Quan Chen, Haisheng Tan, and Minyi Guo. 2023. Optimizing Dynamic Neural Net- works with Brainstorm. In USENIX Symposium on Operating Syst...

2023
[10]

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture...

2024
[11]

DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL]

2025 arXiv
[12]

Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Helen Li, and Yiran Chen. 2024. SiDA: Sparsity-Inspired Data- Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. In Proceedings of Machine Learning and Systems...

2024
[13]

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research (JMLR ’22) 23, 1 (2022), 5232–5270

2022
[14]

Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems (MLSys ’23) 5 (2023)

2023
[15]

Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models. InProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programm...

2022
[16]

Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive mixture- of-experts at scale. Proceedings of Machine Learning and Systems (MLSys ’23) (2023)

2023
[17]

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, et al. 2024. Mixtral of experts. arXiv:2401.04088 (2024)

2024 arXiv
[18]

Chenyu Jiang, Ye Tian, Zhen Jia, Shuai Zheng, Chuan Wu, and Yida Wang. 2024. Lancet: Accelerating mixture-of-experts training via whole graph computation- communication overlapping. (2024)

2024
[19]

Rashika Kheria, Wenkai Du, Yongseok Koh, Raghu Raja, gilbertlee amd, James Dinan, Denis Maryin, dmitrygx, Ryan Hankins, Stanley Tsang, David Addison, AWSNB, Eric Badger, Gina Sitaraman, Nicholas Sly, Shi Jin, Sylvain Jeaugey, Theofilos Manitaras, and YoonGi Kim. 2024. ROCm/aws...

2024
[20]

Heehoon Kim, Junyeol Ryu, and Jaejin Lee. 2024. TCCL: Discovering Better Communication Paths for PCIe GPU Clusters (ASPLOS ’24)

2024
[21]

Dally, Steve Scott, and Dennis Abts

John Kim, William J. Dally, Steve Scott, and Dennis Abts. 2008. Technology- Driven, Highly-Scalable Dragonfly Topology. In 35th International Symposium on Computer Architecture (ISCA 2008) . IEEE Computer Society, 77–88

2008
[22]

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princip...

2023
[24]

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. CoRR abs/2006.16668 (2020). arXiv:2006.16668

2020 arXiv
[25]

Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al . 2024. Deepseek- v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434 (2024)

2024 arXiv
[26]

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clus...

2021
[27]

NVIDIA Corporation. 2023. NVIDIA TensorRT: Programmable Inference Accelera- tor. https://developer.nvidia.com/tensorrt

2023
[28]

OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023)

2023 arXiv
[29]

OpenAI. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https://arxiv.org/ abs/2410.21276

2024 arXiv
[30]

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations

2019
[31]

Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed- MoE: Advancing Mixture-of-Experts Inference and Training to Power Next- Generation AI Scale. In International Conference on Machin...

2022
[32]

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- Speed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(KDD 20) . 3505–3506

2020
[33]

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)

2017 arXiv
[34]

Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. 2023. A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing (SC ’2...

2023
[35]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

2023 arXiv
[36]

Philippe Tillet, Hsiang-Tsung Kung, and David D. Cox. 2019. Triton: an in- termediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (SIGPLAN 2019)

2019
[37]

XAI. 2025. Grok. https://x.ai/grok

2025
[38]

Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze
[39]

Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. 2025. DeepEP: an efficient expert-parallel communication library. https://github.com/deepseek-ai/DeepEP

2025
[40]

Gonzalez, Clark W

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2023. Efficiently Programming Large Language Models using SGLang. CoRR abs/2312.07104 (2023). X-Mo...

2023 arXiv
[2025]

arXiv preprint arXiv:2501.01005 (2025)

FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv preprint arXiv:2501.01005 (2025)

2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Meta AI. 2024. Introducing Meta LLaMA-3. https://ai.meta.com/blog/meta-llama- 3/

work page 2024

[2] [2]

Meta AI. 2025. Llama 4: Multimodal Intelligence. https://ai.meta.com/blog/llama- 4-multimodal-intelligence/

work page 2025

[3] [3]

Anthropic. 2024. Claude 3 haiku: our fastest model yet. https://www.anthropic. com/news/claude-3-haiku

work page 2024

[4] [4]

Argonne National Laboratory. 2024. Aurora Supercomputer. https://www.alcf. anl.gov/aurora

work page 2024

[5] [5]

Scott Atchley, Christopher Zimmer, John Lange, and et al. 2023. Frontier: Explor- ing Exascale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023

work page 2023

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, and et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS ’20)

work page 2020

[7] [7]

Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. 2024. Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communi- cation Partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating...

work page 2024

[8] [8]

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174 [cs.LG]

arXiv 2016