Pith. sign in

REVIEW 3 major objections 5 minor 42 references

CAP places and prunes MoE experts by communication cost, cutting distributed inference latency 1.23–1.86× while keeping accuracy higher at the same speedup.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 08:34 UTC pith:KQJE4VGG

load-bearing objection Solid systems paper that makes communication volume a first-class objective for large-expert MoE inference and backs it with clear hardware-regime results. the 3 major comments →

arxiv 2607.05116 v1 pith:KQJE4VGG submitted 2026-07-06 cs.DC

Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

Xiao Shi , Yingying Sun , Jiangsu Du , Zhiguang Chen , Yutong Lu This is my paper
classification cs.DC
keywords Mixture-of-Expertsexpert parallelismexpert placementexpert pruningcommunication-aware schedulingdistributed LLM inferenceload balance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mixture-of-experts models now pack dozens or hundreds of experts per layer, so where those experts sit and which ones a token actually visits decide how much data must cross GPUs and nodes. CAP treats that communication volume as a first-class design variable alongside compute balance and accuracy. It first groups experts that tend to fire together onto the same devices and nodes, then walks a spectrum of placements that trade a little more communication for better load balance, and finally prunes whole devices rather than individual experts when the accuracy budget allows. A short offline profile picks the right point on that spectrum for each machine. The result is higher throughput on both bandwidth-poor and bandwidth-rich hardware, and better accuracy than ordinary pruning when the same latency target must be met.

Core claim

In modern large-expert-count MoE models the volume of all-to-all traffic is governed by the number of distinct devices a token must visit, not merely by how many experts it activates. CAP shows that co-activation-aware placement, a tunable communication–load spectrum, and device-level communication-aware pruning together let a single offline pipeline select an operating point that simultaneously reduces communication, balances work, and preserves accuracy better than load-balance-only or sequential baselines.

What carries the argument

The three-stage CAP pipeline: (1) co-activation graph partitioning that seeds high-degree experts and greedily grows balanced device and node groups, (2) iterative expert swaps that generate a Pareto spectrum of placements trading C(π) against load imbalance B(π), and (3) device-level pruning that retains destinations ordered by accuracy-contribution over communication-cost until a user threshold 1−p is met.

Load-bearing premise

A single offline co-activation snapshot taken from a few thousand tokens stays stable enough, both inside one workload and across different domains, that the same expert grouping can be reused without frequent re-profiling.

What would settle it

Measure device-per-token and max/avg load on a held-out traffic mix (different domain or much longer session) after applying a grouping built on only 8 000 tokens; if either metric drifts enough that the profiled spectrum no longer contains the true latency-optimal placement, the stability claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents CAP, a three-component framework for distributed MoE inference that jointly optimizes expert placement and dynamic pruning for communication volume, load balance, and accuracy. Co-activation-driven placement (Algorithm 1, Eqs. 2–3) groups frequently co-activated experts to reduce device/node visits per token; a subsequent adjustment procedure generates a spectrum of placements trading communication against load imbalance (Eqs. 4–5); and device-level communication-aware pruning (Eq. 6) removes low-contribution routing destinations. A four-step pipeline (Section VII) first collects pruned routing traces, then places, adjusts, and profiles. On Qwen3-30B-A3B and DeepSeek-V2-Lite across three single-node regimes (comm.-constrained 3090s, balanced A100s, comm.-rich H100s) and two multi-node clusters, CAP reports 1.23×–1.86× throughput gains over sequential (vLLM-style) placement and DeepSeek EPLB while preserving higher accuracy at matched speedup under pruning.

Significance. If the reported gains hold under broader deployment, the work is a useful systems contribution for modern large-expert-count MoEs, where communication—not only load balance—has become first-order. The explicit communication–load spectrum and hardware-aware selection are more complete than pure load-balance placers, and the device-level pruning formulation correctly targets routing destinations rather than individual experts. Strengths include clear algorithmic statements (Alg. 1, Eqs. 2–6), multi-regime evaluation that isolates the spectrum (Figs. 12–13), pruning accuracy–speedup comparison (Fig. 14 + Table I), and direct stability/transferability measurements (Fig. 15, Tables III–IV). The offline overhead claim (≤6 min) is practical. These elements make the central throughput and accuracy claims falsifiable and useful to practitioners.

major comments (3)
  1. Section VIII-B / Fig. 11: the headline 1.23×–1.86× throughput numbers are reported without variance, confidence intervals, or repeated runs under the same RPS/SLO regime. Because the claim is the paper’s primary result and the gains vary sharply by hardware class, at least summary statistics (or error bars) over multiple request-stream seeds are needed to establish that the ranking over Default and EPLB is stable.
  2. Section VIII-F and Tables III–IV: the weakest load-bearing assumption is that a single offline co-activation grouping remains adequate. The paper supplies supportive evidence (in-workload variation <2.9 % communication / <5.8 % imbalance; cross-domain Device-per-Token still better than Default), yet the evaluation uses only three source windows and three datasets. A short sensitivity experiment that forces re-profiling after a domain shift (or quantifies degradation when the grouping is stale) would strengthen the claim that frequent regrouping is unnecessary.
  3. Section V / Eq. (5): λ is introduced as the continuous trade-off weight, but the implemented Adjustment Approach generates a discrete spectrum by iterative swaps and then selects by profiling. The manuscript never states how many swaps, which experts are eligible, or how the spectrum is guaranteed to approximate the Pareto front of Eq. (5). Clarifying the algorithmic correspondence (or acknowledging that the spectrum is a heuristic proxy) is required for reproducibility of the “optimal placement” step.
minor comments (5)
  1. Figure 6 and the surrounding text use “Comm. & Comp.” and “comm.-comp.” inconsistently; expand once and keep a single abbreviation.
  2. Eq. (3) defines the inter-node cost c, yet the concrete values used for Node A/B/C and Clusters A/B are only partially stated later; a short table of c and bandwidth ratios would help.
  3. Table I reports accuracy only for Qwen3 under three p values; a parallel row or column for DeepSeek-V2-Lite would make the “preserves better accuracy” claim fully symmetric with the throughput plots.
  4. Algorithm 1 line 12–13 falls back to uniform random selection when score = 0; a one-sentence note on how often this occurs in practice would be useful.
  5. Typos / style: “Communicationcomputation” in the abstract, “F .” spacing before “Stability”, and occasional missing spaces around × symbols.

Circularity Check

0 steps flagged

No significant circularity: CAP is empirical systems engineering whose throughput and accuracy claims are measured against external baselines, not derived by construction from fitted constants.

full rationale

The paper proposes three algorithmic components (co-activation-driven placement via a greedy graph partition, a spectrum of load-balance swaps, and device-level communication-aware pruning) and evaluates them by end-to-end throughput under SLOs, latency of the placement spectrum, and accuracy on HumanEval/MMLU/GSM8K/PPL. Placement quality is measured by Device-per-Token and Max/Avg load on held-out windows and cross-domain datasets; pruning is compared to a naive score-threshold baseline under the same p; hardware regimes are ablated on three node types and two clusters. No equation claims a first-principles prediction that is algebraically identical to a fitted input; co-activation statistics are collected offline and then used as an ordinary heuristic input, which is standard systems practice rather than circular derivation. Mild pipeline dependence (pruning traces feed placement) is acknowledged and ordered explicitly; it does not force the reported speedups. Self-citations are absent from the load-bearing chain. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claims rest on standard MoE communication models, a few hand-chosen trade-off parameters, and the empirical premise that co-activation graphs are stable. No new physical entities are postulated; the free parameters are ordinary systems knobs.

free parameters (3)
  • pruning threshold p
    User-chosen accuracy–speed knob; set to 0.3 (Qwen3) or 0.1 (DeepSeek-V2-Lite) so that perplexity rises ~10 %. Directly controls reported speedups and accuracy tables.
  • inter-node communication cost c
    Relative weight of cross-node versus intra-node hops in the pruning objective (Eq. 6). Experiments use c=1 and c=5; value is chosen by hand from bandwidth ratios.
  • trade-off weight λ (implicit)
    Controls the communication–load objective (Eq. 5). Realized by the number of expert-swap steps rather than an explicit numeric λ; still a free design choice that selects the final placement.
axioms (4)
  • domain assumption Communication volume of a token equals the number of distinct devices (or nodes) that host its activated experts, not the number of experts (device-centric all-to-all).
    Stated in Section II-C and used throughout the placement and pruning objectives.
  • domain assumption Pairwise co-activation frequencies collected offline are a reliable proxy for future routing destinations.
    Foundation of the co-activation graph and Algorithm 1; validated only empirically in Section VIII-F.
  • domain assumption Routing-score sum on a device is a sufficient proxy for that device’s contribution to final token accuracy.
    Used to define a_d in the pruning ILP (Eq. 6); inherited from prior expert-pruning literature.
  • ad hoc to paper Greedy seed-and-grow partitioning sufficiently approximates the capacity-constrained weighted graph partition that minimizes C(π).
    Algorithm 1 is presented without approximation guarantees; optimality is assumed for practical purposes.
invented entities (2)
  • CAP three-stage pipeline (Placement + Adjustment + Pruning) no independent evidence
    purpose: Unifies communication-aware placement spectrum generation with device-level pruning under a single offline-to-online workflow.
    The ordered four-step pipeline (Fig. 8) is the paper’s central engineering construct; no independent existence outside this work.
  • Communication–load placement spectrum no independent evidence
    purpose: Produces a Pareto front of expert assignments that trade communication volume against load imbalance for hardware-aware selection.
    Generated by iterative expert swaps starting from the co-activation placement; used to claim hardware adaptivity.

pith-pipeline@v1.1.0-grok45 · 21111 in / 2913 out tokens · 23935 ms · 2026-07-11T08:34:47.630411+00:00 · methodology

0 comments
read the original abstract

As MoE models scale to hundreds of experts, placement and pruning decisions increasingly dictate communication volume, affecting the performance of distributed inference across GPUs and nodes. We propose CAP (Communication-Aware Assignment and Pruning), a framework that considers computation, communication and accuracy together for efficient MoE inference through expert placement and pruning. It consists of three components: (1) Co-activation driven expert placement, which groups frequently co-activated experts to reduce inter-device and inter-node communication; (2) Communicationcomputation trade-off adjustment, which generates placements with different computational load and communication volume; and (3) Communication-aware expert pruning, which selectively removes routing destinations to reduce communication with limited accuracy degradation. By combining these components, CAP selects an efficient operating strategy for different hardware configurations. Across our single-node and multi-node experiments, it achieves 1.23x - 1.86 x throughput improvement over DeepSeek EPLB and sequential placement in vLLM, and preserves better model accuracy at the same target speedup under lossy acceleration.

Figures

Figures reproduced from arXiv: 2607.05116 by Jiangsu Du, Xiao Shi, Yingying Sun, Yutong Lu, Zhiguang Chen.

Figure 1
Figure 1. Figure 1: Expert parallelism combined with data parallelism [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Zero-redundancy Token Transfer dense components such as attention and projection layers are replicated. During inference, each device first processes different re￾quests through the dense part of the model, and then uses the gating network to determine the experts required by each token. Since the selected experts may reside on different devices, tokens must be routed to the corresponding devices for exper… view at source ↗
Figure 3
Figure 3. Figure 3: Expert co-activation matrix. Each point represents the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Expert placement impacts the communication. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Grouping frequently co-activated experts onto the same [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comm. & Comp. Trade-off Adjustment performed on every MoE layer to generate an expert placement for the entire model. This hierarchical placement reduces the average number of devices each token needs to interact with, thereby reducing the communication overhead of expert parallelism and alleviating expensive cross-device and cross-node transfers. Meanwhile, the algorithm ensures that each device hosts the… view at source ↗
Figure 7
Figure 7. Figure 7: Communication-aware pruning with a 0.3 threshold [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The 4-step Pipeline of CAP 0.0 0.2 0.4 Pruning threshold p 1.6 1.7 1.8 Load Imbalance (Max/Avg) [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Load balance un￾der different pruning thresh￾olds. 2.5 5.0 7.5 10.0 Popularity Decile (cold -> hot) 0.120 0.125 Avg. Routing Score [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: End-to-end Performance 0 2 4 6 Placement Index 4500 4750 5000 Latency (ms) Node A (Comm.-constrained) 0 2 4 6 Placement Index 4500 5000 Node B (Balanced) 0 2 4 6 Placement Index 400 500 Node C (Comm.-rich) [PITH_FULL_IMAGE:figures/full_fig_p009_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance and Communication-Computation Characteristics of Placement Spectrum 33% Speedup [PITH_FULL_IMAGE:figures/full_fig_p009_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: In-workload stability of expert grouping on LMSYS Chat. Groupings generated from three different windows (w0, [PITH_FULL_IMAGE:figures/full_fig_p010_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 29 linked inside Pith

  1. [1]

    Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wuet al., “Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models,”arXiv preprint arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Llama-moe: Building mixture-of-experts from llama with continual pre- training,

    T. Zhu, X. Qu, D. Dong, J. Ruan, J. Tong, C. He, and Y . Cheng, “Llama-moe: Building mixture-of-experts from llama with continual pre- training,”arXiv preprint arXiv:2406.16554, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...

  5. [5]

    Available: https://arxiv.org/abs/2005.14165

    [Online]. Available: https://arxiv.org/abs/2005.14165

    Pith/arXiv arXiv 2005

  6. [6]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  7. [7]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Grace-moe: Grouping and replication with locality-aware routing for efficient distributed moe inference,

    Y . Han, L. Pan, J. Peng, Z. Tao, W. Zhang, and Y . Zhang, “Grace-moe: Grouping and replication with locality-aware routing for efficient distributed moe inference,” 2025. [Online]. Available: https://arxiv.org/abs/2509.25041

    Pith/arXiv arXiv 2025

  9. [9]

    Toward efficient inference for mixture of experts,

    H. Huang, N. Ardalani, A. Sun, L. Ke, S. Bhosale, H.-H. S. Lee, C.-J. Wu, and B. Lee, “Toward efficient inference for mixture of experts,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview.net/forum?id=stXtBqyTWX

    2024

  10. [10]

    Moesys: A distributed and efficient mixture-of-experts training and inference system for internet services,

    D. Yu, L. Shen, H. Hao, W. Gong, H. Wu, J. Bian, L. Dai, and H. Xiong, “Moesys: A distributed and efficient mixture-of-experts training and inference system for internet services,” 2024. [Online]. Available: https://arxiv.org/abs/2205.10034

    Pith/arXiv arXiv 2024

  11. [11]

    Accelerating distributed moe training and inference with lina,

    J. Li, Y . Jiang, Y . Zhu, C. Wang, and H. Xu, “Accelerating distributed moe training and inference with lina,” 2024. [Online]. Available: https://arxiv.org/abs/2210.17223

    Pith/arXiv arXiv 2024

  12. [12]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” 2022. [Online]. Available: https://arxiv.org/abs/2201.05596

    Pith/arXiv arXiv 2022

  13. [13]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,

    X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,”Proceedings of the ACM on Management of Data, vol. 1, no. 1, p. 1–19, May 2023. [Online]. Available: http://dx.doi.org/10.1145/3588964

    doi:10.1145/3588964 2023

  14. [14]

    Memory- efficient nllb-200: Language-specific expert pruning of a massively multilingual machine translation model,

    Y . Koishekenov, A. Berard, and V . Nikoulina, “Memory- efficient nllb-200: Language-specific expert pruning of a massively multilingual machine translation model,” 2023. [Online]. Available: https://arxiv.org/abs/2212.09811

    Pith/arXiv arXiv 2023

  15. [15]

    Lynx: Enabling efficient moe inference through dynamic batch-aware expert selection,

    V . Gupta, K. Sinha, A. Gavrilovska, and A. P. Iyer, “Lynx: Enabling efficient moe inference through dynamic batch-aware expert selection,”

  16. [16]

    Available: https://arxiv.org/abs/2411.08982

    [Online]. Available: https://arxiv.org/abs/2411.08982

    Pith/arXiv arXiv

  17. [17]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,

    X. Lu, Q. Liu, Y . Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li, “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.14800

    Pith/arXiv arXiv 2024

  18. [18]

    Task-specific expert pruning for sparse mixture-of-experts,

    T. Chen, S. Huang, Y . Xie, B. Jiao, D. Jiang, H. Zhou, J. Li, and F. Wei, “Task-specific expert pruning for sparse mixture-of-experts,”

  19. [19]

    Available: https://arxiv.org/abs/2206.00277

    [Online]. Available: https://arxiv.org/abs/2206.00277

    Pith/arXiv arXiv

  20. [20]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171–4186

    2019

  22. [22]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,”arXiv preprint arXiv:2006.16668, 2020

    Pith/arXiv arXiv 2006

  23. [23]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 2017. [Online]. Available: https://arxiv.org/abs/1701.06538

    Pith/arXiv arXiv 2017

  24. [24]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, J. Chau, P. Cheng, F. Yang, M. Yang, and Y . Xiong, “Tutel: Adaptive mixture-of-experts at scale,” 2023. [Online]. Available: https://arxiv.org/abs/2206.03382

    Pith/arXiv arXiv 2023

  27. [27]

    Megablocks: Efficient sparse training with mixture-of-experts,

    T. Gale, D. Narayanan, C. Young, and M. Zaharia, “Megablocks: Efficient sparse training with mixture-of-experts,” 2022. [Online]. Available: https://arxiv.org/abs/2211.15841

    Pith/arXiv arXiv 2022

  28. [28]

    Faster- moe: modeling and optimizing training of large-scale dynamic pre- trained models,

    J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Faster- moe: modeling and optimizing training of large-scale dynamic pre- trained models,” inProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022, pp. 120–134

    2022

  29. [29]

    SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,

    M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,” in2023 USENIX Annual Technical Conference (USENIX ATC 23). Boston, MA: USENIX Association, Jul. 2023, pp. 961–975. [Online]. Available: https://www.usenix.org/conference/atc23/presentation/zhai

    2023

  30. [30]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  31. [31]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “Sglang: Efficient execution of structured language model programs,”

  32. [32]

    Available: https://arxiv.org/abs/2312.07104

    [Online]. Available: https://arxiv.org/abs/2312.07104

    Pith/arXiv arXiv

  33. [33]

    Discovering important experts for mixture-of-experts models pruning through a theoretical perspective,

    W. Huang, Y . Zhang, X. Zheng, F. Chao, R. Ji, and L. Cao, “Discovering important experts for mixture-of-experts models pruning through a theoretical perspective,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=7kQjbCQwtT

    2025

  34. [34]

    Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing,

    S. Go and D. Mahajan, “Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing,” 2025. [Online]. Available: https://arxiv.org/abs/2502.06643

    Pith/arXiv arXiv 2025

  35. [35]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2022. [Online]. Available: https://arxiv.org/abs/2101.03961

    Pith/arXiv arXiv 2022

  36. [36]

    Efficientmoe: Optimizing mixture-of-experts model training with adaptive load balance,

    Y . Zeng, C. Huang, Y . Mei, L. Zhang, T. Su, W. Ye, W. Shi, and S. Wang, “Efficientmoe: Optimizing mixture-of-experts model training with adaptive load balance,”IEEE Transactions on Parallel and Dis- tributed Systems, 2025

    2025

  37. [37]

    Scalable and efficient moe training for multitask multilingual models,

    Y . J. Kim, A. A. Awan, A. Muzio, A. F. C. Salinas, L. Lu, A. Hendy, S. Rajbhandari, Y . He, and H. H. Awadalla, “Scalable and efficient moe training for multitask multilingual models,” 2021. [Online]. Available: https://arxiv.org/abs/2109.10465

    Pith/arXiv arXiv 2021

  38. [38]

    Moe-i 2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition,

    C. Yang, Y . Sui, J. Xiao, L. Huang, Y . Gong, Y . Duan, W. Jia, M. Yin, Y . Cheng, and B. Yuan, “Moe-i 2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition,” 2024. [Online]. Available: https://arxiv.org/abs/2411.01016

    Pith/arXiv arXiv 2024

  39. [39]

    Stun: Structured-then-unstructured pruning for scalable moe pruning,

    J. Lee, seung-won hwang, A. Qiao, D. F. Campos, Z. Yao, and Y . He, “Stun: Structured-then-unstructured pruning for scalable moe pruning,”

  40. [40]

    Available: https://arxiv.org/abs/2409.06211

    [Online]. Available: https://arxiv.org/abs/2409.06211

    Pith/arXiv arXiv

  41. [41]

    Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of- experts,

    Z. Zhang, X. Liu, H. Cheng, C. Xu, and J. Gao, “Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of- experts,” 2025. [Online]. Available: https://arxiv.org/abs/2407.09590

    Pith/arXiv arXiv 2025

  42. [42]

    A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts,

    M. N. R. Chowdhury, M. Wang, K. E. Maghraoui, N. Wang, P.-Y . Chen, and C. Carothers, “A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts,” 2024. [Online]. Available: https://arxiv.org/abs/2405.16646

    Pith/arXiv arXiv 2024