Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HierMoE claims that the dominant communication bottleneck in mixture-of-experts (MoE) training can be relieved by deduplicating tokens at every level of the GPU cluster's network hierarchy and by swapping expert placements to balance load,

desk verdict A solid systems contribution with a real dedup/swap idea, but the end-to-end speedup over Tutel-2DH and SmartMoE is claimed, not measured. read the letter →

arxiv 2508.09591 v1 pith:ZB2BIIQP submitted 2025-08-13 cs.DC cs.LG

Wenxiang Lin , Xinglin Pan , Lin Zhang , Shaohuai Shi , Xuan Wang , Xiaowen Chu This is my paper · ORCID
classification cs.DCcs.LG
keywords mixture-of-expertsexpertparallelismAlltoAlltokendeduplicationswaphierarchicaltopologydistributedtrainingMoEcommunicationoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HierMoE, a distributed training system that treats the cluster's multi-level network topology (e.g., NVLink, PCIe, QPI, InfiniBand) as a series of AlltoAll stages and removes duplicated token transfers at each stage. It also proposes a hierarchical expert-swap strategy that rebalances routing workloads across GPUs while preserving the benefits of deduplication. Two theoretical models—one for communication time and one for the cost of swapping any pair of experts—let the system choose the optimal hierarchy dimension and swap pair for a given hardware configuration. On a 32-GPU cluster running DeepSeek-V3 and Qwen3-30B-A3B, the claims are 1.55x to 3.32x faster AlltoAll communication and 1.18x to 1.27x faster end-to-end training over baselines like Tutel-2DH, SmartMoE, and Megatron-LM.

What carries the argument

The central machinery is a pair of linked optimization problems built on a linear communication-cost model. The first, HierD-AlltoAll, computes the number of duplicate-free tokens assigned to each expert group at each hierarchical level and chooses the optimal hierarchy dimension $d^*$ by minimizing the modeled time $t_d$ across all dimensions. The second, HierD-ES, constructs a matrix $Q_d[r,c]$ estimating the communication time after swapping expert $r$ with expert $c$, then selects the pair $(r^*,c^*)$ that minimizes this matrix. Both rely on the assumption that AlltoAll time is well approximated by $t = \alpha + \beta \cdot n$, where $n$ is the product of the number of participating GPUs

What would settle it

Measure NCCL AlltoAll time on a cluster with deliberately skewed MoE routing distributions and compare it against the linear prediction $\alpha+\beta \cdot G \cdot \max(p)$. If the prediction error changes which hierarchy dimension $d^*$ minimizes the modeled time, or if the expert pair chosen by Eq. (10) is no longer the fastest on the actual hardware, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the conventional one-shot AlltoAll used in MoE training ignores the cluster's hierarchical bandwidth structure, and this oversight causes both redundant traffic and load imbalance. HierMoE first groups experts according to the cluster's natural hierarchy and performs a sequence of inter-level and intra-level AlltoAll operations, deduplicating tokens that would otherwise be sent multiple times to experts residing on the same GPU or group. It then formulates the expected communication time for every possible hierarchy depth as a linear function of the largest per-group token count, selects the depth that minimizes that time, and, within that choice, finds the expert p

Load-bearing premise

The whole strategy—the chosen hierarchy depth and the chosen expert swap—rests on a linear model that assumes AlltoAll time equals a startup constant plus a per-byte cost times the largest padded per-GPU message, so if real NCCL AlltoAll time deviates from this linear fit under imbalanced routing, the selected configuration may not be the true optimum.

Editorial extensions

If this is right

  • If the model holds, MoE training systems can systematically reduce AlltoAll traffic without changing routing algorithms or model accuracy, preserving convergence behavior.
  • The optimal dimension selection lets the same training framework adapt to clusters with different topologies, from a single NVLink node to multi-level switched clusters.
  • The expert-swap mechanism becomes compatible with deduplication, unlike prior load-balancing schemes that counted raw token assignments and could increase traffic.
  • The reported speedups grow with higher top-$K$ values and larger expert counts, where duplicated tokens are more common, suggesting the benefit will increase in newer, wider MoE models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The deduplication-plus-swap logic could transfer to MoE inference or to other sparse collectives such as top-$K$ gather and scatter patterns, where traffic is similarly dominated by duplicate small messages.
  • The reliance on $\max(p)$ may understate the true cost under heavily skewed routing; a quantile-based or distribution-aware overhead estimator could be more robust in production traffic and is a natural extension to test.
  • On larger clusters with more hierarchy levels, deduplication at intermediate levels should yield proportionally larger savings, a claim the paper does not directly measure beyond 32 GPUs.
  • Because the expert swap is recomputed every iteration, adapting its frequency to routing drift or router entropy could capture most of the benefit at lower overhead than a fixed per-iteration update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. HierMoE proposes two topology-aware optimizations for MoE training on hierarchical GPU clusters: hierarchical token deduplication (HierD-AlltoAll) and hierarchical expert swap (HierD-ES). The paper builds a linear communication model for hierarchical AlltoAll, uses it to select the optimal hierarchy dimension d* and the best expert pair to swap, and implements the system atop Megatron-LM. Experiments on a 32-GPU A6000 cluster with DeepSeek-V3 and Qwen3-30B-A3B report 1.55×–3.32× faster AlltoAll communication and 1.18×–1.27× faster end-to-end training over Tutel-2DH, SmartMoE, and Megatron-LM, with ablations over K, E, G, hierarchy dimension, and swap update frequency.

Significance. If the claims hold, HierMoE addresses a well-known bottleneck in MoE training: AlltoAll communication over hierarchical interconnects. The paper's strengths include a communication model that fits microbenchmarks with high R² (0.997–0.999997), measured communication speedups over multiple baselines, and systematic ablations that probe sensitivity to key parameters. However, the headline end-to-end claim is not fully supported by the reported experiments, and the core performance model is validated only on uniform traffic. These issues are substantial but appear fixable, so the underlying idea is worth pursuing.

major comments (3)
  1. [Abstract; §I; §V-C, Fig. 10] The abstract and §I claim 1.18–1.27× end-to-end speedup 'compared to ... Tutel-2DH, SmartMoE, and Megatron-LM'. The end-to-end evidence in §V-C/Fig. 10 compares only HierMoE against Megatron-LM (plus HD2-MoE and HD2-MoE-Smart). Tutel-2DH does not appear in any end-to-end plot, and SmartMoE appears only as HD2-MoE-Smart, which combines the authors' HD2-MoE with SmartMoE's swap strategy, not as a standalone baseline. Since AlltoAll is 30–60% of training time and baselines may overlap or hide communication, communication-only speedups from Fig. 11 do not imply end-to-end speedups. The headline overstates what is measured. Either add end-to-end training-time measurements for the named baselines or revise the abstract and conclusion accordingly.
  2. [§III-D, Algorithm 1, Line 9] Algorithm 1 uses 'I(k+1,E)_route ← process(I(k,E)_route)' without defining the 'process' operation. This step is central to computing p(k+1,G)_a2a for the next hierarchy level, i.e., how the routing mask changes after an Inter-level-(k) communication and how duplicated tokens are removed. Without a formal definition of this transformation, the deduplication algorithm is not reproducible. Please specify the operation precisely.
  3. [§III-B, Eq. (2)–(5); Fig. 9] The communication model uses max(p) (and max of pInter/pIntra) as the per-rank message volume, effectively assuming every GPU sends the maximum number of tokens to every destination. This is a padded-volume proxy. The microbenchmark fits in Fig. 9 are conducted on uniform traffic; the paper does not validate the model under imbalanced MoE routing distributions, where actual per-rank volumes differ. Since d* (Eq. (6)) and the expert swap pair (Eq. (10)) are selected by minimizing this model, an inaccurate max-based approximation could produce suboptimal choices. Please either justify that NCCL AlltoAll time is determined solely by the maximum per-rank volume in the tested regime, or validate the model with real routing distributions.
minor comments (6)
  1. [§III-C, Eq. (7)] The notation I(I(l,g)_route)[i,j] is confusing: the outer I is apparently the indicator function but it is not defined. Please distinguish the routing mask from the indicator function.
  2. [§II-B] Typo: 'DeepSeed-MoE' should be 'DeepSpeed-MoE'.
  3. [§IV-A, Theorem 1] Theorem 1 is essentially a definition of the optimization step (argmin over the precomputed matrix Q*), and its proof restates the construction. Relabeling it as an optimization procedure rather than a theorem would be more accurate.
  4. [§V-B, last paragraph] The sentence 'The total time required for communication in the performance models is under 300 seconds' is unclear. It presumably refers to the time to collect microbenchmark measurements; please rephrase.
  5. [§V-C] The text says HierMoE is compared with Megatron-LM and SmartMoE, but Fig. 10 does not show standalone SmartMoE. Clarify that SmartMoE is evaluated only in the combined HD2-MoE-Smart configuration.
  6. [Abstract; §II-C] The paper states that the approach 'does not compromise model accuracy' but provides no loss or accuracy curves. Since token deduplication and expert swap preserve the underlying computation, a short formal argument would substantiate this claim; as written, it is not evidenced.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cost model is calibrated on hardware microbenchmarks and the speedups are measured, not predicted by fitted parameters.

full rationale

The derivation chain is self-contained. Eq. (1)-(5) define a linear AlltoAll time model; the alpha/beta parameters are fit from nccl-tests microbenchmarks on the target cluster and validated with R^2 >= 0.997 (Section V-B, Fig. 9). These fitted parameters are used only to choose the hierarchical dimension d* (Eq. (6)) and the expert-swap pair (Eq. (10)); they are not renamed as the reported speedups. The reported 1.55-3.32x communication and 1.18-1.27x end-to-end improvements are measured comparisons in Fig. 10 and Fig. 11, so no fitted input is converted into a predicted outcome. Eq. (6) and Theorem 1 are minimizer statements relative to the defined cost matrix Q*, not predictions derived from the result they claim; the theorem's proof is a finite exhaustive-search argument, and the actual reduction from swapping is confirmed empirically, not by the argmin definition. The linear-model form is cited to [22], which shares authors, but the present paper re-fits and validates the model on its own hardware, so the citation is not load-bearing. The only notable issue is outside circularity: the abstract and Section I claim end-to-end speedups over Tutel-2DH and SmartMoE, but Section V-C/Fig. 10 report end-to-end speedups only over Megatron-LM (plus HD2-MoE ablations), while Tutel-2DH appears only in the AlltoAll comparison (Fig. 11) and SmartMoE is represented by the hybrid HD2-MoE-Smart variant. This is a missing-support/reporting gap, not a circular derivation, and does not change the circularity score. A possible model-mismatch for imbalanced routing (max(p) in Eq. (2)) would be a robustness concern, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The system's design choices (optimal dimension, swap pair) are made by minimizing a calibrated linear model of AlltoAll time. The calibration constants are fit to hardware microbenchmarks; they are inputs, not outputs of the evaluation. The smooth-max exponent gamma is chosen by hand but shown insensitive. No invented entities are introduced.

free parameters (2)
  • alpha and beta per AlltoAll level = alpha_a2a=7.22e-1, beta_a2a=5.70e-7; Inter-level-1: 4.97e-1, 5.29e-7; etc. (Fig. 9)
    Fitted via least squares on nccl-tests microbenchmarks; used in Eq. (1) and (3)-(6) to select optimal dimension and expert swap.
  • gamma (smooth-max exponent) = 10 (default)
    Hand-chosen for Eq. (11); sensitivity tested over [5,19] with speedup 1.16x-1.17x, so not load-bearing.
assumptions (3)
  • domain assumption AlltoAll time is linear in message volume with a fixed startup cost per hierarchy level
    Eq. (1)-(5); validated only on uniform-size messages in microbenchmarks, extrapolated to imbalanced MoE routing.
  • domain assumption Tokens selecting multiple experts in the same group can be deduplicated without changing training semantics
    Used throughout Section III; correctness requires that a token is processed once per expert after a single inter-group transfer, which depends on the hierarchy preserving the routing mask as modeled.
  • domain assumption Swapping the positions of two experts (with their optimizer states) leaves training trajectories mathematically unchanged
    Invoked in Section IV and stated in the introduction ('does not compromise model accuracy'); no convergence experiment is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap." pith.science (2026). https://pith.science/paper/ZB2BIIQP

@misc{pith2026250809591,
  author       = {Pith},
  title        = {Pith review of: HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZB2BIIQP}},
  note         = {Machine review of arXiv:2508.09591}
}
abstract

The sparsely activated mixture-of-experts (MoE) transformer has become a common architecture for large language models (LLMs) due to its sparsity, which requires fewer computational demands while easily scaling the model size. In MoE models, each MoE layer requires to dynamically choose tokens to activate particular experts for computation while the activated experts may not be located in the same device or GPU as the token. However, this leads to substantial communication and load imbalances across all GPUs, which obstructs the scalability of distributed systems within a GPU cluster. To this end, we introduce HierMoE to accelerate the training of MoE models by two topology-aware techniques: 1) token deduplication to reduce the communication traffic, and 2) expert swap to balance the workloads among all GPUs. To enable the above two proposed approaches to be more general, we build theoretical models aimed at achieving the best token duplication and expert swap strategy under different model configurations and hardware environments. We implement our prototype HierMoE system atop Megatron-LM and conduct experiments on a 32-GPU cluster with DeepSeek-V3 and Qwen3-30B-A3B models. Experimental results show that our HierMoE achieves $1.55\times$ to $3.32\times$ faster communication and delivers $1.18\times$ to $1.27\times$ faster end-to-end training compared to state-of-the-art MoE training systems, Tutel-2DH, SmartMoE, and Megatron-LM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods

    cs.DC 2026-07 conditional novelty 6.0 of 10

    UBEP replaces BSP All-to-All for MoE on multi-tier superpods with dependency-driven kernel decomposition, topology-aware token scheduling, and Data-as-Flag atomics, cutting All-to-All latency up to 52.4% and TPOT up to 11.1%.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” in International Conference on Learning Representations, 2016

    work page 2016

  2. [2]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” in International Conference on Learning Representations, 2020

    work page 2020

  3. [3]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    arXiv 2024

  4. [4]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv et al. , “Qwen3 technical report,” arXiv preprint arXiv:2505.09388, 2025

    arXiv 2025

  5. [5]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,

    DeepSeek-AI, “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,” 2024

    work page 2024

  6. [6]

    Jetmoe: Reaching llama2 performance with 0.1m dollars,

    Y . Shen, Z. Guo, T. Cai, and Z. Qin, “Jetmoe: Reaching llama2 performance with 0.1m dollars,” CoRR, vol. abs/2404.07413, 2024

    arXiv 2024

  7. [7]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  8. [8]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al. , “Tutel: Adaptive mixture-of-experts at scale,” Proceedings of Machine Learning and Systems , vol. 5, 2023

    work page 2023

Show all 39 references
  1. [9]

    Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,

    R. Liu, Y . J. Kim, A. Muzio, and H. Hassan, “Gating dropout: Communication-efficient regularization for sparsely activated transform- ers,” in International Conference on Machine Learning . PMLR, 2022, pp. 13 782–13 792

    2022

  2. [10]

    Accelerating distributed {MoE} training and inference with lina,

    J. Li, Y . Jiang, Y . Zhu, C. Wang, and H. Xu, “Accelerating distributed {MoE} training and inference with lina,” in USENIX Annual Technical Conference, 2023, pp. 945–959

    2023

  3. [11]

    Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,

    S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y . Yang, B. Li, and X. Chu, “Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,” in Proceedings of the Nineteenth European Conference on Computer Systems , 2024, pp. 236–249

    2024

  4. [12]

    Fsmoe: A flexible and scalable training system for sparse mixture- of-experts models,

    X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, and X. Chu, “Fsmoe: A flexible and scalable training system for sparse mixture- of-experts models,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operati...

    2025

  5. [13]

    BASE layers: Simplifying training of large, sparse models,

    M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer, “BASE layers: Simplifying training of large, sparse models,” in Interna- tional Conference on Machine Learning. PMLR, 2021, pp. 6265–6274

    2021

  6. [14]

    Mixture-of-experts with expert choice routing,

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. Dai, Z. Chen, Q. Le, and J. Laudon, “Mixture-of-experts with expert choice routing,” arXiv preprint arXiv:2202.09368 , 2022

    2022 arXiv

  7. [15]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”The Journal of Machine Learning Research , vol. 23, no. 1, pp. 5232–5270, 2022

    2022

  8. [16]

    On the representation collapse of sparse mixture of experts,

    Z. Chi, L. Dong, S. Huang, D. Dai, S. Ma, B. Patra, S. Singhal, P. Bajaj, X. Song, X.-L. Mao et al., “On the representation collapse of sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 35, pp. 34 600–34 613, 2022

    2022

  9. [17]

    From sparse to soft mixtures of experts,

    J. Puigcerver, C. Riquelme, B. Mustafa, and N. Houlsby, “From sparse to soft mixtures of experts,” arXiv preprint arXiv:2308.00951 , 2023

    2023 arXiv

  10. [18]

    Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale,” in International Conference on Machine Learning . PMLR, 2022, pp. 18 332–18 346

    2022

  11. [19]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley et al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in International Conference for High Performance Computing, ...

    2022

  12. [20]

    Bagualu: targeting brain scale pretrained models with over 37 million cores,

    Z. Ma, J. He, J. Qiu, H. Cao, Y . Wang, Z. Sun, L. Zheng, H. Wang, S. Tang, T. Zheng et al. , “Bagualu: targeting brain scale pretrained models with over 37 million cores,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Program- ming, 2...

    2022

  13. [21]

    Faster- MoE: modeling and optimizing training of large-scale dynamic pre- trained models,

    J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Faster- MoE: modeling and optimizing training of large-scale dynamic pre- trained models,” in Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022, pp. 120–134

    2022

  14. [22]

    PipeMoE: Accelerating mixture- of-experts through adaptive pipelining,

    S. Shi, X. Pan, X. Chu, and B. Li, “PipeMoE: Accelerating mixture- of-experts through adaptive pipelining,” in IEEE INFOCOM 2023-IEEE Conference on Computer Communications , 2023

    2023

  15. [23]

    SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,

    M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,” in USENIX Annual Technical Conference , 2023, pp. 961–975

    2023

  16. [24]

    Janus: A unified distributed training framework for sparse mixture-of-experts models,

    J. Liu, J. H. Wang, and Y . Jiang, “Janus: A unified distributed training framework for sparse mixture-of-experts models,” in Proceedings of the ACM SIGCOMM 2023 Conference , 2023, pp. 486–498

    2023

  17. [25]

    Parm: Efficient training of large sparsely-activated models with dedicated schedules,

    X. Pan, W. Lin, S. Shi, X. Chu, W. Sun, and B. Li, “Parm: Efficient training of large sparsely-activated models with dedicated schedules,” in IEEE INFOCOM 2024-IEEE Conference on Computer Communications, 2024

    2024

  18. [26]

    Cen- tauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning,

    C. Chen, X. Li, Q. Zhu, J. Duan, P. Sun, X. Zhang, and C. Yang, “Cen- tauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning,” in Proceedings of the 29th ACM International Conference on Architectural Su...

    2024

  19. [27]

    Lancet: Accelerating mixture-of-experts training by overlapping weight gradient computation and all-to-all communication,

    C. Jiang, Y . Tian, Z. Jia, C. Wu, Y . Wang, and S. Zheng, “Lancet: Accelerating mixture-of-experts training by overlapping weight gradient computation and all-to-all communication,” Proceedings of Machine Learning and Systems , vol. 6, pp. 74–86, 2024

    2024

  20. [28]

    Sp- moe: Expediting mixture-of-experts training with optimized pipelining planning,

    N. Wang, W. Lin, L. Zhang, S. Shi, R. Zhou, and B. Li, “Sp- moe: Expediting mixture-of-experts training with optimized pipelining planning,” in IEEE INFOCOM 2025-IEEE Conference on Computer Communications. IEEE, 2025, pp. 1–10

    2025

  21. [29]

    Mitigating contention in stream multiprocessors for pipelined mixture of experts: An sm-aware scheduling approac,

    X. Pan, R. Wang, W. Lin, S. Shi, and X. Chu, “Mitigating contention in stream multiprocessors for pipelined mixture of experts: An sm-aware scheduling approac,” in 2025 IEEE 45th International Conference on Distributed Computing Systems (ICDCS) . IEEE, 2025

    2025

  22. [30]

    Mast: Efficient training of mixture-of-experts transformers with task pipelining and ordering,

    W. Lin, X. Pan, S. Shi, X. Wang, B. Li, and X. Chu, “Mast: Efficient training of mixture-of-experts transformers with task pipelining and ordering,” in 2025 IEEE 45th International Conference on Distributed Computing Systems (ICDCS) . IEEE, 2025

    2025

  23. [31]

    Scheinfer: Efficient inference of large language models with task scheduling on moderate gpus

    W. Lin, X. Pan, S. Shi, X. Wang, and X. Chu, “Scheinfer: Efficient inference of large language models with task scheduling on moderate gpus.” in European Conference on Parallel Processing. Springer, 2025

    2025

  24. [32]

    Data center tcp (dctcp),

    M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prab- hakar, S. Sengupta, and M. Sridharan, “Data center tcp (dctcp),” in Proceedings of the ACM SIGCOMM 2010 Conference, 2010, pp. 63–74

    2010

  25. [33]

    A scalable, commodity data center network architecture,

    M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity data center network architecture,” ACM SIGCOMM computer communication review, vol. 38, no. 4, pp. 63–74, 2008

    2008

  26. [34]

    {TopoOpt}: Co-optimizing network topol- ogy and parallelization strategy for distributed training jobs,

    W. Wang, M. Khazraee, Z. Zhong, M. Ghobadi, Z. Jia, D. Mudigere, Y . Zhang, and A. Kewitsch, “{TopoOpt}: Co-optimizing network topol- ogy and parallelization strategy for distributed training jobs,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI ...

    2023

  27. [35]

    Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures,

    C. Zhao, C. Deng, C. Ruan, D. Dai, H. Gao, J. Li, L. Zhang, P. Huang, S. Zhou, S. Ma et al. , “Insights into deepseek-v3: Scaling challenges and reflections on hardware for ai architectures,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture , ...

    2025

  28. [36]

    Large scale distributed deep networks,

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ran- zato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,” Advances in neural information processing systems , vol. 25, 2012

    2012

  29. [37]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

    2020 arXiv

  30. [38]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,

    X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,” Proceedings of the ACM on Management of Data, vol. 1, no. 1, pp. 1–19, 2023

    2023

  31. [39]

    \ell 1, p-norm regularization: Error bounds and convergence rate analysis of first-order methods,

    Z. Zhou, Q. Zhang, and A. M.-C. So, “ \ell 1, p-norm regularization: Error bounds and convergence rate analysis of first-order methods,” in International conference on machine learning. PMLR, 2015, pp. 1501– 1510

    2015

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.