Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Predicting only the aggregate token-to-expert distribution, not exact token routes, improves end-to-end MoE inference by more than 23% over Token-to-Expert Prediction on Mixtral 8x7B/MMLU.

desk verdict Qualitative guidance is sound, but the 23% speedup rests on an unvalidated error-distribution model that needs a sensitivity analysis. read the letter →

arxiv 2506.07366 v1 pith:LOC5Y6S5 submitted 2025-06-09 cs.LG cs.AR

Haiyue Ma , Zhixu Du , Yiran Chen This is my paper
classification cs.LGcs.AR
keywords Mixture-of-Expertsinferenceloadbalancingexpertduplicationdistribution-onlypredictiontoken-to-expertprefilllatencyLLMsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that choosing a prediction strategy for dynamic expert duplication in Mixture-of-Experts inference is a system-level tradeoff, not just an accuracy problem, and that a cheap predictor—one that estimates only the overall fraction of tokens going to each expert—is often the best choice. It introduces MoE-GPS, a simulation framework that takes a model architecture and hardware setup and returns the prediction strategy with the shortest end-to-end prefill latency. Using Mixtral 8x7B on the MMLU dataset, the paper reports that this Distribution-Only Prediction improves end-to-end inference performance by more than 23% compared with Token-to-Expert Prediction, which predicts exactly which token goes to which expert. If that is right, expensive token-level predictors may be unnecessary in many common configurations, especially when communication is not the bottleneck.

What carries the argument

The load-bearing object is an augmented block-level simulator of LLM inference, extended with MoE expert parallelism, Mixtral architecture support, tunable predictor accuracy and overhead, and an expert-duplication algorithm that iteratively copies experts from overloaded to underloaded GPUs until per-GPU token counts are balanced. Alongside it, the paper defines a skewness metric—tokens in the most popular expert divided by the average tokens per expert—and a three-scenario error model that converts a predictor's error rate into a runtime multiplier on the bottleneck GPU. In the default typical scenario, errors are assumed to spread uniformly across GPUs, so the most loaded GPU processes up to $(1+\epsilon)$ times the average tokens. This accuracy-to-runtime mapping is what allows MoE-GPS to compare strategies and produce the 23% claim.

What would settle it

Run Mixtral 8x7B prefill on four A100 GPUs with a trained token-level predictor and record each GPU's actual token count per layer; if the busiest GPU routinely exceeds $(1+\epsilon)$ times the average for the measured error rate, the typical error model that generates the 23% claim is not representative.

Watch

Extended reading notes

Core claim

The central claim is that expert duplication guided by Distribution-Only Prediction—estimating only the proportion of tokens each expert receives—delivers lower end-to-end MoE inference latency than Token-to-Expert Prediction in typical settings, because it removes predictor overhead while still fixing the dominant compute imbalance. The paper's simulator shows a U-shaped latency curve for Token-to-Expert Prediction: higher accuracy improves expert placement but raises overhead, so the best token-level configuration sits at an intermediate accuracy, and the zero-overhead Distribution-Only strategy beats even that best configuration by more than 23% on Mixtral 8x7B/MMLU. The framework also produces guidelines: Distribution-Only wins when skewness is low and interconnects are fast, while Token-to-Expert wins when skewness is high and communication is expensive, because accurate token-level predictions become cheaper to obtain and the communication savings dominate.

Load-bearing premise

The 23% speedup rests on the simulator's default assumption that a predictor's errors are spread evenly across GPUs, so the busiest GPU ends up processing at most $(1+\epsilon)$ times the average token load.

Editorial extensions

If this is right

  • If the central claim holds, system designers should treat exact token-to-expert prediction as a niche tool for communication-bound or high-skewness settings, not as the default choice.
  • On fast interconnects such as NVLink and with typical MMLU-like skewness of about 1.4, Distribution-Only Prediction is the better strategy, giving the reported 23% end-to-end gain over the best token-level configuration.
  • For Token-to-Expert Prediction, maximizing accuracy is counterproductive; the simulated latency curve is U-shaped, so the optimal design sits at an intermediate accuracy that balances placement quality against overhead.
  • At long sequence lengths, token-level predictors either hit an accuracy floor or fail to parallelize well, making Distribution-Only Prediction more favorable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 23% is sensitive to the simulator's default assumption that prediction errors are spread evenly across GPUs; on a cluster where errors concentrate on one device, the gap between Distribution-Only and Token-to-Expert Prediction would narrow and the better strategy could flip.
  • A natural validation experiment is to replay Mixtral prefill traces through the simulator with measured per-GPU token loads; agreement with the 'typical' error model would build confidence, while systematic mismatch would require revising the error model.
  • The framework assumes expert-duplication communication can be hidden behind attention compute; at very small batch sizes or with unusually large expert weights, that hiding may fail and change the strategy tradeoff.
  • Because the Distribution-Only estimator is a per-layer static multinomial, workloads whose expert popularity drifts during a long prefill would degrade its accuracy; a moving-average or drift-aware estimator is a natural follow-up the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MoE-GPS, a simulation framework built on LLMCompass to guide the selection of expert prediction strategies for dynamic expert duplication in multi-GPU Mixture-of-Experts inference. It compares two strategies: Distribution-Only Prediction, which estimates only the aggregate token-to-expert proportions, and Token-to-Expert Prediction, which predicts exact token-to-expert mappings. Using the simulator with Mixtral 8x7B and MMLU-like skewness, the paper claims that Distribution-Only Prediction improves end-to-end inference performance by more than 23% relative to the best Token-to-Expert configuration, and it formulates qualitative guidelines linking the preferred strategy to skewness and interconnect bandwidth.

Significance. If validated, the paper provides a practical tool and design guidance for a real systems problem: dynamic expert duplication in MoE inference. The framework covers a useful design space (predictor accuracy, overhead, skewness, and interconnect), and the authors provide the MLE derivation, predictor architecture details, and generalization experiments on LLaMA-MoE and Switch Transformer. The central result, however, is entirely simulation-based and depends on an unvalidated error-distribution model and fitted overhead curves; the quantitative headline should be treated as conditional unless these are strengthened. The qualitative insights—that low-overhead distribution-level prediction can beat token-level prediction when communication is not the bottleneck—are plausible and worth reporting, but the paper's current evidence does not definitively establish the claimed margin.

major comments (4)
  1. [Section 3.3, Figure 5, Figure 6] The default 'Typical' error model, under which the most loaded GPU handles (1+epsilon) times the average token count, is the only channel through which prediction error affects the simulated runtime, and it directly generates the >23% speedup reported at skewness=1.4 in Section 4. No empirical justification is provided for the uniform distribution of errors; the pessimistic case is dismissed as 'unlikely' without measurement. Moreover, the pessimistic formula is stated inconsistently: the text says the bottleneck GPU handles N*(1+epsilon)*avg_tokens, while Figure 5 gives T*(1+epsilon*(N-1)). Because the relative advantage of Distribution-Only versus Token-to-Expert can change under a concentrated-error scenario, the central claim rests on an arbitrary modeling choice. The paper should report a sensitivity analysis over the three error scenarios and, ideally, measure the actual error concentration from real routing on the target hardware.
  2. [Section 3.2.2, Figure 4, Section 4] The comparison that produces the headline 23% figure is computed from the paper's own fitted curves: an exponential fit for accuracy-versus-overhead and a polynomial fit for accuracy-versus-performance, with overhead interpolated for skewness values without direct measurements. Distribution-Only Prediction is assigned zero overhead by construction (Section 4: 'Since distribution is estimated offline, there is no prediction overhead'), while Token-to-Expert Prediction carries overhead derived from these fits. Thus the magnitude of the claimed advantage is substantially shaped by the chosen curve fits and by the zero-overhead assignment. Without validating the fitted curves against end-to-end runtime measurements of the actual predictors at the studied batch size and sequence length, the quantitative claim is not established.
  3. [Section 3.3, Section 3.2.1] The error rate defined in Section 3.2.1 as |p-hat - p| divided by (1/number of experts) is an average normalized L1 distance over experts, but it is then used in the simulator as a single scalar epsilon applied to the bottleneck GPU's load in the 'Typical' model. This conflation of average distribution error with worst-case load inflation is not justified: a small average error can lead to very different bottleneck loads depending on which expert's estimate is wrong. The paper should state explicitly how the defined error rate maps to the runtime model and validate that mapping with experiments, otherwise the simulated speedup is partly an artifact of this conversion.
  4. [Section 3.3, Section 4, Figure 6c/6d] The same uniform-error assumption is applied to communication overhead for Token-to-Expert Prediction (last paragraph of Section 3.3), but for communication the paper's own argument says that misrouted tokens inevitably trigger additional transfers. Since Distribution-Only Prediction does not reduce communication cost, the relative benefit of Token-to-Expert in communication-bottlenecked setups (PCIe, Figure 6d and Figure 7) depends heavily on whether errors in the communication stage are also uniformly distributed. The paper provides no evidence for this and no sensitivity bound; the discussion should at least test the concentrated-error case for communication, or state the conditions under which the qualitative conclusion would reverse.
minor comments (6)
  1. [Title] The title contains a typo: 'Guidlines' should be 'Guidelines'.
  2. [Figure 7 and Section 4] There is an inconsistency in interconnect bandwidth values: Section 4 and Figure 6 state PCIe 4.0 with 32 GB/s, while Figure 7 says the 64 GB/s setting corresponds to Figure 6d. Please align these numbers.
  3. [Figure 6 caption] The caption contains 'preditor', which should be 'predictor'.
  4. [Section 4] The method for interpolating overhead for skewness values without a matching dataset is not described; please state the interpolation scheme (e.g., linear in skewness) and its uncertainty.
  5. [Appendix B] The appendix begins with the fragment 'e formulate'; it should be 'We formulate'.
  6. [Abstract and Section 4] The abstract states a 23% improvement without specifying that it is relative to the best Token-to-Expert configuration found at that skewness; please state the baseline and the operating point clearly (as done in Section 4).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central comparison is a simulation that uses measured error rates and externally validated LLMCompass, with assumptions that are correctness risks rather than circular reductions.

full rationale

The paper's derivation chain is not circular. Distribution-Only Prediction's error rate is measured on held-out test data (Section 3.2.1), Token-to-Expert Prediction's accuracy and overhead are measured from trained predictors (Section 3.2.2), and the end-to-end latencies are produced by LLMCompass, an externally validated simulator (Section 3.4). The 23% speedup claim is a simulation output under the Section 3.3 'Typical' error model; that model is an unvalidated assumption and the internal inconsistency between the text and Figure 5 for the pessimistic case is a correctness risk, but the claim is not equivalent to the model's inputs by construction. The only self-citation (Reference [5]) appears in related work and is not load-bearing. Therefore no circular step is exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a small number of modeling assumptions and fitted curve parameters. The free parameters are the coefficients of the exponential and polynomial curves used to map accuracy to overhead and performance; these are fitted to the paper's own measurements and not reported. The axioms are mostly domain assumptions about how error translates to runtime, how communication scales, and the validity of the i.i.d. multinomial model. No new physical entities are introduced.

free parameters (2)
  • overhead-vs-accuracy curve parameters = not reported
    Exponential fits to measured accuracy-overhead points (Section 3.2.2) are used in simulation to set Token-to-Expert overhead; the fitted coefficients are not given, so the performance comparison depends on these unstated parameters.
  • performance-vs-accuracy curve parameters = not reported
    Polynomial fits to accuracy-performance points are used to interpolate simulation results across skewness values; the coefficients are not reported.
assumptions (5)
  • domain assumption Token-to-expert selection is i.i.d. draws from a fixed multinomial per layer.
    Appendix A: used to justify MLE estimate p_i = n_i/N; ignores token content and positional dependencies that actually drive routing.
  • ad hoc to paper Prediction errors are uniformly distributed across GPUs, so the bottleneck GPU handles (1+epsilon)*average tokens.
    Section 3.3: the 'typical' scenario is chosen as the default without empirical support. This directly determines the runtime benefit of both prediction strategies.
  • domain assumption Communication runtime under skew scales as (N-1)*skewness/N^2 per all-to-all.
    Section 2: an analytic model of token movement; not validated against measured network behavior.
  • domain assumption Expert duplication via Algorithm 1 can reach a balanced placement within the analyzed memory and copy limits.
    Algorithm 1: assumes the iterative shifting terminates at per-GPU balance and that weight transfer can be hidden within Attention runtime.
  • domain assumption Attention runtimes can be overestimated without FlashAttention without changing the qualitative conclusions.
    Section 3.4 and Section 5: the simulator overestimates attention latency, which shifts the relative weight of FFN savings; the paper argues the relative overhead ratio is consistent with measurements, but no end-to-end validation is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing." pith.science (2026). https://pith.science/paper/LOC5Y6S5

@misc{pith2026250607366,
  author       = {Pith},
  title        = {Pith review of: MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOC5Y6S5}},
  note         = {Machine review of arXiv:2506.07366}
}
read the original abstract

In multi-GPU Mixture-of-Experts (MoE) network, experts are distributed across different GPUs, which creates load imbalance as each expert processes different number of tokens. Recent works improve MoE inference load balance by dynamically duplicating popular experts to more GPUs to process excessive tokens, which requires predicting the distribution before routing. In this paper, we discuss the tradeoff of prediction strategies, accuracies, overhead, and end-to-end system performance. We propose MoE-GPS, a framework that guides the selection of the optimal predictor design under various system configurations, by quantifying the performance impact to system-level model runtime. Specifically, we advocate for Distribution-Only Prediction, a prediction strategy that only predicts overall token distribution which significantly reduces overhead compared to the traditional Token-to-Expert Prediction. On Mixtral 8x7B MMLU dataset, MoE-GPS suggests Distribution-Only Prediction which improves end-to-end inference performance by more than 23% compared with Token-to-Expert Prediction.

Figures

Figures reproduced from arXiv: 2506.07366 by the authors.

Figure 1
Figure 1. MoE-GPS guidelines for selecting op￾timal expert prediction strategies that minimizes end-to-end inference latency based on model and hardware characteristics. Mixture-of-Experts (MoE) [13, 15, 7, 28] mod￾els reduce the computation of Large Language Models (LLMs) by activating only a subset of experts per token. In large multi-GPU datacen￾ters, Expert Parallelism (EP) [17] is typically used for Feed Forward Network … view at source ↗
Figure 2
Figure 2. Overview of a typical Mixture-of-Expert inference based on a Transformer network, with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Integration of expert duplication and different prediction strategies into the MoE model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Trade-off between prediction accuracy and end-to-end system performance for Token-to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Modeling the impact of prediction errors on end-to-end system runtime. Three scenarios for the same prediction error rate ϵ: (1) Optimistic—errors do not affect load balancing; (2) Typical—errors are evenly dis￾tributed across devices, leading to moderate slowdown; (3)…
Figure 6
Figure 6. Figure 6: Simulated prefill latency for a single layer of Mixtral 8 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Simulated effectiveness of two predic￾tion strategies’ best savings for Mixtral 8X7B on 4 fully-connected A100 with different system inter￾connect settings. To clearly draw our insights on which predic￾tion strategy is better under different scenar￾ios, we plotted the …
Figure 8
Figure 8. Figure 8: Simulated prefill latency for a single layer of Llama-MoE model [ [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Simulated prefill latency for a single layer of Switch Transformer model [ [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mixture of Raytraced Experts

    cs.LG 2025-07 conditional novelty 7.0 of 10

    A stacked Mixture-of-Experts model that dynamically selects ordered sequences of experts via a ray-like stochastic routing process, improving accuracy as more experts are activated.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  2. [2]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  3. [3]

    Prediction is all moe needs: Expert load distribution goes from fluctuating to stabilizing.arXiv preprint arXiv:2404.16914, 2024

    Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. Prediction is all moe needs: Expert load distribution goes from fluctuating to stabilizing.arXiv preprint arXiv:2404.16914, 2024

    arXiv 2024

  4. [4]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  5. [5]

    Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models.Proceedings of Machine Learning and Systems, 6:224–238, 2024

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Li, and Yiran Chen. Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models.Proceedings of Machine Learning and Systems, 6:224–238, 2024

    work page 2024

  6. [6]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    arXiv 2024

  7. [7]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  8. [8]

    Fastmoe: A fast mixture-of-expert training system.arXiv preprint arXiv:2103.13262, 2021

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system.arXiv preprint arXiv:2103.13262, 2021

    arXiv 2021

Show all 37 references
  1. [9]

    Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, ...

    2022

  2. [10]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  3. [11]

    Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    1997

  4. [12]

    Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference.arXiv preprint arXiv:2303.06182, 2023

    Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference.arXiv preprint arXiv:2303.06182, 2023

    2023 arXiv

  5. [13]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  6. [14]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  7. [15]

    Hierarchical mixtures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994

    1994

  8. [16]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  9. [17]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020. 11

    2006 arXiv

  10. [18]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  11. [19]

    Transmla: Multi-head latent attention is all you need.arXiv preprint arXiv:2502.07864, 2025

    Fanxu Meng, Zengwei Yao, and Muhan Zhang. Transmla: Multi-head latent attention is all you need.arXiv preprint arXiv:2502.07864, 2025

    2025 arXiv

  12. [20]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement.Proceedings of the ACM on Management of Data, 1(1):1–19, 2023

    Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement.Proceedings of the ACM on Management of Data, 1(1):1–19, 2023

    2023

  13. [21]

    Mixture of experts package

    NVIDIA. Mixture of experts package. https://docs.nvidia.com/megatron-core/ developer-guide/latest/api-guide/moe.html, 2025

    2025

  14. [22]

    Nvlink and nvlink switch

    NVIDIA. Nvlink and nvlink switch. https://www.nvidia.com/en-us/data-center/ nvlink/, 2025

    2025

  15. [23]

    Bandwidth optimal all-reduce algorithms for clusters of worksta- tions.Journal of Parallel and Distributed Computing, 69(2):117–124, 2009

    Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of worksta- tions.Journal of Parallel and Distributed Computing, 69(2):117–124, 2009

    2009

  16. [24]

    Pci express® base specification revision 4.0, version 1.0

    PCI-SIG. Pci express® base specification revision 4.0, version 1.0. Technical report, PCI Special Interest Group, 2017. Available at https://pcisig.com/specifications/pciexpress/

    2017

  17. [25]

    Chimera: Communication fusion for hybrid parallelism in large language models

    Le Qin, Junwei Cui, Weilin Cai, and Jiayi Huang. Chimera: Communication fusion for hybrid parallelism in large language models. InProceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25. ACM, 2025

    2025

  18. [26]

    Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale. InInternational conference on machine learni...

    2022

  19. [27]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  20. [28]

    Outrageously large neural networks: The sparsely-gated mixture-of- experts layer

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. InInternational Conference on Learning Representations, 2017

    2017

  21. [29]

    Se-moe: A scalable and efficient mixture-of- experts distributed training and inference system.arXiv preprint arXiv:2205.10034, 2022

    Liang Shen, Zhihua Wu, WeiBao Gong, Hongxiang Hao, Yangfan Bai, HuaChao Wu, Xinxuan Wu, Jiang Bian, Haoyi Xiong, Dianhai Yu, et al. Se-moe: A scalable and efficient mixture-of- experts distributed training and inference system.arXiv preprint arXiv:2205.10034, 2022

    2022 arXiv

  22. [30]

    Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  23. [31]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard,...

    2013

  24. [32]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  25. [33]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664, 2024

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts.arXiv preprint arXiv:2408.15664, 2024

    2024 arXiv

  26. [34]

    Prophet: Fine-grained load balancing for parallel training of large-scale moe models

    Wei Wang, Zhiquan Lai, Shengwei Li, Weijie Liu, Keshi Ge, Yujie Liu, Ao Shen, and Dongsheng Li. Prophet: Fine-grained load balancing for parallel training of large-scale moe models. In 2023 IEEE International Conference on Cluster Computing (CLUSTER), pages 82–94. IEEE, 2023. 12

    2023

  27. [35]

    Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023

    Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023

    2023 arXiv

  28. [36]

    Llmcompass: Enabling efficient hardware design for large language model inference

    Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, and David Wentzlaff. Llmcompass: Enabling efficient hardware design for large language model inference. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1080–1096. IEEE, 2024

    2024

  29. [37]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-training. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15913– 15923, 2...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.