Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

HAP argues that Mixture-of-Experts inference should use different parallel strategies for attention and expert modules, and for prefill versus decode, picked by an ILP solver that minimizes simulated end-to-end latency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

HAP uses ILP over module-specific hybrid parallel choices to speed up MoE inference, reporting up to 1.77x versus tensor parallelism on tested GPUs.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Phase-specific hybrid parallelism for MoE inference is a real and useful system contribution, but the unvalidated latency-simulator ranking and missing code/data keep the headline speedups from being fully supported yet. the 5 major comments →

arxiv 2508.19373 v1 pith:YQNSCRXN submitted 2025-08-26 cs.DC

HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference

Haoran Lin , Xianzhi Yu , Kang Zhao , Han Bao , Zongyuan Zhan , Ting Hu , Wulong Liu , Zekun Yin
show 2 more authors
Xin Li Weiguo Liu
This is my paper
classification cs.DC
keywords mixture-of-experts inferencehybrid parallelismadaptive parallelisminteger linear programminglatency simulationtensor parallelismexpert parallelismLLM serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mixture-of-Experts (MoE) inference is usually served with one static parallel strategy, typically tensor parallelism, applied uniformly to every layer and both inference phases. HAP argues that this is systematically suboptimal: attention computation, expert computation, and collective communication respond differently to parallelism depending on batch size, context length, and interconnect bandwidth. The paper's central proposal is to decompose each MoE layer into an Attention module and an Expert module, build lightweight latency simulators for each, then let an integer linear program pick the best hybrid strategy for prefill and decode separately, with a cheap transition between phases. If the reported results hold, the payoff is practical: up to 1.68x, 1.77x, and 1.57x speedups over a tensor-parallel baseline on A100, A6000, and V100 nodes, with no change to the model itself. The authors also claim the search overhead is under a second and is included in all reported latencies.

Core claim

The paper establishes that the optimal parallel configuration for MoE inference is not a single strategy but a phase- and module-dependent combination. Its central discovery is that, by splitting the MoE layer into attention and expert modules and simulating each module's compute and communication latency (a FLOP-count model corrected by a random forest, and a bandwidth model similarly corrected), the search over hybrid parallel configurations can be formulated as an integer linear program whose objective is predicted end-to-end latency including the cost of switching strategies between prefill and decoding. The experiments then show HAP-chosen configurations match or beat the standard tenso

What carries the argument

The central mechanism is HAP's two-level decomposition plus ILP search: (1) module-specific latency estimators T_cal = (FLOPs / peak FLOPs/s) * eta and T_comm = (data volume / bandwidth) * rho, where eta and rho are fitted with a random-forest regression on benchmarked operation latencies; (2) a hierarchical search space over Attention strategies (DP, TP, or DP+TP) and Expert strategies (EP, TP, or EP+TP); and (3) an ILP objective that minimizes N_layer times [prefill cost + S_output times decode cost] plus a switching-cost term E_i^T C_ij E_j, with divisibility and memory constraints. The switching cost is itself minimized by comparing direct AllGather/AllToAll weight redistribution against

Load-bearing premise

The load-bearing premise is that a layer's total latency is the sum of independently simulated attention, expert, and communication latencies; if memory-bandwidth contention, kernel overlap, or hardware-specific coupling breaks that additivity, the configuration that minimizes the simulated sum will not minimize real end-to-end latency.

What would settle it

Run HAP and a full factorial sweep of all feasible hybrid configurations on a new GPU, batch-size, and context-length combination and compare actual end-to-end latencies: if the ILP-selected configuration is beaten by another configuration by more than the simulation model's reported error margin, the additive latency decomposition or the fitted simulators are not sufficient. A second check: measure the transition between prefill and decode strategies directly; if asynchronous CPU-side dequantization does not overlap with the first decode layers and the switching cost exceeds the predicted C_i

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single inference system can switch its expert-module parallelism between prefill and decode, getting EP-style prefill latency and TP-style decode latency with small transition overhead.
  • In communication-bound workloads, choosing low-communication strategies for both modules is what produces the largest reported wins (1.57x to 1.77x).
  • The search itself costs less than a second for 4- and 8-GPU nodes, and because that time is included in reported end-to-end latency, the optimizer does not have to be amortized over long runs.
  • The same solver can handle MoE designs with few large experts or many small or shared experts by changing only the module latency inputs, which is why the reported gains appear across different MoE architecture families.
  • When decoding dominates, the solver picks tensor parallelism, so HAP should not be much worse than the TP baseline even in regimes where TP is already optimal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same module-decomposition plus ILP search could be applied to other sparse or hybrid architectures, such as models with grouped-query attention and shared experts, by swapping in new per-module simulators.
  • The prefill/decode split is a coarse two-phase approximation; continuous batching and chunked prefill introduce more than two phases, and the framework could be extended to search over a schedule of configurations rather than one transition.
  • The quantized-CPU-backup transition is a special case of hiding configuration changes behind computation; one could generalize it to online model re-sharding or to swapping experts in and out during serving.
  • If the random-forest corrections are recalibrated on new hardware, the approach should transfer to other interconnects and GPU generations, but transferability across batch sizes and hardware is an empirical claim the paper tests only on three platforms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper presents HAP, a system that automatically selects per-module hybrid parallel strategies for MoE LLM inference. It decomposes a transformer layer into Attention and Expert modules, estimates per-module compute and communication latencies using FLOP/bandwidth formulas corrected by random-forest regressors, enumerates DP/TP for the Attention module and EP/TP for the Expert module, and solves an ILP to choose prefill and decode strategies, including a transition-cost model. A quantized CPU-side backup of expert weights is used to reduce the cost of changing the Expert module's parallelization between phases. Experiments with DeepSpeed-FastGen on Mixtral-8x7B, Qwen1.5-MoE-A2.7B, and Qwen2-57B-A14B across A6000, A100, and V100 platforms report speedups over a TP baseline of up to 1.77x, strongest in long-context / constrained-output scenarios.

Significance. If the search and latency simulation are faithful, HAP is a practical, scenario-adaptive alternative to static TP/EP for MoE inference, with modest search overhead (ILP solved in <1 s, and that time is included in the reported end-to-end latencies) and a concrete strategy-transition mechanism. The evaluation is broad, spans heterogeneous interconnects and model families, and honestly reports configurations where TP remains preferable. The main risk is that the central 'optimal or near-optimal strategy' claim rests on simulator ranking fidelity, which is not directly validated. The contribution is therefore conditional on additional validation rather than fundamentally unsound.

major comments (5)
  1. [Section III-C, Eq. (4)-(5)] The ILP as written is not a well-defined mathematical program: S_k and E_i are one-hot vectors, while E_t, E_d, E_e are scalar degrees; the objective uses T_C^{ki} and T_C^{kj} without definition; and the constraint N = A_t × A_d = E_d × E_t × E_e is ambiguous because E_d appears both as a scalar degree and, via E_i, in the one-hot strategy notation. Please rewrite with consistent notation for scalar degrees and strategy indicators, and define T_C(i,j) explicitly.
  2. [Section IV-B vs. Section III-B] Component-level relative errors of 5-10% (Fig. 5) do not establish that the additive sum in Eqs. (1)-(3) preserves the true ranking of candidate configurations. A systematic bias in, say, the AllReduce model would systematically favor communication-light hybrids and inflate the reported speedups. Please add an exhaustive-search ranking check for at least one model/GPU/scenario: enumerate every feasible (Attention, Expert) strategy pair, compare measured vs. predicted end-to-end latency, and report rank correlation and the gap between HAP's selected latency and the measured best. Also test the additivity assumption by measuring whether isolated module latencies combine without overlap or memory-bandwidth contention.
  3. [Section III-A2, Eq. (5)] The '2× baseline TP activation footprint' upper bound for EP activation memory is an unvalidated ad-hoc multiplier. It enters the memory constraint and can change which configurations are feasible, and thus the ILP optimum. Provide empirical activation-memory measurements under EP with realistic load imbalance, or a sensitivity analysis showing that the ILP solution is invariant for a plausible range of this multiplier (e.g., 1.0× to 2.5×).
  4. [Section III-C] The pruning of DP+EP+TP combinations for the Expert module is asserted to be 'generally suboptimal' based on 'existing experience' but no evidence is given. Because the paper claims to search for the optimal configuration, this pruning needs justification, e.g., a benchmark subset showing that pruned configurations are always dominated by unpruned ones. Otherwise the claim should be restricted to the unpruned search space.
  5. [Section IV] The empirical section reports no run-to-run variance or error bars, and many headline claims are 'up to' maxima over batch sizes. Report mean ± std over multiple repeated runs for representative configurations and report median (typical) speedups in addition to maxima. Also reconcile the abstract platform ordering (1.68×/1.77× on A100/A6000) with Section IV-C3, where 1.68× is reported on A6000 and 1.77× on A100.
minor comments (4)
  1. [Throughout] Typos: 'implys' in Section III-B; 'are are' in Section III-C; 'dopted' in Section III-D. Please proofread.
  2. [Section IV-B] Specify how many samples were used to train/test the random-forest correctors and whether the test set is disjoint from the configurations and batch sizes used in the end-to-end experiments.
  3. [Eq. (6)] The dequantization-latency dictionary V_dequant → T_dequant is introduced but its construction and measurement are not described. Add a sentence on how this dictionary was obtained and how sensitive C_ij is to its accuracy.
  4. [Section IV-A] The comparison baseline is TP only. Since DeepEP and other EP-based systems are cited as related work, a direct comparison against a state-of-the-art EP or static-hybrid baseline would strengthen the positioning, even if only for the long-context constrained-output scenario.

Circularity Check

0 steps flagged

No significant circularity: final speedups are measured end-to-end; simulator calibration and ILP search are not equivalent to the reported outcomes.

full rationale

The central reported results are actual end-to-end latencies: HAP vs TP comparisons in Figs. 4-9 are measured, not derived from the simulator. The ILP in Eq. (4) minimizes an objective built from the calibrated simulation models of Sec. III-B, where eta and rho are random-forest corrections fitted to benchmarked operator runtimes. This is calibration of a search heuristic, not a fitted input renamed as a prediction of the headline speedups. The final speedup numbers are independently measured on real inference runs, so the claim 'HAP achieves 1.68x/1.77x/1.57x speedup' does not reduce by construction to the fitted model. The additive latency decomposition (Eqs. 1-3) and the lack of an exhaustive ranking-fidelity check are legitimate soundness/generalization concerns: a systematically biased simulator could misrank hybrid configurations. However, misranking is an accuracy limitation, not circularity, because the paper's performance claims are validated by measured runs rather than by the simulator's own predictions. No load-bearing self-citation chain or imported uniqueness theorem was found; references to frameworks and prior systems are used as background or baselines, not as the sole justification for HAP's effectiveness. The reported component-error numbers are fitting errors on the same platform, which limits external validity but does not make the derivation circular.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The approach rests on two calibrated regression corrections and several hand-set engineering choices: the additive module latency model, the 2x EP activation memory bound, the pruning of DP+EP+TP, and a runtime dequantization table. None of these are independently verified beyond the paper's own small validation set.

free parameters (5)
  • eta (compute latency correction factor) = trained random forest regression
    Multiplies Fmodule/MaxFLOPS in Section III-B; fitted to measured operator runtimes with polynomial-expanded features of batch size, sequence length, and hidden size.
  • rho (communication latency correction factor) = trained random forest regression
    Multiplies Vdata/Bandwidth in Section III-B; fitted to measured collective communication latency using data volume and device bandwidth.
  • EP activation memory multiplier = 2
    Section III-A2 doubles the baseline TP activation footprint as a conservative bound for EP; no measurement supports the factor.
  • dequantization latency dictionary (Vdequant to Tdequant) = runtime lookup table
    Section III-D builds a dictionary of measured dequantization times queried during transition cost estimation; treated as an empirical calibration table.
  • random forest hyperparameters and polynomial feature expansion degree = unspecified
    The regression model is described only as an efficient random forest with polynomial feature expansion; exact settings are not given, so an independent replication cannot reproduce them.
axioms (5)
  • domain assumption MoE layer latency decomposes additively as T_attn + T_expert + T_comm per layer
    Eqs. 1 to 3 define total latency as a sum of module and communication latencies, ignoring overlap, memory bandwidth contention, and kernel launch overhead.
  • domain assumption Compute and communication times are proportional to FLOPs/peak_FLOPS and data volume/bandwidth, with correction factors eta and rho learned by random forest
    Section III-B builds the ILP objective on these calibrated models; if the corrections are not transferable across configurations, the search is biased.
  • domain assumption Attention module and Expert module can be optimized independently and their optimal strategies combined
    Section III-C assumes the search space is the Cartesian product of module strategies with additive T_comm; cross-module interactions are not modeled.
  • ad hoc to paper EP activation memory can be upper-bounded by 2x the TP activation footprint
    Section III-A2 asserts this empirical upper bound to handle EP load imbalance, with no measurement justifying the factor.
  • ad hoc to paper DP+EP+TP combinations for the Expert module are generally suboptimal and can be pruned
    Section III-C removes these from the search space based on existing experience, without experiments or citations substantiating the claim.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference." pith.science (2026). https://pith.science/paper/YQNSCRXN

@misc{pith2026250819373,
  author       = {Pith},
  title        = {Pith review of: HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQNSCRXN}},
  note         = {Machine review of arXiv:2508.19373}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Current inference systems for Mixture-of-Experts (MoE) models primarily employ static parallelization strategies. However, these static approaches cannot consistently achieve optimal performance across different inference scenarios, as they lack the flexibility to adapt to varying computational requirements. In this work, we propose HAP (Hybrid Adaptive Parallelism), a novel method that dynamically selects hybrid parallel strategies to enhance MoE inference efficiency. The fundamental innovation of HAP lies in hierarchically decomposing MoE architectures into two distinct computational modules: the Attention module and the Expert module, each augmented with a specialized inference latency simulation model. This decomposition promotes the construction of a comprehensive search space for seeking model parallel strategies. By leveraging Integer Linear Programming (ILP), HAP could solve the optimal hybrid parallel configurations to maximize inference efficiency under varying computational constraints. Our experiments demonstrate that HAP consistently determines parallel configurations that achieve comparable or superior performance to the TP strategy prevalent in mainstream inference systems. Compared to the TP-based inference, HAP-based inference achieves speedups of 1.68x, 1.77x, and 1.57x on A100, A6000, and V100 GPU platforms, respectively. Furthermore, HAP showcases remarkable generalization capability, maintaining performance effectiveness across diverse MoE model configurations, including Mixtral and Qwen series models.

Figures

Figures reproduced from arXiv: 2508.19373 by Han Bao, Haoran Lin, Kang Zhao, Ting Hu, Weiguo Liu, Wulong Liu, Xianzhi Yu, Xin Li, Zekun Yin, Zongyuan Zhan.

Figure 1
Figure 1. Figure 1: (a) The typical dense transformer layer consists of an [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The per-layer latency breakdown during both prefill and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The workflow of parallelism transition during the prefill [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The performance comparison conducted under a scenario featuring a 256-token context and 64-token generation between [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparative evaluation of numerical accuracy in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The performance comparison conducted under a scenario featuring a 256-token context and 2048-token generation [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The performance comparison conducted under a scenario featuring a 4096-token context and 64-token generation between [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The performance evaluation using Mixtral-8x7B MoE [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The performance comparison conducted under a scenario featuring a 4096-token context and 2048-token generation [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch

    cs.DC 2026-06 unverdicted novelty 7.0

    Moebius enables runtime switching between EP and TP for MoE LLMs by resharing weights and KV cache, matching the best static choice and improving RL rollouts by 1.16-1.25x with 215-434 ms switches and 2.4% memory overhead.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Efficient large scale language modeling with mixtures of experts,

    M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V . Lin, J. Du, S. Iyer, R. Pasunuru, et al. , “Efficient large scale language modeling with mixtures of experts,” arXiv preprint arXiv:2112.10684 , 2021

    Pith/arXiv arXiv 2021

  2. [2]

    No language left behind: Scaling human-centered machine translation,

    M. R. Costa-Juss `a, J. Cross, O. C ¸ elebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al. , “No language left behind: Scaling human-centered machine translation,” arXiv preprint arXiv:2207.04672 , 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research , vol. 23, no. 120, pp. 1–39, 2022

    2022

  4. [4]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Scaling vision with sparse mixture of experts,

    C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and N. Houlsby, “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583–8595, 2021

    work page 2021

  6. [6]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    work page 2023

  7. [7]

    Deepspeed- fastgen: High-throughput text generation for llms via mii and deepspeed- inference,

    C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhandari, R. Y . Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko,et al., “Deepspeed- fastgen: High-throughput text generation for llms via mii and deepspeed- inference,” arXiv preprint arXiv:2401.08671 , 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Moesys: A distributed and efficient mixture-of-experts training and inference system for internet services,

    D. Yu, L. Shen, H. Hao, W. Gong, H. Wu, J. Bian, L. Dai, and H. Xiong, “Moesys: A distributed and efficient mixture-of-experts training and inference system for internet services,” IEEE Transactions on Services Computing, vol. 17, no. 5, pp. 2626–2639, 2024

    work page 2024

  9. [9]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668 , 2020

    Pith/arXiv arXiv 2006

  10. [10]

    Designing effective sparse expert models,

    B. Zoph, I. Bello, S. Kumar, N. Du, Y . Huang, J. Dean, N. Shazeer, and W. Fedus, “Designing effective sparse expert models,” arXiv preprint arXiv:2202.08906, vol. 2, no. 3, p. 17, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. , “Mixtral of experts,” arXiv preprint arXiv:2401.04088 , 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters

    Q. Team, “Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters”,” February 2024

    work page 2024

  13. [13]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. , “Qwen2 technical report,” 2024

    work page 2024

  14. [14]

    A tensorrt toolbox for optimized large language model inference

    NVIDIA, “A tensorrt toolbox for optimized large language model inference.” https://github.com/NVIDIA/TensorRT-LLM, 2024

    work page 2024

  15. [15]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, et al. , “Sglang: Efficient execution of structured language model programs,” Advances in Neural Information Processing Systems , vol. 37, pp. 62557–62583, 2024

    work page 2024

  16. [16]

    Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference,

    Q. Li, B. Zhang, L. Ye, Y . Zhang, W. Wu, Y . Sun, L. Ma, and Y . Xie, “Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference,” arXiv preprint arXiv:2412.04964 , 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference,

    H. Huang, N. Ardalani, A. Sun, L. Ke, H.-H. S. Lee, A. Sridhar, S. Bhosale, C.-J. Wu, and B. Lee, “Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference,” arXiv preprint arXiv:2303.06182, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Deepspeed-moe: Advancing mixture- of-experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture- of-experts inference and training to power next-generation ai scale,” in International conference on machine learning , pp. 18332–18346, PMLR, 2022

    work page 2022

  19. [19]

    A hybrid tensor-expert-data parallelism approach to optimize mixture- of-experts training,

    S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y . He, and A. Bhatele, “A hybrid tensor-expert-data parallelism approach to optimize mixture- of-experts training,” in Proceedings of the 37th International Conference on Supercomputing, pp. 203–214, 2023

    work page 2023

  20. [20]

    Deepep: an efficient expert-parallel communication library

    C. Zhao, S. Zhou, L. Zhang, C. Deng, Z. Xu, Y . Liu, K. Yu, J. Li, and L. Zhao, “Deepep: an efficient expert-parallel communication library.” https://github.com/deepseek-ai/DeepEP, 2025

    work page 2025

  21. [21]

    Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y . Zhuang, Z. Chen, Y . Huang, Y . Wang, Y . Xu, D. Zhuo, E. P. Xing, et al. , “Alpa: Automating inter-and {Intra- Operator} parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 559–578, 2022

    work page 2022

  22. [22]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram, et al. , “Tutel: Adaptive mixture-of-experts at scale,” Proceedings of Machine Learning and Systems , vol. 5, pp. 269–287, 2023

    work page 2023

  23. [23]

    Pcie gen-5 design challenges of high-speed servers,

    M. Vasa, C.-L. Liao, S. Kumar, C.-H. Chen, and B. Mutnury, “Pcie gen-5 design challenges of high-speed servers,” in 2020 IEEE 29th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS) , 2020

    work page 2020

  24. [24]

    Nvidia a100 gpu: Performance & innova- tion for gpu computing,

    J. Choquette and W. Gandhi, “Nvidia a100 gpu: Performance & innova- tion for gpu computing,” in 2020 IEEE Hot Chips 32 Symposium (HCS) , pp. 1–43, IEEE Computer Society, 2020

    work page 2020

  25. [25]

    Bitsandbytes: a lightweight python wrapper around cuda custom func- tions

    “Bitsandbytes: a lightweight python wrapper around cuda custom func- tions.” https://github.com/bitsandbytes-foundation/bitsandbytes, 2024

    work page 2024

  26. [26]

    A white paper on neural network quantization,

    M. Nagel, M. Fournarakis, R. A. Amjad, Y . Bondarenko, M. Van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295 , 2021

    Pith/arXiv arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.