Pith. sign in

REVIEW 2 major objections 4 minor 47 references

Director cuts distributed MoE serving latency 11-55% by predicting upcoming expert activations and proactively re-placing experts before the next batch runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 07:32 UTC pith:MC2IGGYO

load-bearing objection Clean systems paper that actually delivers online-proactive MoE placement with real 11-55% latency cuts and a proper (1+ε) algorithm; predictor robustness under wilder shifts is the main open question. the 2 major comments →

arxiv 2607.08782 v1 pith:MC2IGGYO submitted 2026-06-13 cs.LG cs.AI

Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement

Qianli Liu , Kaibin Guo , Zicong Hong , Peng Li , Fahao Chen , Haodong Wang , Jian Lin , Song Guo This is my paper
classification cs.LG cs.AI
keywords Mixture-of-Expertsexpert parallelismexpert placementonline reconfigurationrouting predictionlive migrationlatency optimizationapproximation algorithm
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mixture-of-Experts models are served by scattering experts across GPUs, but the resulting all-to-all communication and load imbalance become bottlenecks when request patterns change. Existing placement methods either fix a layout from history or react after a batch has already been seen, so they lag behind shifting traffic. Director instead peeks at the request queue, forecasts which experts the next tokens will activate, then migrates those experts onto the right GPUs before the tokens arrive. Migrations ride inside compute-bound phases so they add almost no downtime, and a polynomial-time optimizer finds a placement whose latency is within a (1+ε) factor of the best possible under capacity limits. On Mixtral, DeepSeek and Qwen models the system therefore finishes requests 11-55% faster than both static and reactive baselines, showing that anticipation can remove the lag that has limited online expert management.

Core claim

A distributed MoE serving system that couples routing prediction of queued requests with a relaxation-based placement optimizer and compute-overlapped live migration can keep expert placement continuously aligned with incoming traffic, reducing end-to-end latency by 11-55% relative to offline and online-reactive methods while preserving a polynomial-time (1+ε) approximation guarantee under GPU capacity constraints.

What carries the argument

The relaxation-based expert placement optimizer: it formulates latency minimization as a feasibility LP over a binary-search budget, then uses iterative randomized rounding that preserves feasibility and yields a (1+ε)-approximate integer placement in polynomial time.

Load-bearing premise

The cascade or 4-bit quantized predictor must stay accurate enough that the optimizer still produces near-optimal placements even when real multi-user expert activations shift faster than the prediction window.

What would settle it

Run the same multi-task workload (math, code, wikitext) while deliberately degrading predictor top-k match rate below the reported 77% floor; if the latency advantage over the reactive baseline disappears or reverses, the proactive claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Serving systems can reconfigure expert layouts on every batch without stopping inference, because migrations hide inside attention/FFN compute windows.
  • Larger GPU counts amplify the gain: a single poorly placed expert creates a straggler that stalls more devices, so proactive placement becomes more valuable as clusters scale.
  • Operators obtain a tunable accuracy-overhead trade-off by choosing the lightweight cascade predictor for frequent tweaks or the quantized replica for high-fidelity major rearrangements.
  • The same prediction-plus-relaxed-optimizer pattern can be applied to other sparse or conditional computation layouts beyond pure expert parallelism.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If routing predictors continue to improve with model-scale quantization, the remaining 1-8% gap to oracle placement could shrink further, making proactive methods the default rather than a specialized option.
  • The same queue-peeking idea may transfer to sequence-parallel or tensor-parallel rebalancing whenever future tokens can be partially forecasted.
  • Because migration cost is bounded by the length of compute phases, hardware with faster interconnects will still benefit mainly from better load balance rather than from reduced communication volume.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces DIRECTOR, a distributed MoE serving system that performs online proactive expert placement. It predicts token-to-expert routing for queued requests via either a cascade of single-layer transformers or a 4-bit quantized MoE replica, solves a capacity-constrained placement problem with an LP-relaxation + iterative-rounding algorithm that runs in polynomial time and returns a (1+ε)-approximate solution relative to the LP optimum under the predicted workload matrices, and enacts the resulting migrations by overlapping them with compute-bound phases so that downtime is near zero. Extensive experiments on Mixtral-8×7B, DeepSeekMoE-16B, DeepSeek-V2-Lite and Qwen3-30B-A3B across two clusters (up to 32 H200 GPUs) report 11–55 % end-to-end latency reductions versus static offline and online-reactive baselines, with ablations on predictor fidelity and optimizer quality.

Significance. If the end-to-end gains hold under realistic multi-user traffic, the work supplies a practically useful and theoretically grounded advance for large-scale MoE serving: a clean (1+ε) guarantee (Theorems 1–3) for an NP-hard placement problem, a concrete live-migration schedule that hides transfer cost, and a working prototype evaluated on contemporary fine-grained MoE models. The combination of prediction-driven reconfiguration with a polynomial-time approximation algorithm is a clear step beyond purely reactive or offline expert-placement schemes and is therefore of interest to both systems and theory audiences in the MoE literature.

major comments (2)
  1. [§V-A, Table II, Fig. 10, Theorem 2] Theorem 2 guarantees T(P) ≤ (1+ε)β★ only with respect to the LP optimum of the predicted workload matrices W. The realized latency is governed by the true (unknown) routing. Finding 1 and Fig. 2 document rapid, batch-to-batch shifts in expert selection; the only quantitative stress of prediction error is the 1–8 % gap versus an oracle on the same mixed MATH/Wiki/Code traces used for training (Fig. 10). That ablation does not re-create the distribution shifts of Fig. 2, so it remains unclear whether the measured 11–55 % gains survive when top-k match falls below the 77–96 % range of Table II. A controlled experiment that injects stronger temporal non-stationarity (or reports the realized latency gap under the exact traces of Fig. 2) is needed to substantiate the central proactive claim.
  2. [§VI-A, Eqs. (2),(4); Fig. 6c] The latency model (Eqs. 2 and 4) treats f_comp and f_comm as fixed, offline-profiled functions of load and data volume. Under live migration the communication fabric is shared with all-to-all traffic, and the compute-overlapped schedule of Fig. 6c can still produce contention when migration windows are mis-estimated. No sensitivity analysis or online re-profiling of these functions appears in §VIII; without it the (1+ε) guarantee and the reported speed-ups rest on an idealized cost model that may not hold once migrations are frequent.
minor comments (4)
  1. [§VIII-C, Figs. 7,9] Fig. 7 and Fig. 9 report only mean normalized latency; error bars or confidence intervals over the five random seeds mentioned in §VIII-B would strengthen the statistical claim.
  2. [§VI-C] Notation for the placement map P(l,i) and the indicator I[·] is introduced cleanly, but the multi-index v used later for variables x_v is never formally defined; a short clarifying sentence would help.
  3. [§V-A, Table II] The cascade predictor depth n is listed as a free parameter yet never varied in the experiments; a one-sentence statement of the depth used for Table II would improve reproducibility.
  4. [Abstract, §I] Typographical inconsistency: the system is called both DIRECTOR and DIRECTORuses in the abstract and introduction.

Circularity Check

0 steps flagged

No significant circularity found; approximation guarantee is relative to the LP of the (predicted) instance, predictors are evaluated on held-out accuracy and end-to-end latency against external baselines, and the derivation chain does not reduce by construction to its inputs.

full rationale

The paper formulates an NP-hard expert-placement problem (Eq. 5) whose objective is the sum of profiled computation and communication latencies T_comp(P) + T_comm(P) under a predicted workload matrix W (Eqs. 1-4). Algorithm 1 solves an LP relaxation of a feasibility version of that instance and applies iterative rounding; Theorems 1-3 prove a (1+ε) guarantee only relative to the LP optimum β★ of the same predicted instance, using standard external tools (Freedman martingale concentration, potential-function arguments from Bansal et al.). The predictors (cascade or 4-bit quantized replica) are trained on historical gate logits and separately measured by top-k match accuracy (Table II) and by the end-to-end latency gap versus an oracle (Fig. 10); the system-level claims are then measured against independent baselines (Vanilla, Offline, Reactive) on held-out multi-task traces. No step equates a fitted parameter to a claimed prediction of the same quantity, no uniqueness theorem is imported from overlapping authors to force the design, and no ansatz is smuggled via self-citation. The derivation is therefore self-contained against external benchmarks; any residual risk (prediction error under rapid distribution shift) is an empirical assumption, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on standard combinatorial-optimization machinery, domain assumptions about MoE routing and GPU memory, and a small set of free algorithmic parameters (epsilon, cascade depth, quantization bit-width, migration window estimates). No new physical entities are postulated; the predictors and the placement polytope are engineering constructs with independent empirical handles.

free parameters (4)
  • approximation tolerance ε
    Controls the binary-search termination and the step-size γ of the random walk; chosen by the implementer and directly trades solution quality for runtime.
  • cascade depth / number of predictor stages n
    Hand-tuned trade-off between prediction accuracy and overhead; reported as two-stage in the main experiments.
  • quantization bit-width (INT4)
    Chosen for the high-fidelity predictor; accuracy and size numbers in Table II depend on this choice.
  • C_max (experts per GPU per layer)
    Hard capacity constraint derived from GPU memory; treated as an input but its concrete value is cluster- and model-specific.
axioms (5)
  • standard math Expert placement under capacity and latency objectives is NP-hard (reducible to Generalized Assignment Problem).
    Invoked in §I and §VI-B to justify the need for approximation; standard combinatorial result.
  • domain assumption Computation and communication latency functions f_comp(λ) and f_comm(D) can be accurately profiled offline and remain stable under the evaluated workloads.
    Used throughout the LP formulation (§VI-A) and the migration scheduler; if the functions drift, both the optimizer and the overlap schedule become incorrect.
  • domain assumption Routing decisions of deeper MoE layers are sufficiently predictable from token embeddings plus earlier predicted routes (cascade) or from a low-bit replica.
    Core premise of §V-A; without it the proactive placement has no useful input.
  • domain assumption All-to-all communication and expert-parameter migration contend for the same network bandwidth, so they must not be scheduled concurrently.
    Justifies the compute-overlapped migration design in §V-B and Fig. 6.
  • standard math The iterative-rounding random walk on the scaled polytope yields a constant-factor reduction in fractional variables per macro-iteration with high probability (Lemmas 1.1-1.3).
    Relies on martingale concentration and discrepancy-rounding arguments from the cited literature; treated as established technique.
invented entities (2)
  • Cascade predictor (stacked single-layer transformers conditioned on prior routing) independent evidence
    purpose: Low-overhead forecast of layer-wise expert selections for pending requests.
    Engineering construct; independent evidence is the reported top-k accuracy on held-out batches, but the architecture itself is new to this paper.
  • Relaxation-based expert placement polytope with scale-preserving constraints no independent evidence
    purpose: Enable polynomial-time (1+ε) approximation of the NP-hard placement problem.
    Mathematical object constructed for the algorithm; its correctness is proved inside the paper relative to the LP relaxation.

pith-pipeline@v1.1.0-grok45 · 21164 in / 3323 out tokens · 26766 ms · 2026-07-13T07:32:18.666696+00:00 · methodology

0 comments
read the original abstract

Expert parallelism has become the prevailing paradigm to serve Mixture-of-Experts (MoE) models. Its efficiency depends on the communication and computation latencies of the GPUs, which are linked to the placement of experts in the GPUs. Existing works for optimizing expert placement focus on leveraging past requests' expert activation patterns. However, they demonstrate deficiencies facing diverse and rapidly changing request patterns, calling for an online, proactive approach. Implementing such an approach requires addressing several challenges: the uncertainty associated with incoming requests' expert activation, the cost of expert migration, and the NP-hard complexity in optimization. Therefore, we present Director, a new distributed MoE serving system that minimizes end-to-end latency via prediction-driven, online expert placement. Director uses either a lightweight cascaded predictor or a low-bit quantized replica for expert activation patterns of incoming requests. An online migration module then enacts the changes with near-zero downtime by executing migrations in compute-bound phases, keeping disruption bounded. At its core, a relaxation-based expert placement optimizer operates under capacity constraints, runs in polynomial time, and achieves a $(1+\epsilon)$ approximation ratio. Finally, we implement a prototype and demonstrate, through extensive experiments, a reduction in end-to-end latency of $11\sim55\%$ for popular MoE models (e.g., Mistral, DeepSeek and Qwen) compared to existing work.

Figures

Figures reproduced from arXiv: 2607.08782 by Fahao Chen, Haodong Wang, Jian Lin, Kaibin Guo, Peng Li, Qianli Liu, Song Guo, Zicong Hong.

Figure 1
Figure 1. Figure 1: A comparison of offline expert placement, online reac [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The maximum reduction in latency achieved by the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Two options of predictors in DIRECTOR V. ONLINE EXPERT PLACEMENT RECONFIGURATION MECHANISM As mentioned in § I and § III, to maintain an effective placement, the first two challenges are: how to predict the expert routing for pending requests, and how to make expert migrations without interrupting the normal serving process. Therefore, we introduce an online expert placement reconfig￾uration mechanism. As … view at source ↗
Figure 6
Figure 6. Figure 6: Execution timelines of serving and reconfiguration. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: End-to-end normalized latency of DIRECTOR against baselines on different models and clusters. H200 GPUs; GPUs are connected by NVLINK within a node and by InfiniBand across nodes. Models and Datasets. We evaluate DIRECTOR on four representative MoE models: Mistral 8×7B [46], DeepSeekMoE-16B [5], DeepSeek-V2-Lite [6], and Qwen3-30B-A3B [4]. Selected models cover a wide spectrum of configurations: the number… view at source ↗
Figure 10
Figure 10. Figure 10: Performance under different predictor options. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 11 linked inside Pith

  1. [1]

    The llama 4 herd: Native multimodality with a mixture-of-experts architecture,

    Meta AI Team, “The llama 4 herd: Native multimodality with a mixture-of-experts architecture,” https://ai.meta.com/blog/ llama-4-multimodal-intelligence/, April 2025

    2025

  2. [2]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huanget al., “Qwen technical report,”arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Qwen2 technical report,

    Qwen Team, “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wuet al., “Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models,” inACL, 2024

    2024

  6. [6]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,

    A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guoet al., “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,”arXiv preprint arXiv:2405.04434, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,”arXiv preprint arXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  9. [9]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, 2022

    2022

  10. [10]

    Training and serving system of foundation models: A comprehensive survey,

    J. Zhou, Y . Chen, Z. Hong, W. Chen, Y . Yu, T. Zhang, H. Wang, C. Zhang, and Z. Zheng, “Training and serving system of foundation models: A comprehensive survey,”IEEE Open Journal of the Computer Society, 2024

    2024

  11. [11]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,”arXiv preprint arXiv:2006.16668, 2020

    Pith/arXiv arXiv 2006

  12. [12]

    Expert Parallelism Load Balancer,

    Deepseek Team, “Expert Parallelism Load Balancer,” 2025. [Online]. Available: https://github.com/deepseek-ai/EPLB

    2025

  13. [13]

    Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing,

    S. Go and D. Mahajan, “Moetuner: Optimized mixture of expert serving with balanced expert placement and token routing,”arXiv preprint arXiv:2502.06643, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,

    J. He, J. Zhai, T. Antunes, H. Wang, F. Luo, S. Shi, and Q. Li, “Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,” inPPoPP, 2022

    2022

  15. [15]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,

    X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,” inSIGMOD, 2023

    2023

  16. [16]

    Tutel: Adaptive mixture-of-experts at scale,

    C. Hwang, W. Cui, Y . Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ramet al., “Tutel: Adaptive mixture-of-experts at scale,” in MLSys, 2023

    2023

  17. [17]

    Smartmoe: Efficiently training sparsely-activated models through combining offline and online parallelization,

    M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “Smartmoe: Efficiently training sparsely-activated models through combining offline and online parallelization,” inATC, 2023

    2023

  18. [18]

    Janus: A unified distributed training framework for sparse mixture-of-experts models,

    J. Liu, J. H. Wang, and Y . Jiang, “Janus: A unified distributed training framework for sparse mixture-of-experts models,” inSIGCOMM, 2023

    2023

  19. [19]

    A branch-and-price algorithm for the generalized assignment problem,

    M. Savelsbergh, “A branch-and-price algorithm for the generalized assignment problem,”Operations research, 1997

    1997

  20. [20]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” inICLR, 2017

    2017

  21. [21]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,

    X. Lu, Q. Liu, Y . Xu, A. Zhou, S. Huang, B. Zhang, J. Yan, and H. Li, “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” inACL, Aug. 2024

    2024

  22. [22]

    Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs,

    E. Liu, J. Zhu, Z. Lin, X. Ning, M. B. Blaschko, S. Yan, G. Dai, H. Yang, and Y . Wang, “Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs,” arXiv preprint arXiv:2407.00945, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Ta-moe: topology-aware large scale mixture-of-expert training,

    C. Chen, M. Li, Z. Wu, D. Yu, and C. Yang, “Ta-moe: topology-aware large scale mixture-of-expert training,” inNIPS, 2022

    2022

  24. [24]

    Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,

    S. Shi, X. Pan, Q. Wang, C. Liu, X. Ren, Z. Hu, Y . Yang, B. Li, and X. Chu, “Schemoe: An extensible mixture-of-experts distributed training system with tasks scheduling,” inEuroSys, 2024

    2024

  25. [25]

    Fsmoe: A flexible and scalable training system for sparse mixture-of- experts models,

    X. Pan, W. Lin, L. Zhang, S. Shi, Z. Tang, R. Wang, B. Li, and X. Chu, “Fsmoe: A flexible and scalable training system for sparse mixture-of- experts models,” inASPLOS, 2025

    2025

  26. [26]

    Pipemoe: Accelerating mixture-of- experts through adaptive pipelining,

    S. Shi, X. Pan, X. Chu, and B. Li, “Pipemoe: Accelerating mixture-of- experts through adaptive pipelining,” inINFOCOM, 2023

    2023

  27. [27]

    Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,

    Z. Fang, Y . Huang, Z. Hong, Y . Lyu, W. Chen, Y . Yu, F. Yu, and Z. Zheng, “Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,” inASPLOS, 2025

    2025

  28. [28]

    Parm: Efficient training of large sparsely-activated models with dedicated schedules,

    X. Pan, W. Lin, S. Shi, X. Chu, W. Sun, and B. Li, “Parm: Efficient training of large sparsely-activated models with dedicated schedules,” in INFOCOM, 2024

    2024

  29. [29]

    Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference,

    X. He, S. Zhang, Y . Wang, H. Yin, Z. Zeng, S. Shi, Z. Tang, X. Chu, I. Tsang, and O. Y . Soon, “Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference,”arXiv preprint arXiv:2410.17954, 2024

    arXiv 2024

  30. [30]

    Netmoe: Accelerating moe training through dynamic sample placement,

    X. Liu, Y . Wang, F. Fu, X. Miao, S. Zhu, X. Nie, and C. Bin, “Netmoe: Accelerating moe training through dynamic sample placement,” inICLR, 2025

    2025

  31. [31]

    Communication-efficient sparsely-activated model training via sequence migration and token condensation,

    F. Chen, P. Li, Z. Hong, Z. Su, and S. Guo, “Communication-efficient sparsely-activated model training via sequence migration and token condensation,”IEEE Transactions on Networking, 2024

    2024

  32. [32]

    {PopFetcher}: Towards accelerated{Mixture-of-Experts} training via popularity based{Expert-Wise}prefetch,

    J. Zhang, C. Ma, X. Wang, Y . Nie, Y . Li, Y . Xu, X. Liao, B. Li, and H. Jin, “{PopFetcher}: Towards accelerated{Mixture-of-Experts} training via popularity based{Expert-Wise}prefetch,” inATC, 2025

    2025

  33. [33]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016

    2016

  34. [34]

    Let’s verify step by step,

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inICLR, 2023

    2023

  35. [35]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,”arXiv preprint arXiv:2403.07974, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,

    Z. Du, S. Li, Y . Wu, X. Jiang, J. Sun, Q. Zheng, Y . Wu, A. Li, H. Li, and Y . Chen, “Sida: Sparsity-inspired data-aware serving for efficient and scalable large mixture-of-experts models,” 2024

    2024

  37. [37]

    D2moe: Dual routing and dynamic scheduling for efficient on-device moe-based llm serving,

    H. Wang, Q. zhou, Z. Hong, and S. Guo, “D2moe: Dual routing and dynamic scheduling for efficient on-device moe-based llm serving,” in Mobicom, 2025

    2025

  38. [38]

    On tail probabilities for martingales,

    D. A. Freedman, “On tail probabilities for martingales,”the Annals of Probability, pp. 100–118, 1975

    1975

  39. [39]

    Approximation-friendly discrepancy rounding,

    N. Bansal and V . Nagarajan, “Approximation-friendly discrepancy rounding,” inIPCO, 2016

    2016

  40. [40]

    Constructive algorithms for discrepancy minimization,

    N. Bansal, “Constructive algorithms for discrepancy minimization,” in2010 IEEE 51st Annual Symposium on Foundations of Computer Science. IEEE, 2010, pp. 3–10

    2010

  41. [41]

    Constructive discrepancy minimization by walking on the edges,

    S. Lovett and R. Meka, “Constructive discrepancy minimization by walking on the edges,”SIAM Journal on Computing, vol. 44, no. 5, pp. 1573–1582, 2015

    2015

  42. [42]

    L. C. Lau, R. Ravi, and M. Singh,Iterative methods in combinatorial optimization. Cambridge University Press, 2011, vol. 46

    2011

  43. [43]

    Colossal-ai: A unified deep learning system for large-scale parallel training,

    S. Li, H. Liu, Z. Bian, J. Fang, H. Huang, Y . Liu, B. Wang, and Y . You, “Colossal-ai: A unified deep learning system for large-scale parallel training,” 2023

    2023

  44. [44]

    SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,

    P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey,˙I. Polat, Y . Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henrik- se...

    2020

  45. [45]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han, “Awq: Activation-aware weight quantization for on-device llm compression and acceleration,”GetMobile: Mobile Comp. and Comm., Jan. 2025

    2025

  46. [46]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” inICML, 2022

    2022