Pith. sign in

REVIEW 3 major objections 4 minor 50 references

A mixture-of-experts design with vector-level atomic experts and product routing is both more accurate and 10.9x faster at inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:08 UTC pith:AZ66R6US

load-bearing objection Plausible system-algorithm co-design with real measured speedups, but the abstract's O(N)-to-O(sqrt(N)) routing claim does not survive contact with the paper's own Appendix A, and the accuracy margins lack error bars. the 3 major comments →

arxiv 2602.05711 v2 pith:AZ66R6US submitted 2026-02-05 cs.CL cs.AI

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

Jingze Shi , Zhangyang Peng , Yizhang Zhu , Yifan Wu , Guang Liu , Yuyu Luo This is my paper
classification cs.CL cs.AI
keywords Mixture-of-Expertsfine-grained MoEatomic expertsCartesian product routerexpert-centric schedulinginference latencyzero-shot accuracyparameter efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OmniMoE tries to settle a trade-off that has shaped mixture-of-experts language models: coarse-grained experts are hardware-friendly but waste computation on irrelevant parameters, while fine-grained experts are parameter-efficient but suffer scattered memory access. The paper's proposal is to push granularity to the extreme — each routable unit is a single vector pair ('atomic expert') — and then fix the resulting routing and memory problems with two system-algorithm moves: a Cartesian Product Router that scores experts through two small projections, and Expert-Centric Scheduling that groups active experts into dense batched matrix operations. A sympathetic reader would care because the paper reports that the 6.4B-parameter model with 1.7B active parameters reaches 50.9% average zero-shot accuracy across seven benchmarks, ahead of both coarse- and fine-grained baselines, while inference latency drops from 73 ms to 6.7 ms (a 10.9x speedup). If correct, the result shows that massive fine-grained MoE is not inherently memory-bound, and the design space of expert count and granularity opens up for practical use.

Core claim

The central claim is that fine-grained MoE can be made fast and accurate by reconsidering all three parts — expert unit, routing, and execution — together. Each atomic expert is a pair of vectors acting as a minimal routable unit; Dynamic Expert Assembly gathers the top-K such units into a token-conditioned block on the fly. Routing over millions of experts is decomposed into a Cartesian product of two lower-dimensional routers, cutting per-token routing cost from O(N) to O(sqrt N) in projection FLOPs and storage. Execution is inverted from token-centric to expert-centric: active experts are grouped, tasks are sorted, and computation runs as grouped matrix multiplications, turning scattered

What carries the argument

The load-bearing machinery is the trio of (1) the Atomic Expert — a minimal routable unit parameterized by an input vector and an output vector, stored in two global matrices; (2) the Cartesian Product Router — a factorized router that treats an expert id as a coordinate on an Nr x Nc grid and scores it as the sum of a row log-probability and a column log-probability, reducing routing complexity from O(N) to O(sqrt N); and (3) Expert-Centric Scheduling — a reordering that compresses active experts into contiguous groups, hierarchically sorts tasks, and executes each group as a fused grouped GEMM, converting scattered memory lookups into coalesced dense operations. In a single sentence, the a

Load-bearing premise

The headline latency comparison assumes the competing models were measured with near-optimal kernels; the appendix states that state-of-the-art kernels were used, but gives no exact versions or scripts, so if the baseline's 73 ms includes avoidable overhead, the 10.9x speedup is not an architectural invariant.

What would settle it

Re-run the released code at the reported settings (4,096 tokens, ~28M activated parameters) and replace the baseline kernels with the latest vendor implementations; if the fine-grained baseline's latency falls close to OmniMoE's 6.7 ms, the claimed speedup is not an invariant of the architecture. Alternatively, measure the memory-traffic ratio L*K / |E_active| in a small batch; if it approaches 1, expert-centric scheduling yields little benefit and latency would match token-centric execution.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Massive fine-grained MoE no longer needs to be memory-bound: at 4,096 tokens the design runs at 6.7 ms versus 73 ms for the strong fine-grained baseline, so fine-grained routing can be used in latency-sensitive inference.
  • Expert count can grow to millions without growing router cost proportionally; the factorized router reduces projection work and storage to O(sqrt N), and distributed communication saturates once the expert pool exceeds the number of activated experts.
  • Smoother capacity scaling: activated parameters can be tuned finely by choosing how many atomic experts are routed per token, avoiding the discrete jumps of coarse expert blocks.
  • The split between a universally shared dense MLP and a routed fine-grained branch improves both knowledge-heavy and reasoning-heavy tasks, with the shared branch supporting reasoning and the routed branch supporting long-tail recall.
  • Under matched training budgets, the architecture's scaling-law curve is the best among the compared variants, so the accuracy gains are not a one-size artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The speedup is reported at a fixed batch/hardware setting; a natural test is to re-run at longer sequences, on newer GPUs, and with larger K, where grouped-GEMM efficiency and the L*K / |E_active| reuse ratio may change the margin.
  • The factorized router's independence assumption (row score plus column score) is a modeling approximation; a correlated or multi-factor version might improve routing quality on tasks where expert choices interact, at some extra cost.
  • The shared dense MLP is always active, so OmniMoE's compute is never as sparse as a pure fine-grained model; trading off shared-branch size against routed-branch size per domain could yield further gains.
  • If top-K selection (which still scans N scores per token) becomes the bottleneck as N grows past millions, hierarchical or approximate selection could be coupled with the Cartesian product structure; that would be an extension the paper only partially addresses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. OmniMoE proposes a fine-grained MoE layer in which the routed experts are vector-level atomic experts stored as rows of two global parameter matrices and assembled dynamically per token. Routing is performed by a Cartesian Product Router that decomposes the N-expert space into a row/column grid and factorizes the score as a sum of row and column log-probabilities. Execution is reorganized by Expert-Centric Scheduling, which groups tokens by active experts and runs grouped GEMMs. The paper reports controlled pretraining from scratch at several scales and claims that a 6.4B-A1.7B model reaches 50.9% average zero-shot accuracy over seven tasks, outperforming DeepSeekMoE and PEER, and that inference is 10.9x faster than PEER.

Significance. If the claims hold, the paper addresses a real tension in MoE design: fine-grained experts improve parameter efficiency but typically scatter memory traffic and slow execution. The controlled pretraining setup, the matched activated-parameter budgets, and the open-source code release are strengths, and the architectural combination of atomic experts, product-structured routing, and expert-centric scheduling is worth taking seriously. However, the central complexity claim is only partially supported by the paper's own derivation, and the empirical evidence for both accuracy and speedup lacks reproducibility details and error characterization.

major comments (3)
  1. [§2.2 and Appendix A, Eq. (v)] The abstract and Section 2.2 claim that the Cartesian Product Router reduces routing complexity from O(N) to O(sqrt(N)). The derivation in Appendix A establishes this only for the gate projection: C_cart ≈ 4d·sqrt(N) in Eq. (iii). The full routing pipeline also requires top-K selection over the implicit N_r×N_c grid, and Eq. (v) gives C_select = O(N·K + (N/B_sel)·K^2), which is linear in N per token. The statement that this score-evaluation work is 'negligible wall-clock overhead in practice' is unsupported: no routing-only timing is reported. Table 2's 'w/o Cartesian Product Router' ablation replaces both the projection and the selection method, so it does not isolate the selection cost. Please either report a separate routing-time measurement at N≈10^5–10^6 or revise the headline claim to say that the projection cost is O(sqrt(N)) while end-to-end routing remains O(N) in score evaluati
  2. [Table 1 and Figure 5] All quality comparisons are presented as single runs without error bars, seeds, or statistical testing. The headline accuracy difference is +0.7 average over DeepSeekMoE and +2.0 over PEER on a seven-task benchmark; at this scale these gaps can plausibly be within run-to-run noise. Figure 5 shows scaling curves without confidence intervals. Because the central claim is that OmniMoE outperforms coarse- and fine-grained baselines under matched budgets, the paper should provide multiple seeds or at least a per-seed breakdown and report variance.
  3. [§3.2 and Appendix B] The 10.9x latency speedup is a headline result, but the experimental protocol for the efficiency benchmark is not sufficiently reproducible. Appendix B names kernel families (CuTile for coarse-grained baselines, Triton fused kernels for fine-grained baselines) but gives no kernel versions, commit hashes, benchmark scripts, or measurement methodology, and no latency error bars are reported. The claim that 'strict end-to-end latency' includes all scheduling/reordering overheads cannot be checked from the manuscript. Please provide a detailed benchmark protocol, per-component latency breakdowns (routing, scheduling, GEMM), and repeated measurements.
minor comments (4)
  1. [§2.2] The text says 'See Appendix B for the full derivation and details,' but the complexity derivation is in Appendix A; Appendix B is experimental setup. Please fix the cross-reference.
  2. [§2.1] 'SWIGLU' should be 'SwiGLU'.
  3. [Appendix B] The statement that 'we verify that all methods adhere to predictable scaling laws' overstates what is shown in Figure 5, which displays scaling trends without fitted scaling-law parameters or extrapolation.
  4. [Appendix C] The distributed training claim is based on communication-only benchmarks. Figure B shows communication cost, but the paper does not report end-to-end distributed training throughput, so the statement that OmniMoE 'can scale to millions of experts with constant communication cost' should be scoped to the communication stage.

Circularity Check

0 steps flagged

No significant circularity: the architecture's gains are empirical measurements and the router/scheduling analyses are not derived from the target accuracy/latency numbers.

full rationale

I walked the paper's derivation chain. The Atomic Expert / Dynamic Expert Assembly formulation (Eqs. 4-7) is a definition, not a prediction derived from the reported accuracy or latency. The Cartesian Product Router is introduced as an explicit modeling assumption p(i,j|x) ≈ p_r(i|x)·p_c(j|x) (Eq. 8), and its projection-cost reduction is derived from the sizes of W_r and W_c (Appendix A Eqs. ii-iv), not from the final benchmark numbers. The top-K selection cost in the paper's own Appendix A Eq. (v) is O(N·K + (N/B_sel)·K^2), which does undercut the abstract's end-to-end O(√N) routing-complexity phrasing; however, this is an internal consistency / overclaim issue, not circularity, because the claimed complexity reduction is not obtained by defining a quantity in terms of the conclusion. The headline 10.9x speedup and 50.9% zero-shot accuracy are measured against external baselines (PEER, DeepSeekMoE, etc.) under matched activated-parameter budgets and controlled pre-training from scratch, with no parameter fitted to those targets. No self-citation or imported uniqueness theorem is load-bearing; the product-factorization ansatz is acknowledged as in-spirit with PKM/PEER rather than smuggled in through author self-citation. The ablations compare the architecture to its own variants, which is standard practice and not circular. I therefore find no step that reduces to its own inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims rest on a handful of modeling and experimental premises rather than on analytic first principles. The factorized-router independence (Eq. 8) is the main modeling axiom; the latency claim depends on baseline-implementation fairness; and scaling-law extrapolation is a domain assumption. No constants were fitted to the target results, so circularity burden is low, but the design has several hand-chosen hyperparameters (K, grid split, group size) that are not swept in the main accuracy comparison.

free parameters (3)
  • Top-K budget K = 512-4096 in sweeps; 4096 for the 28M-active latency point
    Hand-chosen; directly controls activated parameter count, routing cost, and scheduling group sizes. Central to the method's capacity/efficiency trade-off.
  • Grid dimensions (Nr, Nc) = Not reported; N=102,400 implies a balanced 320x320 split
    The router's factorization quality and O(sqrt(N)) projection cost depend on this split; the paper never states the actual choice in main text or tables.
  • Group size B = Not reported
    Expert-Centric Scheduling groups contiguous active experts into blocks of size B; B determines Grouped GEMM shapes and load balance but is not listed.
axioms (4)
  • ad hoc to paper p(i,j|x) approx p_r(i|x)*p_c(j|x): joint expert score factorizes into independent row and column marginals (Eq. 8, Section 2.2).
    This product-structure assumption is what makes the router cheap; if the true joint has strong row-column interactions, the top-K indices from S_ij = pr[i]+pc[j] are not the true top-K and accuracy/load balance should degrade.
  • domain assumption Baseline implementations (NVIDIA CuTile for coarse, Triton fused for fine) are representative near-optimal references for latency comparison.
    Appendix B states this but provides no kernel versions, commit hashes, or measurement scripts; the 10.9x speedup is only meaningful if PEER is not artificially slowed.
  • domain assumption Single-run pretraining on 40B tokens of SmolLM is sufficient to rank architectural variants and extrapolate via scaling laws.
    Table 1 and Figure 5 have no seeds or variance information; comparisons like +0.7 average over DeepSeekMoE are assumed to exceed run-to-run noise.
  • standard math The tiled merge-selection kernel recovers the exact global top-K indices over the implicit grid.
    Appendix A (Eq. v) gives complexity but no formal proof or test; the merge is a straightforward exact procedure, so this is a low-risk algorithmic assumption.
invented entities (1)
  • Atomic Expert no independent evidence
    purpose: Minimal routable unit parameterized by an input vector w_in_i and output vector w_out_i, computed as sigma(x w_in_i^T) w_out_i (Eq. 4), enabling dynamic assembly of token-conditioned expert blocks.
    New architectural primitive with only in-paper empirical support; no external falsifiable handle such as a theoretical prediction of which tasks benefit or a precise bandwidth bound.

pith-pipeline@v1.3.0-alltime-deepseek · 18332 in / 15374 out tokens · 157945 ms · 2026-08-03T04:08:16.938299+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution efficiency. We propose OmniMoE, a system-algorithm co-designed framework that pushes expert granularity to its logical extreme. OmniMoE introduces vector-level Atomic Experts, enabling scalable routing and execution within a single MoE layer, while retaining a shared dense MLP branch for general-purpose processing. Although this atomic design maximizes capacity, it poses severe challenges for routing complexity and memory access. To address these, OmniMoE adopts a system-algorithm co-design: (i) a Cartesian Product Router that decomposes the massive index space to reduce routing complexity from O(N) to O(sqrt(N)); and (ii) Expert-Centric Scheduling that inverts the execution order to turn scattered, memory-bound lookups into efficient dense matrix operations. Validated on seven benchmarks, OmniMoE (with 1.7B active parameters) achieves 50.9% zero-shot accuracy across seven benchmarks, outperforming coarse-grained (e.g., DeepSeekMoE) and fine-grained (e.g., PEER) baselines. Crucially, OmniMoE reduces inference latency from 73ms to 6.7ms (a 10.9-fold speedup) compared to PEER, demonstrating that massive-scale fine-grained MoE can be fast and accurate. Our code is open-sourced at https://github.com/flash-algo/omni-moe.

Figures

Figures reproduced from arXiv: 2602.05711 by Guang Liu, Jingze Shi, Yifan Wu, Yizhang Zhu, Yuyu Luo, Zhangyang Peng.

Figure 1
Figure 1. Figure 1: Activation Patterns and System Optimization. (a) Coarse-grained MoE activates large experts, inevitably involving redundant parameters and wasting computation. (b) Fine-grained MoE improves parameter efficiency, but suffers from bandwidth bottlenecks due to scattered, fragmented memory accesses. (c) Our OmniMoE employs a universally activated shared dense MLP, and uses expert-centric scheduling to reorgani… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the OmniMoE Architecture. The framework operates via two parallel pathways to balance efficiency and expressivity. (a) Dynamic Expert Assembly (Top): For Longtail Knowledge Retrieval objective, we employ a Cartesian Product Router (decomposed into Row/Column routers) to efficiently compute routing scores gx and identify the top-K expert indices Ix. Then the system dynamically retrieves specific… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of Execution Paradigms: Token-Centric vs. Expert-Centric Scheduling. (a) Conventional: Tokens independently fetch parameters from scattered experts, leading to random memory accesses (high load overhead) and fragmented vector-vector computations that underutilize on-chip SMs. (b) Our Approach: We invert the execution order using expert-centric scheduling. Left-to-Right: First, tasks are reordere… view at source ↗
Figure 4
Figure 4. Figure 4: End-to-End Efficiency Comparison. (a, b) Infer￾ence latency and (c, d) peak memory versus activated parameters (left column) and input token count (right column). Baselines include Dense, Gshard, DeepSeekMoE, PKM, and PEER. Omni￾MoE achieves consistently lower latency than DeepSeekMoE and fine-grained baselines (PKM/PEER), while maintaining a peak memory footprint comparable to coarse-grained MoEs. 3.2. Ma… view at source ↗
Figure 5
Figure 5. Figure 5: Scaling Laws. Validation perplexity (lower is better) versus (a) training FLOPs and (b) activated parameters. OmniMoE consistently outperforms all baselines, achieving the best trade-off between model quality and computational cost. compute-bound. Interestingly, although DeepSeekMoE uses coarse-grained FFN experts, it can be slower than fine-grained PEER at large token counts or activated budgets. This is … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 2 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Ainslie, J., Lee - Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebr \' o n, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 ...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  3. [3]

    Smollm-corpus, July 2024

    Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Smollm-corpus, July 2024

    2024

  4. [4]

    P I Q A : Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. P I Q A : Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on Artificial Intelligence, volume 34, 2020

    2020

  5. [5]

    Black, S., Biderman, S., and Hallahan, E. et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Mixture of neuron experts, 2025

    Cheng, R., Guan, Y., Ding, Y., Hu, Q., Wei, Y., Yuan, C., Shen, Y., Chen, W., and Gong, Y. Mixture of neuron experts, 2025. URL https://arxiv.org/abs/2510.05781

    arXiv 2025

  7. [7]

    Clark, A., de Las Casas, D., and Guy, A. et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research, pp.\ 4057--4086. PMLR , 2022. URL https://proceedings.mlr.press/v162/clark22a.html

    2022

  8. [8]

    Think you have solved question answering? try A R C , the A I 2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try A R C , the A I 2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  9. [9]

    Dai, D., Deng, C., and Zhao, C. et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    DeepSeek-AI , Liu, A., and Feng, B. et al. DeepSeek-V3 Technical Report , February 2025

    2025

  11. [11]

    Du, N., Huang, Y., and Dai, A. M. et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research, pp.\ 5547--5569. PMLR , 2022. URL https://proceedings.mlr.press/v162/du22c.html

    2022

  12. [12]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23: 0 120:1--120:39, 2022. URL https://jmlr.org/papers/v23/21-0998.html

    2022

  13. [13]

    Lighteval: A lightweight framework for llm evaluation, 2023

    Fourrier, C., Habib, N., Kydlíček, H., Wolf, T., and Tunstall, L. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/lighteval

    2023

  14. [14]

    Megablocks: Efficient sparse training with mixture-of-experts

    Gale, T., Narayanan, D., Young, C., and Zaharia, M. Megablocks: Efficient sparse training with mixture-of-experts. In Song, D., Carbin, M., and Chen, T. (eds.), Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023. mlsys.org, 2023

    2023

  15. [15]

    Sonicmoe: Accelerating moe with io and tile-aware optimizations, 2025

    Guo, W., Mishra, M., Cheng, X., Stoica, I., and Dao, T. Sonicmoe: Accelerating moe with io and tile-aware optimizations, 2025. URL https://arxiv.org/abs/2512.14080

    arXiv 2025

  16. [16]

    Scaling laws and compute-optimal training beyond fixed training durations

    H \"a gele, A., Bakouch, E., Kosson, A., Von Werra, L., Jaggi, M., et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37: 0 76232--76264, 2024

    2024

  17. [17]

    FastMoE : A Fast Mixture-of-Expert Training System , March 2021

    He, J., Qiu, J., Zeng, A., Yang, Z., Zhai, J., and Tang, J. FastMoE : A Fast Mixture-of-Expert Training System , March 2021

    2021

  18. [18]

    He, X. O. Mixture of A million experts. CoRR, abs/2407.04153, 2024. doi:10.48550/ARXIV.2407.04153. URL https://doi.org/10.48550/arXiv.2407.04153

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.04153 2024

  19. [19]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  20. [20]

    Hoffmann, J., Borgeaud, S., and Mensch, A. et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems (NeurIPS), 35: 0 30016--30030, 2022

    2022

  21. [21]

    Q., Sablayrolles, A., and Roux, A

    Jiang, A. Q., Sablayrolles, A., and Roux, A. et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017

    2017

  23. [23]

    Large memory layers with product keys

    Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., and Jégou, H. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pp.\ 8546--8557, 2019

    2019

  24. [24]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb

    2021

  25. [25]

    Predictable scale: Part i -- optimal hyperparameter scaling law in large language model pretraining, 2025

    Li, H., Zheng, W., Wang, Q., Zhang, H., Wang, Z., Xuyang, S., Fan, Y., Zhou, S., Zhang, X., and Jiang, D. Predictable scale: Part i -- optimal hyperparameter scaling law in large language model pretraining, 2025. URL https://arxiv.org/abs/2503.04715

    Pith/arXiv arXiv 2025

  26. [26]

    Li, Z., You, C., and Bhojanapalli, S. et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=TJ2nxciYCk-

    2023

  27. [27]

    A Survey on Inference Optimization Techniques for Mixture of Experts Models , January 2025

    Liu, J., Tang, P., Wang, W., Ren, Y., Hou, X., Heng, P.-A., Guo, M., and Li, C. A Survey on Inference Optimization Techniques for Mixture of Experts Models , January 2025

    2025

  28. [28]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Fixing weight decay regularization in adam. ArXiv, abs/1711.05101, 2017. URL https://api.semanticscholar.org/CorpusID:3312944

    Pith/arXiv arXiv 2017

  29. [29]

    Ludziejewski, J., Krajewski, J., and Adamczewski, K. et al. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=yoqdlynCRs

    2024

  30. [30]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    Pith/arXiv arXiv 2018

  31. [31]

    and Lin, S

    Mu, S. and Lin, S. A Comprehensive Survey of Mixture-of-Experts : Algorithms , Theory , and Applications , April 2025

    2025

  32. [32]

    T., Pham, Q., Bui, N

    Nguyen, H., Doan, T. T., Pham, Q., Bui, N. D. Q., Ho, N., and Rinaldo, A. On DeepSeekMoE : Statistical Benefits of Shared Experts and Normalized Sigmoid Gating , May 2025

    2025

  33. [33]

    Memory augmented language models through mixture of word experts

    Nogueira dos Santos, C., Lee-Thorp, J., Noble, I., Chang, C.-C., and Uthus, D. Memory augmented language models through mixture of word experts. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers...

    work page doi:10.18653/v1/2024.naacl-long.249 2024

  34. [34]

    Pytorch container image

    NVIDIA, M. Pytorch container image. https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch, 2022

    2022

  35. [35]

    Y., Awan, A

    Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. DeepSpeed-MoE : Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale , July 2022

    2022

  36. [36]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial W inograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  37. [37]

    GLU variants improve transformer

    Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202

    Pith/arXiv arXiv 2002

  38. [38]

    V., Hinton, G

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.n...

    2017

  39. [39]

    Sun, W., Hu, J., and Zhou, Y. et al. Speed Always Wins : A Survey on Efficient Architectures for Large Language Models , August 2025

    2025

  40. [40]

    Exploiting activation sparsity with dense to dynamic-k mixture-of-experts conversion

    Szatkowski, F., W \' o jcik, B., Pi \' o rczynski, M., and Scardapane, S. Exploiting activation sparsity with dense to dynamic-k mixture-of-experts conversion. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information...

    2024

  41. [41]

    Scattered Mixture-of-Experts Implementation , October 2024

    Tan, S., Shen, Y., Panda, R., and Courville, A. Scattered Mixture-of-Experts Implementation , October 2024

    2024

  42. [42]

    Team, K., Bai, Y., and Bao, Y. et al. Kimi K2 : Open Agentic Intelligence , July 2025

    2025

  43. [43]

    Qwen3, April 2025

    Team, Q. Qwen3, April 2025. URL https://qwenlm.github.io/blog/qwen3

    2025

  44. [44]

    Towards Greater Leverage : Scaling Laws for Efficient Mixture-of-Experts Language Models , October 2025

    Tian, C., Chen, K., Liu, J., Liu, Z., Zhang, Z., and Zhou, J. Towards Greater Leverage : Scaling Laws for Efficient Mixture-of-Experts Language Models , October 2025

    2025

  45. [45]

    Wolf, T., Debut, L., and Sanh, V. et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 38--45, Online, October 2020. Association for Computational Linguistics

    2020

  46. [46]

    XM o E : Sparse models with fine-grained and adaptive expert selection

    Yang, Y., Qi, S., Gu, W., Wang, C., Gao, C., and Xu, Z. XM o E : Sparse models with fine-grained and adaptive expert selection. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 11664--11674, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v...

    work page doi:10.18653/v1/2024.findings-acl.694 2024

  47. [47]

    Hella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  48. [48]

    Zheng, N., Jiang, H., and Zhang, Q. et al. PIT: optimization of dynamic sparse deep learning models via permutation invariant transformation. In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023 , pp.\ 331--347. ACM...

    arXiv 2023

  49. [49]

    Dropping experts, recombining neurons: Retraining-free pruning for sparse mixture-of-experts llms

    Zhou, Y., Zhao, Z., Cheng, D., Wu, Z., Gui, J., Yang, Y., Wu, F., Cheng, Y., and Fan, H. Dropping experts, recombining neurons: Retraining-free pruning for sparse mixture-of-experts llms. CoRR, abs/2509.10377, 2025. doi:10.48550/ARXIV.2509.10377. URL https://doi.org/10.48550/arXiv.2509.10377

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.10377 2025

  50. [50]

    St-moe: Designing stable and transferable sparse expert models

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022

    Pith/arXiv arXiv 2022