Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-grained MoE beats standard MoE at 56B scale

desk verdict Fine-grained MoE wins at 56B with controlled comparisons, but the single untuned training recipe leaves room for recipe bias. read the letter →

arxiv 2506.02890 v1 pith:WNACM6UD submitted 2025-06-03 cs.LG cs.AIcs.CL

Jakub Krajewski , Marcin Chochowski , Daniel Korzekwa This is my paper
classification cs.LGcs.AIcs.CL
keywords fine-grainedmixtureofexpertsMoEscalingexpertgranularitylargelanguagemodelsrouterdesignvalidationlossdownstreambenchmarkstrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that fine-grained Mixture of Experts—using many more, smaller experts instead of a few large ones—improves both pretraining loss and downstream accuracy at scales beyond 50B total parameters, with equal total parameters and matched compute. The authors compare four architectures at 11B and 56B parameters, and report that at the largest scale the fine-grained variants beat their standard counterparts, with the gain growing as training data increases. The paper also identifies practical router design choices and training dynamics that explain when the advantage appears. If correct, this gives a concrete recipe for building cheaper, higher-quality large language models.

What carries the argument

The central object is expert granularity G: total experts become G times more numerous and G times smaller, with G times more experts selected per token, leaving non-router FLOPs and parameter count unchanged. Granularity is what the argument varies; the comparison pairs (1xFLOPs-G1 vs 1xFLOPs-G8, and 2xFLOPs-G1 vs 2xFLOPs-G8) isolate the effect of fine-graining from the effect of activation budget. A second mechanism is the router's softmax-after-Top-k ordering, which the paper shows yields significantly lower validation loss than softmax-before-Top-k for fine-grained models (2.183 vs 2.219 for 1xFLOPs-G8, and 2.166 vs 2.194 for 2xFLOPs-G8).

What would settle it

Train the 56B models with per-architecture hyperparameter sweeps—at minimum learning rate over {1e-4, 2e-4, 4e-4} and auxiliary loss coefficient over {0, 1e-3, 1e-2}—and check whether the standard MoE baselines, when tuned, reach or exceed the validation loss and benchmark averages of the fine-grained variants at the same token budget.

Watch

Extended reading notes

Core claim

At 56B total parameters trained on 300B tokens, replacing standard MoE experts with eight times more, eight times smaller experts—keeping total parameters and non-router FLOPs fixed—lowers validation loss and improves average downstream benchmark scores for both Top-1 (Switch-style) and Top-2 (Mixtral-style) routing. The fine-grained Top-1 variant (1xFLOPs-G8) not only outperforms its standard counterpart but matches the more expensive standard Top-2 model (2xFLOPs-G1) while activating roughly half the MoE parameters. The fine-grained Top-2 variant (2xFLOPs-G8) achieves the best overall scores. The advantage of granularity grows with training length: step-savings to reach the Switch baseline loss increase from 21.6% at 25B tokens to 33.6% at 100B tokens for the Top-1 fine-grained model, and the paper links this to the router gradually learning to use more than the top expert.

Load-bearing premise

All four architectures are trained with one identical hyperparameter recipe (learning rate, auxiliary loss, z-loss, capacity factor) and no per-architecture tuning, so the comparison is only fair if that single recipe is near-optimal for every variant.

Editorial extensions

If this is right

  • If the advantage persists at even larger scale, fine-grained MoE becomes the default architecture choice for training large models at fixed compute and parameter budgets.
  • The 1xFLOPs-G8 result implies a model can match a twice-as-expensive Top-2 baseline while activating half the MoE parameters, which directly reduces training and inference cost.
  • The growing step-savings with longer training horizons suggests fine-grained MoE is especially suited to heavily overtrained models, where token budgets far exceed the 20 tokens-per-parameter rule of thumb.
  • The router logit analysis implies that early training benefits little from extra activated experts, so techniques that accelerate router learning could make fine-grained MoE advantageous at short horizons too.
  • Practitioners should adopt softmax-after-Top-k routing for any MoE with k > 1, since the paper's measurements show it improves loss across all tested fine-grained and standard variants.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: at trillion-token budgets, the scaling trend in step-savings predicts an even larger fine-grained advantage, so compute-optimal training runs should favor higher granularity than the G=8 tested here.
  • The softmax-after-Top-k result suggests a testable mechanism—guaranteed normalization of selected-expert weights stabilizes the MoE layer output magnitude—which could be verified by ablating the router normalization order at larger scales.
  • The paper's fixed-recipe comparison could be extended by per-architecture hyperparameter sweeps; if fine-grained MoE is even more robust to learning-rate and auxiliary-loss choices, its practical dominance would be stronger than the current matched-recipe result alone shows.
  • Because the fine-grained Top-1 model matches the Top-2 standard model, an implicit consequence is that deployment-latency-sensitive systems can halve active expert parameters without losing quality, a claim the paper notes but does not benchmark in inference throughput.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a controlled empirical comparison of fine-grained Mixture-of-Experts (MoE) architectures against standard MoE baselines. Four variants are compared: Switch-style Top-1 with 8 experts (1xFLOPs-G1) and its fine-grained 64-expert Top-8 counterpart (1xFLOPs-G8), and Mixtral-style Top-2 with 8 experts (2xFLOPs-G1) and its fine-grained 64-expert Top-16 counterpart (2xFLOPs-G8). Total parameters are matched at 11B and 56B, and non-router FLOPs are matched within each pair. Models are trained on the same data, tokenizer, and framework with a fixed hyperparameter recipe, over token budgets of 25B, 50B, and 100B for the 11B models and 300B for the 56B models. The main results are that fine-grained MoE yields lower validation loss and better downstream benchmark scores, with gains that increase with training duration and model scale. At the largest scale, 1xFLOPs-G8 outperforms 1xFLOPs-G1 and matches 2xFLOPs-G1, while 2xFLOPs-G8 is best overall. The paper also analyzes expert load balance, router logit evolution, and the effect of applying softmax before or after Top-k in the router.

Significance. If the findings are robust, this is a valuable empirical contribution: it extends controlled evidence for fine-grained MoE to 56B parameters and, importantly, provides downstream benchmark results rather than perplexity only. The consistent controlled setup (same data, tokenizer, framework) is a strength, as are the practical analyses of load balance and router logits. The proposed training recipe and the finding about softmax/Top-k ordering will be useful to practitioners. However, the strength of the causal claim about granularity is currently weakened by two methodological gaps: the fixed single-recipe training without per-architecture tuning, and a router-normalization confound in the Switch-style pair. The single-run nature of all experiments also limits the significance of the small differences at 11B.

major comments (3)
  1. [Sec. 4.3, Table 6; Sec. 3.2, Table 2] The comparison between the Switch-style pair (1xFLOPs-G1 vs 1xFLOPs-G8) is confounded by the router's softmax/Top-k ordering. Section 4.3 and Table 6 show that for 1xFLOPs-G8, applying softmax before Top-k gives validation loss 2.219, while softmax after Top-k gives 2.183, a difference of 0.036. The total gap between 1xFLOPs-G1 (2.233) and 1xFLOPs-G8 (2.183) in Table 2 is only 0.050. Since the k=1 baseline necessarily uses softmax before Top-k, the majority of the reported granularity gain in this pair may be attributable to the router normalization change rather than to granularity itself. This is load-bearing for the conclusion in Section 3.4 that '1xFLOPs-G8 outperforms the standard Switch MoE.' The authors should compare against a G8 model using the same router ordering as G1, or explicitly quantify and discuss this confound.
  2. [Sec. 3.1, Training hyperparameters] All four architectures are trained with a single fixed recipe (learning rate 2e-4, capacity factor 1.5, aux-loss coefficient 1e-2, z-loss coefficient 1e-3) and no per-architecture hyperparameter sweeps are reported. This is a fairness concern for the central comparison: changing from 8 to 64 experts changes the scale of the load-balancing loss and the router gradient statistics, so the optimal aux-loss coefficient and capacity factor for G1 are not necessarily appropriate for G8. If the standard baselines are under-tuned, the pairwise wins in Tables 2 and 5 could reflect recipe bias rather than granularity. At minimum, the paper should report sensitivity experiments for the most likely hyperparameters (e.g., aux-loss coefficient, capacity factor, and learning rate) for the 11B models, or justify why one recipe is near-optimal for all variants.
  3. [Sec. 3.2, Table 2; Sec. 3.4, Table 5] All experiments are single runs with no error bars or repeated seeds. At 11B, the Mixtral-style pair has validation losses 2.168 and 2.166 (Table 2), a difference of 0.002 that is likely within run-to-run noise, yet the text interprets this as 'no significant advantage' and later uses the 56B results (Table 5) to claim a granularity benefit. Without multiple seeds, the scale-dependent trend could be an artifact of a single run. The authors should provide repeated seeds for at least the 11B configurations (which are computationally affordable) or clearly state that all results are single-run and temper the conclusions accordingly.
minor comments (5)
  1. [Table 2 versus Sec. 3.1.1] Table 2 omits several benchmarks listed in Sec. 3.1.1 (CommonsenseQA, MMLU, RACE, TruthfulQA) that appear only in Table 5; the inconsistency should be explained.
  2. [Sec. 4.3] The phrase 'significantly better results' is used without statistical testing or confidence intervals; consider rewording to 'substantially better' or adding variance estimates.
  3. [Fig. 1(c)] The figure legend could more clearly indicate that the curves are final validation losses at the three token horizons rather than continuous loss curves across tokens.
  4. [Sec. 3.1, notation] The term '1xFLOPs' might be misread as total training FLOPs; consider adding a footnote at first use clarifying that it refers to MoE-layer FLOPs.
  5. [Table 3] The caption for Table 3 could be more explicit that the reported percentages are savings relative to the total steps of the 1xFLOPs-G1 baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's conclusions are direct empirical measurements rather than derivatives of its inputs.

full rationale

This paper is an empirical benchmark study. Its central claims, that fine-grained MoE variants achieve lower validation loss and higher downstream accuracy at 56B parameters (Sec. 3.4, Table 5) and that their relative advantage grows with training length (Sec. 3.3, Table 3), are supported by directly measured validation losses and held-out benchmark scores. No parameter is fitted to the target quantity and then reported as a prediction. The only self-citation, 'Using granularity G >1 is the optimal choice from the scaling laws perspective (Krajewski et al. 2024)', is motivational background and is not used to derive the empirical ranking; the paper explicitly distinguishes its downstream-evaluation contribution as 'a case which was not considered in (Krajewski et al. 2024)'. The training-step savings in Table 3 are computed from observed loss curves ('how many fewer training steps a specific variant needs to reach the same final validation loss as the baseline Switch model'), not extrapolated from a fitted law. Even the acknowledged limitation that all variants share one fixed hyperparameter recipe (LR 2e-4, capacity factor 1.5, aux-loss 1e-2, z-loss 1e-3) concerns experimental fairness and potential undertuning of baselines, not circularity: the comparison does not define G1 performance in terms of G8 performance or vice versa. No equation is equivalent to another by construction, and no renamed fit is presented as a prediction. The self-citation is present but not load-bearing, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters were fitted to data. All hyperparameters are fixed across all four variants, so none is tuned to make the fine-grained results appear. The central claim rests on the assumption that the controlled comparison is fair and that the private training corpus does not hide a confound.

assumptions (3)
  • domain assumption Uniform hardware utilization across MoE architectures
    The efficiency comparison counts training steps and FLOPs, assuming fine-grained MoE achieves the same MFU as standard MoE. The authors flag this in Sec. 5: 'a key assumption throughout this report is uniform hardware utilization across different MoE architectures'. If fine-grained MoE has lower MFU, wall-clock efficiency gains shrink.
  • domain assumption A single shared hyperparameter recipe is fair for all four variants
    All four variants are trained with identical LR, capacity factor, auxiliary loss coefficient, and z-loss (Sec. 3.1). If the optimal recipe differs per architecture (e.g., fine-grained MoE needs weaker auxiliary loss), the ranking could be an artifact of the shared recipe.
  • domain assumption Continued pretraining on QA pairs affects all variants equally
    Before benchmark evaluation, all models receive continued pretraining on alignment-style QA pairs (10% budget). The paper assumes this step affects all variants equally and does not distort relative rankings. Location: Sec. 3.1 Training data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights." pith.science (2026). https://pith.science/paper/WNACM6UD

@misc{pith2026250602890,
  author       = {Pith},
  title        = {Pith review of: Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNACM6UD}},
  note         = {Machine review of arXiv:2506.02890}
}
read the original abstract

Mixture of Experts (MoE) architectures have emerged as pivotal for scaling Large Language Models (LLMs) efficiently. Fine-grained MoE approaches - utilizing more numerous, smaller experts - have demonstrated potential in improving model convergence and quality. This work proposes a set of training recipes and provides a comprehensive empirical evaluation of fine-grained MoE, directly comparing its scaling properties against standard MoE configurations for models with up to 56B total (17B active) parameters. We investigate convergence speed, model performance on downstream benchmarks, and practical training considerations across various setups. Overall, at the largest scale we show that fine-grained MoE achieves better validation loss and higher accuracy across a set of downstream benchmarks. This study offers empirical grounding and practical insights for leveraging fine-grained MoE in the development of future large-scale models.

Figures

Figures reproduced from arXiv: 2506.02890 by the authors.

Figure 1
Figure 1. (a) - (b): Validation loss curves of 11B models. Granularity improves performance of the Switch model, but doesn’t bring advantage for the Mixtral variant. This observation changes with longer training. (c): Comparison of the final loss for scaling the training length of the 11B models. Fine-grained variants perform relatively the best on the longest token horizon. 3.2 Comparing 11B Models We begin our comparison by… view at source ↗
Figure 2
Figure 2. Validation loss curves of 56B models. (left): Standard MoE (Switch variant, 1xFLOPs-G1) and its fine-grained counterpart. (right): Standard MoE (Mixtral variant, 2xFLOPs-G1) and its fine-grained counterpart. In both cases we show gains from increasing expert granularity. Directly comparing absolute pretraining loss values can be difficult. To provide a more intuitive measure of efficiency gains, in [PITH_FULL_IMAGE… view at source ↗
Figure 3
Figure 3. Fraction of load assigned to each Expert Parallel group for fine-grained models. We generally [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The distribution of the Top-k router logits in the initial layer of fine-grained models at [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of router logits in the middle [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Distribution of router logits in the middle [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Fraction of load assigned to each Expert Parallel group for the 11B 1xFLOPs-G8 model. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Fraction of load assigned to each Expert Parallel group for the 11B 2xFLOPs-G8 model. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 2 canonical work pages

  1. [1]

    arXiv: 1911

    Bisk, Yonatan, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi (2019).PIQA: Reasoning about Physical Commonsense in Natural Language. arXiv: 1911 . 11641 [cs.CL]. url: https : //arxiv.org/abs/1911.11641

    arXiv 2019

  2. [2]

    arXiv: 2202.01169 [cs.CL]

    Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan (2022).Unified Scaling Laws for Routed Language Models. arXiv: 2202.01169 [cs.CL]

    arXiv 2022

  3. [3]

    arXiv: 1803.05457 [cs.AI]

    Clark, Peter, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord (2018).Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv: 1803.05457 [cs.AI]. url: https://arxiv.org/abs/1803.05457

    arXiv 2018

  4. [4]

    Wu, Zhenda Xie, Y

    Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang (2024).DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of- Experts Language Models. arXiv: 2401.06066 [cs.CL]. DeepSeek-AI et al. (2024a).DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:...

    arXiv 2024

  5. [5]

    Sparse Upcycling: Inference Inefficient Finetuning

    Doubov, Sasha, Nikhil Sardana, and Vitaliy Chiley (2024).Sparse Upcycling: Inference Inefficient Finetuning. arXiv: 2411.08968 [cs.LG]. url: https://arxiv.org/abs/2411.08968

    work page Pith review arXiv 2024

  6. [6]

    arXiv: 2112.06905 [cs.CL]

    Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui (2022).GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. arXiv: 2112.06905 [cs.CL]

    arXiv 2022

  7. [7]

    (2024).The Llama 3 Herd of Models

    Dubey, Abhimanyu et al. (2024).The Llama 3 Herd of Models. arXiv: 2407.21783 [cs.AI] . url: https://arxiv.org/abs/2407.21783

    arXiv 2024

  8. [8]

    LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models

    Faiz, Ahmad, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang (2024). LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models. arXiv: 2309.14393 [cs.CL]

    arXiv 2024

Show all 40 references
  1. [9]

    arXiv: 2101.03961 [cs.LG]

    Fedus, William, Barret Zoph, and Noam Shazeer (2022).Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv: 2101.03961 [cs.LG]

    2022 arXiv

  2. [10]

    arXiv: 2211.15841 [cs.LG]

    Gale, Trevor, Deepak Narayanan, Cliff Young, and Matei Zaharia (2022).MegaBlocks: Efficient Sparse Training with Mixture-of-Experts. arXiv: 2211.15841 [cs.LG]. url: https://arxiv.org/abs/2211. 15841

    2022 arXiv

  3. [11]

    Upcycling large language models into mixture of experts

    He, Ethan, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro (2024). “Upcycling large language models into mixture of experts”. In:arXiv preprint arXiv:2410.07524

    2024 arXiv

  4. [12]

    Measuring Massive Multitask Language Understanding

    Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt (2021). Measuring Massive Multitask Language Understanding. arXiv: 2009 . 03300 [cs.CY]. url: https://arxiv.org/abs/2009.03300

    2021 arXiv

  5. [13]

    Rae, Oriol Vinyals, and Laurent Sifre (2022).Training Compute-Optimal Large Language Models

    Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre (2022).Training Compute-Optimal Large Language Models. arXiv: 2203.15556 [cs.CL]

    2022 arXiv

  6. [14]

    arXiv: 2401.04088 [cs.LG]

    Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed (2024).Mixtral of Experts. arXiv: 2401.04088 [cs.LG]

    2024 arXiv

  7. [15]

    arXiv: 2001.08361 [cs.LG]

    Gray, Alec Radford, Jeffrey Wu, and Dario Amodei (2020).Scaling Laws for Neural Language Models. arXiv: 2001.08361 [cs.LG]

    2020 arXiv

  8. [16]

    arXiv: 2402.07871 [cs.LG]

    Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur (2024).Scaling Laws for Fine-Grained Mixture of Experts. arXiv: 2402.07871 [cs.LG]. url: https://arxiv.org/abs/2402.07871

    2024 arXiv

  9. [17]

    arXiv: 1704.04683 [cs.CL]

    Lai, Guokun, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy (2017).RACE: Large-scale ReAding Comprehension Dataset From Examinations. arXiv: 1704.04683 [cs.CL] . url: https: //arxiv.org/abs/1704.04683

    2017 arXiv

  10. [18]

    arXiv: 2006.16668 [cs.CL]

    Krikun, Noam Shazeer, and Zhifeng Chen (2020).GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv: 2006.16668 [cs.CL]

    2020 arXiv

  11. [19]

    arXiv: 2109.07958 [cs.CL]

    Lin, Stephanie, Jacob Hilton, and Owain Evans (2022).TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv: 2109.07958 [cs.CL]. url: https://arxiv.org/abs/2109.07958

    2022 arXiv

  12. [20]

    arXiv: 1711.05101 [cs.LG]

    Loshchilov, Ilya and Frank Hutter (2019).Decoupled Weight Decay Regularization. arXiv: 1711.05101 [cs.LG]. url: https://arxiv.org/abs/1711.05101

    2019 arXiv

  13. [21]

    arXiv: 1809.02789 [cs.CL]

    Mihaylov, Todor, Peter Clark, Tushar Khot, and Ashish Sabharwal (2018).Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. arXiv: 1809.02789 [cs.CL]. url: https://arxiv.org/abs/1809.02789

    2018 arXiv

  14. [22]

    Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi (2024)

    Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi (2024). OLMoE: Op...

    2024 arXiv

  15. [23]

    arXiv: 2402.16819 [cs.CL]

    Parmar, Jupinder, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Ji...

    2024 arXiv

  16. [24]

    Reuse, Don’t Retrain: A Recipe for Continued Pretraining of Language Models

    Parmar, Jupinder, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro (2024b). Reuse, Don’t Retrain: A Recipe for Continued Pretraining of Language Models. arXiv: 2407.07263 [cs.CL]. url: https://arxiv.org/abs/2407.07263

    2024 arXiv

  17. [25]

    arXiv: 1907.10641 [cs.CL]

    Sakaguchi, Keisuke, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi (2019).WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv: 1907.10641 [cs.CL] . url: https: //arxiv.org/abs/1907.10641

    2019 arXiv

  18. [26]

    arXiv: 1904.09728 [cs.CL]

    Sap, Maarten, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi (2019).SocialIQA: Commonsense Reasoning about Social Interactions. arXiv: 1904.09728 [cs.CL] . url: https:// arxiv.org/abs/1904.09728

    2019 arXiv

  19. [27]

    arXiv: 2002.05202 [cs.LG]

    Shazeer, Noam (2020).GLU Variants Improve Transformer. arXiv: 2002.05202 [cs.LG]. url: https: //arxiv.org/abs/2002.05202. 12

    2020 arXiv

  20. [28]

    arXiv: 1701.06538 [cs.LG]

    Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean (2017).Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv: 1701.06538 [cs.LG]

    2017 arXiv

  21. [29]

    arXiv: 1909.08053 [cs.CL]

    Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro (2020).Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv: 1909.08053 [cs.CL]. url: https://arxiv.org/abs/1909.08053

    2020 arXiv

  22. [30]

    arXiv: 1811.00937 [cs.CL]

    Talmor, Alon, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant (2019).CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. arXiv: 1811.00937 [cs.CL] . url: https://arxiv.org/abs/1811.00937

    2019 arXiv

  23. [31]

    arXiv: 2403.08245 [cs.LG]

    Tan, Shawn, Yikang Shen, Rameswar Panda, and Aaron Courville (2024).Scattered Mixture-of-Experts Implementation. arXiv: 2403.08245 [cs.LG]. url: https://arxiv.org/abs/2403.08245

    2024 arXiv

  24. [32]

    (2024).Gemini: A Family of Highly Capable Multimodal Models

    Team, Gemini et al. (2024).Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL]. url: https://arxiv.org/abs/2312.11805

    2024 arXiv

  25. [33]

    arXiv: 2302.13971 [cs.CL]

    Joulin, Edouard Grave, and Guillaume Lample (2023a).LLaMA: Open and Efficient Foundation Language Models. arXiv: 2302.13971 [cs.CL]

    2023 arXiv

  26. [34]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom (2023b). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv: 2307.09288 [cs.CL]

    2023 arXiv

  27. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin (2023).Attention Is All You Need

    Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2023).Attention Is All You Need. arXiv: 1706.03762 [cs.CL]

    2023 arXiv

  28. [36]

    Llama 3 Meets MoE: Efficient Upcycling

    Vavre, Aditya, Ethan He, Dennis Liu, Zijie Yan, June Yang, Nima Tajbakhsh, and Ashwath Aithal (2024a). Llama 3 Meets MoE: Efficient Upcycling. arXiv:2412.09952 [cs.LG]. url: https://arxiv. org/abs/2412.09952. – (2024b). “Llama 3 Meets MoE: Efficient Upcycling”. In:arXiv prepri...

    2024 arXiv

  29. [37]

    arXiv: 2408.15664 [cs.LG]

    Wang, Lean, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai (2024).Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. arXiv: 2408.15664 [cs.LG]. url: https://arxiv.org/ abs/2408.15664

    2024 arXiv

  30. [38]

    OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

    Xue, Fuzhao, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You (2024). OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models. arXiv: 2402.01739 [cs.CL]. url: https://arxiv.org/abs/2402.01739

    2024 arXiv

  31. [39]

    url: https://arxiv.org/ abs/1905.07830

    Zellers, Rowan, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi (2019).HellaSwag: Can a Machine Really Finish Your Sentence?arXiv: 1905.07830 [cs.CL]. url: https://arxiv.org/ abs/1905.07830

    2019 arXiv

  32. [40]

    arXiv: 2202.08906 [cs.CL]

    Zoph, Barret, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus (2022).ST-MoE: Designing Stable and Transferable Sparse Expert Models. arXiv: 2202.08906 [cs.CL]. url: https://arxiv.org/abs/2202.08906. 13 A Distribution of the Router L...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.