Pith. sign in

REVIEW 4 major objections 5 minor 40 references

FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that FLAME-MoE, a fully open family of seven Mixture-of-Experts language models (38M to 1.7B active parameters), improves average accuracy by up to 3.4 points over dense baselines trained at identical FLOPs, and releases…

desk verdict Useful open MoE platform, but the headline 3.4-point accuracy gain over dense baselines is inflated by overtrained dense models at the key compute budget. read the letter →

arxiv 2505.20225 v1 pith:KYZF5XUT submitted 2025-05-26 cs.CL cs.LG

Hao Kang , Zichun Yu , Chenyan Xiong This is my paper
classification cs.CLcs.LG
keywords mixture-of-expertslanguagemodelssparseroutingscalinglawsexpertspecializationroutersaturationopen-sourceresearchplatformtrainingdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces FLAME-MoE, a fully open suite of seven decoder-only Mixture-of-Experts language models (38M to 1.7B active parameters) with 64 experts per layer, top-8 routing, and two shared experts. Its central claim is that at equal training FLOPs the MoE models outperform dense transformers of comparable size by up to 3.4 average accuracy points across six tasks. The paper also reports three training-dynamics findings from full checkpoints: token-level expert specialization grows over time, expert co-activation remains sparse, and router selections converge early. If true, FLAME-MoE gives academic researchers a reproducible platform to study MoE scaling, routing, and expert behavior across model sizes rather than only benchmark final checkpoints.

What carries the argument

The load-bearing object is the MoE layer: $N_E = 64$ expert FFNs per layer, top-$k = 8$ routing, and two shared experts that fire for every token, giving $k = 8$ active experts per token. The scaling-law machinery treats active parameters $N_{\text{active}}$ and tokens $D$ as the compute variables through $C = \kappa N_{\text{active}} D$ with $\kappa = 6$, fitting an IsoFLOP profile and a parametric loss $L_{\text{val}}(N_{\text{active}}, D) = A/N_{\text{active}}^\alpha + B/D^\beta + L_0$ to select compute-optimal sizes. The trace analyses define specialization as the fraction of a token's occurrences routed to one expert, co-activation as the conditional probability that two experts appear together in a token's top-k set, and saturation as the step-by-step overlap of top-k selections with the final checkpoint.

What would settle it

Re-train the same dense-MoE pairs while measuring actual FLOPs with a profiler and equalizing total compute, wall-clock time, or energy, then re-run the six-task evaluation; if the average accuracy gap drops below or reverses the reported 1.8 to 3.4 points, the central comparison fails. A cheaper falsifier is to check whether the power-law exponents reproduce when $\kappa$ is set from measured FLOPs rather than the standard value of 6.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a compute-optimal MoE family with 64 experts and top-8 routing consistently beats dense transformers trained at the same FLOPs budget, with average accuracy gains of 1.8 to 3.4 points that widen at larger scales. The paper attributes this to effective capacity without proportional compute cost, and it supports the claim by training all models in the same codebase and releasing every checkpoint. Its trace analyses add three behavioral findings: token-level expert specialization intensifies during pretraining, expert co-activation is sparse overall yet stronger in deeper layers, and router selections converge to near-final assignments within the first few thousand steps. The paper presents the full release of code, data pipelines, logs, and checkpoints as the infrastructure that makes these observations reproducible.

Load-bearing premise

The equal-FLOPs comparison assumes that FLOPs computed as $\kappa \times \text{active parameters} \times \text{tokens}$ with $\kappa = 6$ captures the full training cost and that the dense baselines are as well tuned as the MoE models; if that accounting favors MoE or the dense baselines are undertuned, the reported accuracy advantage is not evidence of intrinsic MoE efficiency.

Editorial extensions

If this is right

  • At equal FLOPs, the MoE advantage should persist and widen as the training budget grows, since the reported gains rise from 1.8 points at smaller budgets to 3.4 points at the largest.
  • The released checkpoints allow independent replication of the three training-dynamics findings at every scale, not just at the final model.
  • The fitted scaling law with exponents $a \approx 0.690$ and $b \approx 0.310$ gives a resource-allocation rule: for a target FLOPs budget, the compute-optimal active-parameter count and token count follow these power laws.
  • Router saturation within the first few thousand steps implies that routing behavior is largely determined early; comparisons of routing interventions should therefore focus on early training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper measures FLOPs as $\kappa \cdot N_{\text{active}} \cdot D$, so the headline 3.4-point gain should not be read as a wall-clock or memory advantage; the paper itself reports that MoE throughput lags dense models, so the efficiency claim is about theoretical compute, not realized speed.
  • Because the released traces include checkpoints, a natural extension is to test whether freezing or pruning the router after saturation preserves downstream accuracy; the paper does not run this experiment.
  • The specialization score's token-frequency normalization suggests a sharper check: measure per-token accuracy on rare versus frequent tokens to see whether expert specialization improves handling of rare tokens specifically.
  • The seven-scale suite could support scaling-law analyses of routing dynamics themselves, such as how the saturation step count grows with active parameters; the paper only reports descriptive trends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. FLAME-MoE introduces a fully open research platform for Mixture-of-Experts language models: seven decoder-only MoE models (38M–1.7B active parameters, 64 experts, top-8 routing, 2 shared experts), trained on DCLM data, with all code, data pipelines, checkpoints, logs, and routing traces released. The paper fits an empirical scaling law (IsoFLOP profiles plus a parametric loss) to select supposedly compute-optimal model sizes, compares the MoE models against dense baselines trained at the same nominal FLOPs on six downstream tasks, and reports average accuracy gains up to 3.4 points. It also presents analyses of expert specialization, expert co-activation, and router saturation using the released checkpoints.

Significance. If the claims hold, FLAME-MoE is a valuable community asset: it is, to my knowledge, the first MoE suite to release checkpoints plus full routing traces across seven scales, matching the Pythia-style transparent platform that the dense community has long had. The scaling-law fitting is transparent, the evaluation uses standard tasks, and the router-dynamics analyses are a useful template. The central empirical claim—that MoE improves average accuracy over identically-FLOPs dense baselines—is plausible and would be an important data point, but it currently rests on assumptions about dense-baseline optimality, FLOPs accounting, and statistical significance that are not fully validated in the manuscript.

major comments (4)
  1. [§5.1, Table 3, Eq. (6)] The headline 'up to 3.4 points' gain comes from the 8.0e19 row, where Dense-411M is trained with D = C/(κN) ≈ 3.24e10 tokens, i.e., about 79 tokens per parameter. Under the same Chinchilla-style accounting used in the paper, the compute-optimal dense model at this budget would have about 8.2e8 parameters and about 20 tokens per parameter. No dense isoFLOP sweep is reported, so the comparison pits a compute-optimal MoE (selected by the paper's scaling law) against an overtrained dense model. To support the claim, the authors should provide dense isoFLOP profiles for the same budgets or otherwise demonstrate that the dense baselines in Table 3 are at their compute-optimal allocation.
  2. [Table 3, §5.2] All evaluation results are single runs with no error bars, confidence intervals, or significance tests. Some reported gaps are small (e.g., 0.69 points at 1.0e18, 2.3 points at 6.0e18) and could easily be within run-to-run variance on six tasks. The abstract's 'up to 3.4 points' is a single realization. At minimum, multiple seeds for both MoE and dense models, or a bootstrap analysis over tasks, are needed to establish that the improvements are not noise.
  3. [§4.2, §4.3, Fig. 1c, Table 4] The 'compute-optimal' label for the released family relies on a power law fitted to four budgets (1e18, 3e18, 6e18, 3e19) and extrapolated to 8e19 and 2.4e20—factors of 2.7 and 8 beyond the largest fitted budget. No confidence intervals are reported for the fitted exponents (a=0.69, b=0.31), and no held-out budget is used to validate the extrapolation. The claim that FLAME-MoE-721M-3.8B and FLAME-MoE-1.7B-10.3B are compute-optimal is therefore not established; the authors should report uncertainty in the scaling-law parameters and, if possible, validate on an intermediate held-out budget.
  4. [§5.1, Eq. (6)] The 'identical FLOPs' premise is defined by Eq. (6), FLOPs = κ·N_active·D with κ=6, but the paper does not validate this formula against measured FLOPs for either architecture. For MoE models, attention, embedding, router, and communication overheads are not captured by κ·N_active·D, and §5.3 shows that MoE training has lower realized throughput than dense training. Since the central comparison is explicitly 'identical FLOPs,' the authors should report profiler-measured FLOPs for at least the rows in Table 3, or clearly state that the comparison is theoretical and explain why the approximation is unbiased.
minor comments (5)
  1. [Table 3 footnotes] The footnotes 'DCLM 400M-1x', 'DCLM 400M-4x', and 'DCLM 1B-1x' are never defined; please clarify what these data-mixture/repetition labels mean and how they map to the token counts implied by Eq. (6).
  2. [Eq. (4)] The summation index i in the router z-loss appears to range over tokens, but i is also used for expert indices in Eqs. (2)–(3); using t or j for the token index would remove ambiguity.
  3. [Fig. 1a] The IsoFLOP curves are plotted against active parameters, but the four FLOPs budgets are only given in the legend; labeling each curve directly would improve readability.
  4. [§5.3, Fig. 3] The text says the best configuration is PP=1 and EP=8, but the figure also shows PP=2 variants; please state the criterion used to select PP=1/EP=8 (e.g., throughput, memory, or stability).
  5. [Abstract, §5.2] The phrase 'up to 3.4 points' is the maximum row-wise gain, not the average; please also report the mean gain across the seven comparisons so readers can gauge the typical effect.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central accuracy comparison is an independently measured empirical result, and the scaling-law extrapolations are standard model-selection readouts rather than derivations that assume their own conclusions.

full rationale

This paper is an empirical model release and comparison rather than a derivation chain. The central claim—that MoE models trained under equal FLOPs outperform dense baselines on six tasks—is established by actually training both sets of models and evaluating them (Table 3); no fitted parameter is later relabeled as a predicted accuracy. The scaling-law section fits L_val(N_active, D) and IsoFLOP parabolas to compute optimal sizes, but the optimum is a readout of the fitted empirical law, not a derivation that assumes the conclusion. Figure 1b plots fitted values against training data, but the downstream comparison is independent of this in-sample fit. The choice of dense baselines (e.g., Dense-411M at 8.0e19 is overtrained by Chinchilla standards) is a potential fairness threat to the comparison, but it is not a circularity: the dense accuracies are measured, not derived from the MoE fit, and the paper makes no claim that the dense baselines are compute-optimal. All architecture choices and hyperparameters are drawn from external, non-overlapping prior work (DeepSeek-V2, OLMoE, Megatron-LM), not from a self-citation chain. The paper even reports that MoE wall-clock throughput lags dense models (Section 5.3), which runs counter to any suspicion that the efficiency claim is asserted by construction. No self-definitional, fitted-input-as-prediction, self-citation-load-bearing, or uniqueness-importing step was found.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The central comparison is empirical, so the main fitted inputs are the scaling-law parameters used to select model sizes. No new theoretical entities are introduced.

free parameters (9)
  • A (parametric loss) = 148.413257
    Fitted in Section 4.2 to scale-loss data.
  • B (parametric loss) = 3269017.372472
    Fitted in Section 4.2.
  • L0 (irreducible loss) = 2.241716
    Fitted in Section 4.2.
  • alpha = 0.279702
    Fitted exponent in parametric loss.
  • beta = 0.715500
    Fitted exponent in parametric loss.
  • a (optimal N exponent) = 0.689902
    Derived from fitted loss; used to pick model sizes.
  • b (optimal D exponent) = 0.310098
    Derived from fitted loss; used to pick token budgets.
  • gamma (load balancing coefficient) = 0.01
    Chosen following OLMoE, not fitted.
  • eta (router z-loss coefficient) = 0.001
    Chosen following OLMoE, not fitted.
assumptions (4)
  • domain assumption Chinchilla-style scaling laws apply to MoE models when parametrized by active parameters.
    Section 4 uses L(N_active, D) = A/N^alpha + B/D^beta + L0 to select compute-optimal model sizes; this is an extrapolation from fitted data.
  • domain assumption FLOPs per token for MoE is proportional to active parameters with kappa=6.
    Eq. 6 sets D = C/(kappa*N_active), ignoring attention, embedding, and shared-expert overheads.
  • domain assumption DCLM is a representative corpus for studying scaling and downstream behavior.
    All models are trained on DataComp-LM; the scaling law and downstream results may not transfer to other data distributions.
  • domain assumption The architecture choices (64 experts, top-8, 2 shared experts) are effective.
    Section 3.1 adopts these from DeepSeek-V2 and OLMoE without independent ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models." pith.science (2026). https://pith.science/paper/KYZF5XUT

@misc{pith2026250520225,
  author       = {Pith},
  title        = {Pith review of: FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYZF5XUT}},
  note         = {Machine review of arXiv:2505.20225}
}
read the original abstract

Recent large language models such as Gemini-1.5, DeepSeek-V3, and Llama-4 increasingly adopt Mixture-of-Experts (MoE) architectures, which offer strong efficiency-performance trade-offs by activating only a fraction of the model per token. Yet academic researchers still lack a fully open, end-to-end MoE platform for investigating scaling, routing, and expert behavior. We release FLAME-MoE, a completely open-source research suite composed of seven decoder-only models, ranging from 38M to 1.7B active parameters, whose architecture--64 experts with top-8 gating and 2 shared experts--closely reflects modern production LLMs. All training data pipelines, scripts, logs, and checkpoints are publicly available to enable reproducible experimentation. Across six evaluation tasks, FLAME-MoE improves average accuracy by up to 3.4 points over dense baselines trained with identical FLOPs. Leveraging full training trace transparency, we present initial analyses showing that (i) experts increasingly specialize on distinct token subsets, (ii) co-activation matrices remain sparse, reflecting diverse expert usage, and (iii) routing behavior stabilizes early in training. All code, training logs, and model checkpoints are available at https://github.com/cmu-flame/FLAME-MoE.

Figures

Figures reproduced from arXiv: 2505.20225 by the authors.

Figure 1
Figure 1. Scaling law experiments: (a) IsoFLOP profiles; (b) parametric loss function fitting; (c) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Downstream comparison between FLAME-MoE and dense models during pretraining. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Training efficiency of FLAME-MoE-1.7B-10.3B under different parallelization strategies [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Evolution of specialization scores for the top-2 most specialized tokens across Experts 0, 1, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Expert co-activation in FLAME-MoE-1.7B-10.3B at the final checkpoint on the validation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Router saturation across training for FLAME-MoE-1.7B-10.3B. Each subplot shows the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: GPU utilization under different parallelization strategies (EP = Expert Parallel, PP = [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Evolution of expert specialization scores over the course of training in the final layer of [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Evolution of expert co-activation in FLAME-MoE-1.7B-10.3B at 10%, 50%, and 100% of [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Expert co-activation patterns in FLAME-MoE models of varying scales at the final [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Expert co-activation patterns in FLAME-MoE models of varying scales at the final [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Router saturation over the course of training for FLAME-MoE models of different scales [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Router saturation over the course of training for FLAME-MoE models of different scales [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 31 canonical work pages

  1. [1]

    Generated data with fake privacy: Hidden dangers of fine-tuning large language models on generated data

    Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, and Sinem Sav. Generated data with fake privacy: Hidden dangers of fine-tuning large language models on generated data. arXiv preprint arXiv:2409.11423, 2024

    arXiv 2024

  2. [2]

    Emergent and predictable memorization in large language models.Advances in Neural Information Processing Systems, 36:28072–28090, 2023

    Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models.Advances in Neural Information Processing Systems, 36:28072–28090, 2023

    2023

  3. [3]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. InProc. of ICML, 2023

    work page 2023

  4. [4]

    PIQA: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. InProc. of AAAI, 2020

    work page 2020

  5. [5]

    On the representation collapse of sparse mixture of experts.Proc

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. On the representation collapse of sparse mixture of experts.Proc. of NeurIPS, 2022

    work page 2022

  6. [6]

    Think you have solved question answering? Try ARC, the ai2 reasoning challenge.ArXiv preprint, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the ai2 reasoning challenge.ArXiv preprint, 2018

    work page 2018

  7. [7]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.JMLR, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.JMLR, 2022

    2022

  8. [8]

    A framework for few-shot language model evaluation, 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    2023

Show all 40 references
  1. [9]

    Fastmoe: A fast mixture-of-expert training system.ArXiv preprint, 2021

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system.ArXiv preprint, 2021

    2021

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InProc. of ICLR, 2021

    2021

  3. [11]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  4. [12]

    MiniCPM: Unveiling the potential of small language models with scalable training strategies.ArXiv preprint, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies.ArXiv preprint, 2024

    2024

  5. [13]

    Demystifying verbatim memorization in large language models.arXiv preprint arXiv:2407.17817, 2024

    Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models.arXiv preprint arXiv:2407.17817, 2024. 10

    2024 arXiv

  6. [14]

    Tutel: Adaptive mixture-of-experts at scale.Proc

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale.Proc. of MLSys, 2023

    2023

  7. [15]

    Mixtral of experts.ArXiv preprint, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.ArXiv preprint, 2024

    2024

  8. [16]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProc. of ICLR, 2015

    2015

  9. [17]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding.ArXiv preprint, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding.ArXiv preprint, 2020

    2020

  10. [18]

    Base layers: Simplifying training of large, sparse models

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. InProc. of ICML, 2021

    2021

  11. [19]

    DataComp-LM: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. DataComp-LM: In search of the next generation of training sets for language models. InProc. of NeurIPS, 2024

    2024

  12. [20]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model.ArXiv preprint, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model.ArXiv preprint, 2024

    2024

  13. [21]

    Deepseek-v3 technical report.ArXiv preprint, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.ArXiv preprint, 2024

    2024

  14. [22]

    Tending towards stability: Convergence challenges in small language models.arXiv preprint arXiv:2410.11451, 2024

    Richard Diehl Martinez, Pietro Lesci, and Paula Buttery. Tending towards stability: Convergence challenges in small language models.arXiv preprint arXiv:2410.11451, 2024

    2024 arXiv

  15. [23]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    2025

  16. [24]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. InProc. of EMNLP, 2018

    2018

  17. [25]

    OLMoE: Open mixture-of- experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. OLMoE: Open mixture-of- experts language models. InProc. of ICLR, 2025

    2025

  18. [26]

    Attributing mode collapse in the fine-tuning of large language models

    Laura O’Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of F oundation Models, 2024

    2024

  19. [27]

    Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of- experts inference and training to power next-generation ai scale. InProc. of ICML, 2022

    2022

  20. [28]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. InProc. of AAAI, 2020

    2020

  21. [29]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.ArXiv preprint, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.ArXiv preprint, 2017

    2017

  22. [30]

    JetMoE: Reaching llama2 performance with 0.1 m dollars.ArXiv preprint, 2024

    Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. JetMoE: Reaching llama2 performance with 0.1 m dollars.ArXiv preprint, 2024. 11

    2024

  23. [31]

    Megatron-lm: Training multi-billion parameter language models using model parallelism.ArXiv preprint, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.ArXiv preprint, 2019

    2019

  24. [32]

    Layer by layer: Uncovering hidden representations in language models.arXiv preprint arXiv:2502.02013, 2025

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models.arXiv preprint arXiv:2502.02013, 2025

    2025 arXiv

  25. [33]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm

    Databricks Team. Free dolly: Introducing the world’s first truly open instruction-tuned llm. https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm, 2023

    2023

  26. [34]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.ArXiv preprint, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.ArXiv preprint, 2024

    2024

  27. [35]

    Llm circuit analyses are consistent across training and scale.arXiv preprint arXiv:2407.10827, 2024

    Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. Llm circuit analyses are consistent across training and scale.arXiv preprint arXiv:2407.10827, 2024

    2024 arXiv

  28. [36]

    OpenMoE: An early effort on open mixture-of-experts language models.ArXiv preprint, 2024

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. OpenMoE: An early effort on open mixture-of-experts language models.ArXiv preprint, 2024

    2024

  29. [37]

    M6-T: Exploring sparse expert models and beyond.arXiv preprint, 2021

    An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. M6-T: Exploring sparse expert models and beyond.arXiv preprint, 2021

    2021

  30. [38]

    HellaSwag: Can a machine really finish your sentence? InProc

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? InProc. of ACL, 2019

    2019

  31. [39]

    OPT: Open pre-trained transformer language models.ArXiv preprint, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022

  32. [40]

    ST-MoE: Designing stable and transferable sparse expert models.ArXiv preprint, 2022

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing stable and transferable sparse expert models.ArXiv preprint, 2022. 12 Table 4: Scaling law parameters of FLAME-MoE. Name Value A148.413257 B3269017.3724...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.