Pith. sign in

REVIEW 5 cited by

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.15052 v2 pith:PVZ6YYPC submitted 2024-05-23 cs.LG cs.AI

Xianzhi Du , Tom Gunter , Xiang Kong , Mark Lee , Zirui Wang , Aonan Zhang , Nan Du , Ruoming Pang This is my paper
classification cs.LGcs.AI
keywords modeldenseshotmeasuresettingsactivatedadoptbudget
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Mixture-of-Experts (MoE) enjoys performance gain by increasing model capacity while keeping computation cost constant. When comparing MoE to dense models, prior work typically adopt the following setting: 1) use FLOPs or activated parameters as a measure of model complexity; 2) train all models to the same number of tokens. We argue that this setting favors MoE as FLOPs and activated parameters do not accurately measure the communication overhead in sparse layers, leading to a larger actual training budget for MoE. In this work, we revisit the settings by adopting step time as a more accurate measure of model complexity, and by determining the total compute budget under the Chinchilla compute-optimal settings. To efficiently run MoE on modern accelerators, we adopt a 3D sharding method that keeps the dense-to-MoE step time increase within a healthy range. We evaluate MoE and dense LLMs on a set of nine 0-shot and two 1-shot English tasks, as well as MMLU 5-shot and GSM8K 8-shot across three model scales at 6.4B, 12.6B, and 29.6B. Experimental results show that even under these settings, MoE consistently outperform dense LLMs on the speed-accuracy trade-off curve with meaningful gaps. Our full model implementation and sharding strategy has been released at~\url{https://github.com/apple/axlearn}

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    Expert upcycling duplicates experts in an existing MoE checkpoint and continues pre-training to match fixed-size baseline performance with 32% less compute.

  2. SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A multimodal language model that dynamically routes queries to the most relevant scene modalities and modality-specialized experts, achieving state-of-the-art results on five 3D benchmarks.

  3. Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

    cs.CL 2026-07 conditional novelty 6.0 of 10

    MCF-MOE improves MoE routing by combining cross-layer attention and local top-k token similarity, reporting lower perplexity and higher downstream accuracy than several MoE baselines.

  4. Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Expert upcycling expands MoE models by duplicating experts and continuing pre-training, matching baseline performance while saving 32% GPU hours in 7B-13B experiments.

  5. Apple Intelligence Foundation Language Models: Tech Report 2025

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Apple's 3B on-device and larger server language models match or beat several similarly sized open models on MMLU, MMMLU, and MGSM, using new efficiency techniques like KV-cache sharing and 2-bit quantization.

Pith tools