Pith. sign in

REVIEW 9 cited by

Upcycling Large Language Models into Mixture of Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.07524 v2 pith:LUU4NGDJ submitted 2024-10-10 cs.CL cs.AIcs.LG

Ethan He , Abhinav Khattar , Ryan Prenger , Vijay Korthikanti , Zijie Yan , Tong Liu , Shiqing Fan , Ashwath Aithal
show 2 more authors
Mohammad Shoeybi Bryan Catanzaro
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords upcyclingmodelsmodellanguageapproachtrainedachieveddense
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Upcycling pre-trained dense language models into sparse mixture-of-experts (MoE) models is an efficient approach to increase the model capacity of already trained models. However, optimal techniques for upcycling at scale remain unclear. In this work, we conduct an extensive study of upcycling methods and hyperparameters for billion-parameter scale language models. We propose a novel "virtual group" initialization scheme and weight scaling approach to enable upcycling into fine-grained MoE architectures. Through ablations, we find that upcycling outperforms continued dense model training. In addition, we show that softmax-then-topK expert routing improves over topK-then-softmax approach and higher granularity MoEs can help improve accuracy. Finally, we upcycled Nemotron-4 15B on 1T tokens and compared it to a continuously trained version of the same model on the same 1T tokens: the continuous trained model achieved 65.3% MMLU, whereas the upcycled model achieved 67.6%. Our results offer insights and best practices to effectively leverage upcycling for building MoE language models. Code is available.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond instruction-conditioning, MoTE: Mixture of Task Experts for Multi-task Embedding Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MoTE combines task-level mixture-of-experts routing with task-aware contrastive learning and beats instruction-conditioning on most MTEB tasks at equal active parameter count.

  2. Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Sparse Interpolated Mixture-of-Experts learns neuron-level masks and a per-prompt router to upcycle a dense LLM into a sparse expert model, improving instruction-tuning benchmarks with lower inference cost.

  3. Scaling Laws for Upcycling Mixture-of-Experts Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Upcycled MoE language-model loss follows a multiplicative power law in dense and upcycled training tokens, with a log-D1 interaction that caps the benefit of upcycling at large compute budgets.

  4. MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Grouping 495 languages into roughly 16 clusters and routing speech to group-specific LoRA experts improves multilingual ASR error rates over dense and random baselines.

  5. Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights

    cs.LG 2025-06 conditional novelty 5.0 of 10

    At 56B total parameters, fine-grained MoE with smaller, more numerous experts beats standard Switch and Mixtral-style MoE on validation loss and average downstream accuracy at matched FLOPs.

  6. FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FreqMoE upcycles a pretrained low-frequency FNO into a sparse mixture of experts via LoRA-style adapters, achieving up to 16.6% lower error with 47.32x fewer activated parameters.

  7. LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training

    cs.CL 2024-11 conditional novelty 5.0 of 10

    A dense instruction-tuned LLaMA-3-8B can be partitioned into Attention-MoE and MLP-MoE and re-trained with about 7B instruction tokens, recovering usable math/code skills with half the activated parameters.

  8. ScatterPrism: convergence for generative simulation and inverse problems in particle and nuclear physics

    cs.LG 2026-04 unverdicted novelty 4.0 of 10

    Standard Conditional Flow Matching loss is a misleading early plateau; physics-informed metrics keep improving, so ScatterPrism and multi-metric diagnostics are needed for kinematic fidelity.

  9. The Race to Efficiency: A New Perspective on AI Scaling Laws

    cs.LG 2025-01 conditional novelty 4.0 of 10

    A simple formula generalizes static AI scaling laws to time-varying efficiency, showing that efficiency-doubling rates around 2 per year keep loss reduction on a near-exponential path.

Pith tools