REVIEW 19 cited by
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly to train from scratch in the large scale regime. In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget.
Forward citations
Cited by 19 Pith papers
-
ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI
A single 8B backbone unifies spatial perception, decision making, navigation/manipulation, and progress estimation with SSR+ merging, reporting gains on most spatial benchmarks and competitive action/progress results.
-
Beyond instruction-conditioning, MoTE: Mixture of Task Experts for Multi-task Embedding Models
MoTE combines task-level mixture-of-experts routing with task-aware contrastive learning and beats instruction-conditioning on most MTEB tasks at equal active parameter count.
-
NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptation
NoEsis couples DP-trained shared prompt tokens with non-private per-domain LoRA experts, achieving private knowledge transfer across Python, Java, and Go code completion.
-
Training Sparse Mixture Of Experts Text Embedding Models
Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.
-
BTS: Harmonizing Specialized Experts into a Generalist LLM
Branch-Train-Stitch merges frozen domain-specialist LLMs into a generalist by training only lightweight stitch layers, and reports the best average benchmark performance among merging baselines.
-
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
A continual model merging method, OPCM, sequentially projects each new task vector into a subspace orthogonal to the current merged model, achieving 5-8% higher average accuracy than baselines on CLIP-ViT tasks.
-
SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
SEUF restricts MoE LLM unlearning to the top-1 most relevant expert and adds a router anchor loss, improving utility by up to 35% while keeping forget efficacy roughly comparable.
-
Scaling Properties of Diffusion Models for Perceptual Tasks
Diffusion models for depth, optical flow, and amodal segmentation improve along power laws as training and test-time compute scale, and the fitted recipes match prior specialist models with less data.
-
SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
SpecPrefetch trains lightweight adapters to prefetch next-layer experts during offloaded MoE inference while keeping the native router authoritative, improving decoding throughput by up to ~20% on a mobile device.
-
MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...
-
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
Merging specialized LLMs into a MoE can be improved by replacing averaging with Dare/Ties merging and by using perplexity-based routing, while heterogeneous experts can be merged with projectors and a sequence-level router.
-
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
CLIP-UP converts a pre-trained dense CLIP into an MoE model and improves zero-shot text-image retrieval beyond dense baselines at lower inference cost.
-
LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
A dense instruction-tuned LLaMA-3-8B can be partitioned into Attention-MoE and MLP-MoE and re-trained with about 7B instruction tokens, recovering usable math/code skills with half the activated parameters.
-
Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
Pro-Prophet balances MoE training load by moving each expert to only the devices that need it and by pre-scheduling those moves to hide communication behind computation.
-
Sparse Upcycling: Inference Inefficient Finetuning
Sparse upcycling beats continued pretraining on quality by up to roughly 20 percent at matched compute, but cut serving throughput by 34 to 44 percent in vLLM benchmarks.
-
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.
-
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
A survey proposes eight architectural milestones and four control planes for MoE LLMs, arguing the field is moving toward decoupling routing, compute budgets, and physical execution.
-
K-EXAONE 2.0 Technical Report
LG AI Research upcycled K-EXAONE into a 750B-parameter MoE model (37B active) with a 256K context and ten languages, reporting large gains over its predecessor and top scores on long-context and safety benchmarks.
-
The Race to Efficiency: A New Perspective on AI Scaling Laws
A simple formula generalizes static AI scaling laws to time-varying efficiency, showing that efficiency-doubling rates around 2 per year keep loss reduction on a near-exponential path.
Discussion (0). Continue with ORCID to comment.