REVIEW 8 cited by
GRIN: GRadient-INformed MoE
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Mixture-of-Experts (MoE) models scale more effectively than dense models due to sparse computation through expert routing, selectively activating only a small subset of expert modules. However, sparse computation challenges traditional training practices, as discrete expert routing hinders standard backpropagation and thus gradient-based optimization, which are the cornerstone of deep learning. To better pursue the scaling power of MoE, we introduce GRIN (GRadient-INformed MoE training), which incorporates sparse gradient estimation for expert routing and configures model parallelism to avoid token dropping. Applying GRIN to autoregressive language modeling, we develop a top-2 16$\times$3.8B MoE model. Our model, with only 6.6B activated parameters, outperforms a 7B dense model and matches the performance of a 14B dense model trained on the same data. Extensive evaluations across diverse tasks demonstrate the potential of GRIN to significantly enhance MoE efficacy, achieving 79.4 on MMLU, 83.7 on HellaSwag, 74.4 on HumanEval, and 58.9 on MATH.
Forward citations
Cited by 8 Pith papers
-
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.
-
FLIP Reasoning Challenge
The FLIP benchmark of 11,674 blockchain image-story puzzles shows best open and closed AI models reach 75.5% and 77.9% accuracy, below the 95.3% human consensus baseline.
-
Scaling Laws for Upcycling Mixture-of-Experts Language Models
Upcycled MoE language-model loss follows a multiplicative power law in dense and upcycled training tokens, with a log-D1 interaction that caps the benefit of upcycling at large compute budgets.
-
SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation
SlimMoE uses multi-stage expert slimming and distillation to compress Phi-3.5-MoE into 7.6B and 3.8B parameter models that match or exceed similarly sized baselines while using about 10% of the original training data.
-
Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs
Pangu Ultra MoE, a 718B-parameter MoE model trained on 6,000 Ascend NPUs, reaches 30.0% MFU and matches DeepSeek R1 on a selected set of math, coding, and medical benchmarks.
-
ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.
-
A Survey on Inference Optimization Techniques for Mixture of Experts Models
A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.
-
An Introduction to Discrete Variational Autoencoders
A self-contained derivation of the ELBO and REINFORCE-style gradients for a discrete VAE with categorical latents, plus a minimal PyTorch implementation.
Discussion (0). Continue with ORCID to comment.