REVIEW 15 cited by
Mixture-of-Experts with Expert Choice Routing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2x. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.
Forward citations
Cited by 15 Pith papers
-
Language-Assisted Super-Resolution from Real-World Low-Resolution Patches
LA-SR redefines unpaired super-resolution in language space by projecting images into a semantically rich representation and applying vision-language model guided losses to handle real-world degradations extracted fro...
-
A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router
A mean-field limit of a reinforcement-based softmax router for two experts shows a supercritical pitchfork bifurcation, with an external asymmetry unfolding it into a cusp of fold bifurcations.
-
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
Adding temporal memory via LIF, precision-weighted gating, and anticipatory prediction to MoE routers recovers effective expert selection at distribution transitions, with ablation confirming a super-additive beta-ant...
-
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
Loss-Free Balancing keeps expert loads balanced in MoE models by dynamically adjusting routing-score biases based on recent usage, avoiding auxiliary-loss interference and yielding better performance.
-
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Coverage-aware pruning using per-corpus utility profiles on WikiText2 and C4 improves zero-shot accuracy and reduces perplexity degradation in two MoE models at 25-75% retention compared to baselines, without downstream data.
-
Schedule-Level Shared-Prefix Reuse for LLM RL Training
Schedule-level shared-prefix reuse decouples prefix and suffix passes in GRPO training to compute shared prefixes once, delivering up to 4.395x speedup and 59.1% HBM reduction while preserving numerical equivalence.
-
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
Piper introduces resource modeling and pipelined hybrid parallelism for MoE training, delivering 2-3.5X higher MFU than prior frameworks and 1.2-9X better all-to-all bandwidth.
-
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
AGoQ delivers up to 52% lower memory use and 1.34x faster training for 8B-32B LLaMA models by using near-4-bit adaptive activations and 8-bit gradients while preserving pretraining convergence and downstream accuracy.
-
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
HierMoE reduces MoE training time by removing duplicate token copies at each GPU-hierarchy level and swapping experts for load balance, measured at 1.18-1.27x end-to-end speedup on 32 GPUs.
-
Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
MoT decouples non-embedding parameters by modality in transformers to match dense multi-modal performance with roughly one-third to one-half the FLOPs.
-
Language-Assisted Super-Resolution from Real-World Low-Resolution Patches
LA-SR extracts real LR patches from depth-varying regions in single images and uses vision-language models with linguistic content and quality losses for unpaired super-resolution.
-
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
AGoQ cuts LLM training memory by up to 52% and speeds it up by 1.34x using tailored 4-bit activations and 8-bit gradients with special communication, matching baseline accuracy on LLaMA models.
-
Maximum Score Routing For Mixture-of-Experts
MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.
-
Neural Inhibition Improves Dynamic Routing and Mixture of Experts
Neural inhibition gating on MoE router inputs improves a synthetic digit/squares benchmark by about four points over plain MoE, but the language-model evidence is unreliable.
-
Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
A lightweight heterogeneous MoE with a GRU and an FFNN expert trails homogeneous baselines, and its claimed reasoning-type specialization is confounded by unequal expert inputs.
Discussion (0). Sign in to comment.