REVIEW 6 cited by
Mixture-of-Experts with Expert Choice Routing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Sparsely-activated Mixture-of-experts (MoE) models allow the number of parameters to greatly increase while keeping the amount of computation for a given token or a given sample unchanged. However, a poor expert routing strategy (e.g. one resulting in load imbalance) can cause certain experts to be under-trained, leading to an expert being under or over-specialized. Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens. To address this, we propose a heterogeneous mixture-of-experts employing an expert choice method. Instead of letting tokens select the top-k experts, we have experts selecting the top-k tokens. As a result, each token can be routed to a variable number of experts and each expert can have a fixed bucket size. We systematically study pre-training speedups using the same computational resources of the Switch Transformer top-1 and GShard top-2 gating of prior work and find that our method improves training convergence time by more than 2x. For the same computational cost, our method demonstrates higher performance in fine-tuning 11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our method outperforms the T5 dense model in 7 out of the 11 tasks.
Forward citations
Cited by 6 Pith papers
-
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
HierMoE reduces MoE training time by removing duplicate token copies at each GPU-hierarchy level and swapping experts for load balance, measured at 1.18-1.27x end-to-end speedup on 32 GPUs.
-
Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
Chain-of-Experts replaces one parallel MoE routing step with several sequential expert steps inside a layer, reporting lower loss and memory use in small-scale experiments.
-
Maximum Score Routing For Mixture-of-Experts
MaxScore casts MoE routing as min-cost max-flow with SoftTopk and claims better loss and eval scores at equal FLOPs; unverified because the full text is unreadable.
-
Neural Inhibition Improves Dynamic Routing and Mixture of Experts
Neural inhibition gating on MoE router inputs improves a synthetic digit/squares benchmark by about four points over plain MoE, but the language-model evidence is unreliable.
-
Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning
A lightweight heterogeneous MoE with a GRU and an FFNN expert trails homogeneous baselines, and its claimed reasoning-type specialization is confounded by unequal expert inputs.
-
Breaking Thought Patterns: A Multi-Dimensional Reasoning Framework for LLMs
LADDER, a proposed mix of chain-of-thought prompting, mixture-of-experts layers, and linear projections, reportedly improves LLM creativity and diversity, but the evidence is thin and partly contradictory.
Discussion (0). Sign in to comment.