A recipe for setting MH-MoE expert widths, counts, and top-k to match SMoE FLOPs is presented with small perplexity gains, but the recipe's core equation is wrong and the experimental parity is not exact.
On the representation collapse of sparse mixture of experts
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
MH-MoE: Multi-Head Mixture-of-Experts
A recipe for setting MH-MoE expert widths, counts, and top-k to match SMoE FLOPs is presented with small perplexity gains, but the recipe's core equation is wrong and the experimental parity is not exact.