Across six MoE models, expert subspaces overlap substantially, routes stay coherent, the actual prefix narrows the selected expert's geometric advantage in all 39 tested cells, yet later experts still improve next-token prediction in most frozen-route tests.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
Across six MoE models, expert subspaces overlap substantially, routes stay coherent, the actual prefix narrows the selected expert's geometric advantage in all 39 tested cells, yet later experts still improve next-token prediction in most frozen-route tests.