MCF-MOE improves MoE routing by combining cross-layer attention and local top-k token similarity, reporting lower perplexity and higher downstream accuracy than several MoE baselines.
Routing in Sparsely-gated Language Models responds to Context
1 Pith paper cite this work, alongside 1 external citations. Polarity classification is still indexing.
1
Pith paper citing it
1
external citations · Pith
abstract
Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computational budget. Building on previous efforts indicating that token-expert assignments are predominantly influenced by token identities and positions, we trace routing decisions of similarity-annotated text pairs to evaluate the context sensitivity of learned token-expert assignments. We observe that routing in encoder layers mainly depends on (semantic) associations, but contextual cues provide an additional layer of refinement. Conversely, routing in decoder layers is more variable and markedly less sensitive to context.
fields
cs.CL 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
MCF-MOE improves MoE routing by combining cross-layer attention and local top-k token similarity, reporting lower perplexity and higher downstream accuracy than several MoE baselines.