A shared per-token controller jointly routes attention resolution, FFN experts, and KV bit-width and is claimed to Pareto-dominate independently tuned MoD+MoE+KV-quant at matched cost while protecting rare-token accuracy.
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention, September 2024
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
Despite many recent works on Mixture of Experts (MoEs) for resource-efficient Transformer language models, existing methods mostly focus on MoEs for feedforward layers. Previous attempts at extending MoE to the self-attention layer fail to match the performance of the parameter-matched baseline. Our novel SwitchHead is an effective MoE method for the attention layer that successfully reduces both the compute and memory requirements, achieving wall-clock speedup, while matching the language modeling performance of the baseline Transformer. Our novel MoE mechanism allows SwitchHead to compute up to 8 times fewer attention matrices than the standard Transformer. SwitchHead can also be combined with MoE feedforward layers, resulting in fully-MoE "SwitchAll" Transformers. For our 262M parameter model trained on C4, SwitchHead matches the perplexity of standard models with only 44% compute and 27% memory usage. Zero-shot experiments on downstream tasks confirm the performance of SwitchHead, e.g., achieving more than 3.5% absolute improvements on BliMP compared to the baseline with an equal compute resource.
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 2years
2026 2roles
background 1polarities
background 1representative citing papers
Looped-MoE models scale better than dense looped or standard transformers because routing changes across loops, and they enable stronger compute-quality trade-offs via early exits at loop boundaries.
citing papers explorer
-
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
A shared per-token controller jointly routes attention resolution, FFN experts, and KV bit-width and is claimed to Pareto-dominate independently tuned MoD+MoE+KV-quant at matched cost while protecting rare-token accuracy.
-
Sparse Layers are Critical to Scaling Looped Language Models
Looped-MoE models scale better than dense looped or standard transformers because routing changes across loops, and they enable stronger compute-quality trade-offs via early exits at loop boundaries.