Grouped Query Experts applies per-group MoE routing to query heads in GQA, matching baseline accuracy while activating half the query heads on 250M models trained for 30B tokens.
Llama-moe v2: Exploring sparsity of llama from perspective of mixture-of-experts with post-training
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
verdicts
UNVERDICTED 4roles
background 1polarities
background 1representative citing papers
An analytical post-training method restructures FFNs into MoE by partitioning neurons based on activation patterns and building a router from statistics, achieving 1.17x speedup with minimal resources.
MACS reduces Expert Parallelism stragglers in MoE MLLMs via entropy-weighted visual token load and dynamic modality-adaptive expert capacity, without retraining.
Mix-MoE applies separate LM and MT expert groups in two post-pretraining stages with Fourier-enhanced routing to reduce parameter interference and improve multilingual MT over baselines.
citing papers explorer
-
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Grouped Query Experts applies per-group MoE routing to query heads in GQA, matching baseline accuracy while activating half the query heads on 250M models trained for 30B tokens.
-
Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
An analytical post-training method restructures FFNs into MoE by partitioning neurons based on activation patterns and building a router from statistics, achieving 1.17x speedup with minimal resources.
-
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
MACS reduces Expert Parallelism stragglers in MoE MLLMs via entropy-weighted visual token load and dynamic modality-adaptive expert capacity, without retraining.
-
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
Mix-MoE applies separate LM and MT expert groups in two post-pretraining stages with Fourier-enhanced routing to reduce parameter interference and improve multilingual MT over baselines.