Pith. sign in

REVIEW 3 cited by

Multi-Head Mixture-of-Experts

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.15045 v1 pith:32VBHWKA submitted 2024-04-23 cs.CL cs.AIcs.LG

Xun Wu , Shaohan Huang , Wenhui Wang , Furu Wei This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords expertsmulti-headmh-moemodelingsmoeactivationexpertlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization. (2) Lacking fine-grained analytical capabilities for multiple semantic concepts within individual tokens. We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. These sub-tokens are then assigned to and processed by a diverse set of experts in parallel, and seamlessly reintegrated into the original token form. The multi-head mechanism enables the model to collectively attend to information from various representation spaces within different experts, while significantly enhances expert activation, thus deepens context understanding and alleviate overfitting. Moreover, our MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance. Extensive experimental results across three tasks: English-focused language modeling, Multi-lingual language modeling and Masked multi-modality modeling tasks, demonstrate the effectiveness of MH-MoE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective

    cs.LG 2025-02 reject novelty 5.0 of 10

    The paper derives convergence rates for sigmoid gating mixture-of-experts with quadratic scores and uses them to argue sigmoid self-attention is more sample-efficient than softmax, but the link to attention is an unpr...

  2. MolGraph-xLSTM: A graph-based dual-level xLSTM framework with multi-head mixture-of-experts for enhanced molecular representation and interpretability

    cs.LG 2025-01 reject novelty 5.0 of 10

    MolGraph-xLSTM applies xLSTM to atom- and motif-level molecular graphs with a multi-head mixture-of-experts and reports benchmark gains on 10 property prediction datasets.

  3. BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR

    cs.CL 2025-01 conditional novelty 5.0 of 10

    BLR-MoE, which adds language-specific attention experts, expert pruning, and router fine-tuning to the LR-MoE architecture, reduces WER by 16.09% relative on a 10,000-hour multilingual ASR benchmark.

Pith tools