Pith. sign in

REVIEW 4 cited by

Yuan 2.0-M32: Mixture of Experts with Attention Router

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.17976 v2 pith:K45YU5Y2 submitted 2024-05-28 cs.AI cs.CL

Shaohua Wu , Jiangang Luo , Xi Chen , Lingjun Li , Xudong Zhao , Tong Yu , Chao Wang , Yue Wang
show 7 more authors
Fei Wang Weixu Qiao Houbo He Zeru Zhang Zeyu Sun Junxiong Mao Chong Shen
This is my paper · ORCID
classification cs.AIcs.CL
keywords yuanexpertsrouteronlyaccuracyactivearchitectureattention
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts, which improves the accuracy compared to the model with classical router network. Yuan 2.0-M32 is trained with 2000B tokens from scratch, and the training computation consumption is only 9.25% of a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates competitive capability on coding, math, and various domains of expertise, with only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8 respectively. The models and source codes of Yuan 2.0-M32 are released at Github1.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal Pansharpening Model

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A single pansharpening model works across 4-, 7-, 8-, and 10-band satellite images by projecting arbitrary-band MS data into a fixed latent space and fusing with PAN via a latent diffusion bridge.

  2. Multi-modal Collaborative Optimization and Expansion Network for Event-assisted Single-eye Expression Recognition

    cs.CV 2025-05 conditional novelty 4.0 of 10

    MCO-E Net fuses event and RGB eye data via a jointly optimized Mamba and a heterogeneous MoE, achieving 91.3% WAR and 91.9% UAR on the SEE dataset.

  3. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

  4. Mixture of Experts (MoE): A Big Data Perspective

    cs.LG 2025-01 conditional novelty 2.0 of 10

    A survey of MoE methods for big data that catalogs architectures, use cases, and open challenges without adding new results.

Pith tools