Pith. sign in

REVIEW 3 cited by

Yuan 2.0-M32: Mixture of Experts with Attention Router

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.17976 v2 pith:K45YU5Y2 submitted 2024-05-28 cs.AI cs.CL

Yuan 2.0-M32: Mixture of Experts with Attention Router

Shaohua Wu , Jiangang Luo , Xi Chen , Lingjun Li , Xudong Zhao , Tong Yu , Chao Wang , Yue Wang
show 7 more authors
Fei Wang Weixu Qiao Houbo He Zeru Zhang Zeyu Sun Junxiong Mao Chong Shen
This is my paper
classification cs.AI cs.CL
keywords yuanexpertsrouteronlyaccuracyactivearchitectureattention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts, which improves the accuracy compared to the model with classical router network. Yuan 2.0-M32 is trained with 2000B tokens from scratch, and the training computation consumption is only 9.25% of a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates competitive capability on coding, math, and various domains of expertise, with only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8 respectively. The models and source codes of Yuan 2.0-M32 are released at Github1.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Universal Pansharpening Model

    cs.CV 2026-03 conditional novelty 6.0

    A single pansharpening model works across 4-, 7-, 8-, and 10-band satellite images by projecting arbitrary-band MS data into a fixed latent space and fusing with PAN via a latent diffusion bridge.

  2. L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts

    cs.LG 2026-01 unverdicted novelty 6.0

    L2R improves MoE performance by routing in a low-rank space with Lipschitz-controlled saturated inner-product scoring and multi-anchor mechanisms.

  3. Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

    cs.CL 2025-06 conditional novelty 6.0

    MoE models with activation rates in an optimal region outperform dense LLMs of identical total parameter count, training compute, and data budget, with the optimal region consistent across scales.