A 3B-parameter Mixture-of-Experts model with experts shared across layers matches or approaches larger models on several benchmarks at much lower storage cost.
Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Megrez2 Technical Report
A 3B-parameter Mixture-of-Experts model with experts shared across layers matches or approaches larger models on several benchmarks at much lower storage cost.