pith. sign in

arxiv: 2401.04081 · v2 · pith:MRLI72XKnew · submitted 2024-01-08 · 💻 cs.LG · cs.AI· cs.CL

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

Maciej Pi\'oro , Kamil Ciebiera , Krystian Kr\'ol , Jan Ludziejewski , Micha{\l} Krutul , Jakub Krajewski , Szymon Antoniak , Piotr Mi{\l}o\'s
show 2 more authors
Marek Cygan Sebastian Jaszczur
This is my paper
classification 💻 cs.LG cs.AIcs.CL
keywords mambamodelsmoe-mambaperformanceexpertsmixturemodelrecent
0
0 comments X
read the original abstract

State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Style-Decoupled Adaptive Routing Network for Underwater Image Enhancement

    cs.CV 2026-04 unverdicted novelty 7.0

    SDAR-Net decouples degradation style from scene structure in underwater images and applies adaptive routing to achieve SOTA enhancement performance.

  2. Jamba: A Hybrid Transformer-Mamba Language Model

    cs.CL 2024-03 conditional novelty 7.0

    Jamba presents a hybrid Transformer-Mamba MoE architecture for LLMs that delivers state-of-the-art benchmark performance and strong results up to 256K token contexts while fitting in one 80GB GPU with high throughput.

  3. Efficient RWKV-based Representation Learning for 3D Point Clouds

    cs.CV 2026-06 unverdicted novelty 5.0

    Introduces P-RWKV block and PointER self-supervised framework to adapt RWKV for efficient 3D point cloud representation learning.

  4. 3DMambaComplete: Exploring Structured State Space Model for Point Cloud Completion

    cs.CV 2024-04 unverdicted novelty 5.0

    3DMambaComplete applies the Mamba model to point cloud completion via hyperpoint generation, spatial spreading, and mesh deformation, claiming better results than prior methods on benchmarks.

  5. A Survey on Efficient Inference for Large Language Models

    cs.CL 2024-04 accept novelty 3.0

    The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.