Pith. sign in

REVIEW 3 major objections 6 minor 49 references

Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Routing Mamba turns Mamba's linear projections into sparsely activated, jointly routed experts, and claims this scales SSMs far beyond their dense counterparts.

desk verdict Solid MoE-for-Mamba architecture paper with a genuinely useful shared-routing insight, but the abstract's 2.3x scaling claim is unsupported at 1.3B and needs correction. read the letter →

arxiv 2506.18145 v1 pith:PPGWDGKA submitted 2025-06-22 cs.LG cs.AI

Zheng Zhan , Liliang Ren , Shuohang Wang , Liyuan Liu , Yang Liu , Yeyun Gong , Yanzhi Wang , Yelong Shen This is my paper
classification cs.LGcs.AI
keywords mixtureofexpertsstatespacemodelsMambasharedroutingsparsescalinglinearprojectionlanguagemodelinghybridSSM-attention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right way to apply Mixture-of-Experts to a Mamba-style state space model is to turn its linear projection layers — Conv, Gate, and Output — into sparse experts that share a single routing decision, while keeping the smaller internal projections shared. It claims that this design, called Routing Mamba, makes SSMs scale parameter-efficiently: a 1.3B-active-parameter model (10B total) matches the language modeling perplexity of a dense Mamba requiring over 2.3x more active parameters, across 4K to 16K context lengths. The same recipe applied to the hybrid Mamba-attention model Samba gives the same perplexity as widening the dense model while saving about 23% of the FLOPs. The paper's point is that MoE can work for SSMs if the experts are the projections and the routing is coordinated, rather than the naive per-projection or FFN-only schemes that degrade performance.

What carries the argument

The central mechanism is shared routing: a single router per layer computes $\mathrm{Softmax}(X_t W_r)$, and the top-$K$ expert identities selected from the Gate projection are reused as the expert selection for the Conv and Output projections. This makes the three projections act as a coordinated 'expert pathway' for each token, avoiding the fragmentation that occurs when each projection has its own router (the naive MoE-Mamba strategy, which the paper shows degrades Samba from 10.05 to 10.26 PPL at 4K). The other load-bearing choice is selective expertization: only the large, computationally dominant projections become experts, while $x$ Proj, $dt$ Proj, and the 1D convolution stay shared, and no load-balance auxiliary loss is used.

What would settle it

Train a dense Mamba whose total parameter count equals RoM's (for the 1.3B-active case, about 10B, e.g., by widening the hidden size), on the same 20B SlimPajama tokens, and compare validation perplexity at 4K context. If that dense model reaches or beats RoM's 8.52 PPL, the claimed 2.3x active-parameter advantage is a total-parameter effect. A second check: replace RoM's shared router with independent per-projection routers at identical total parameters and training budget; if independent routing matches shared routing, the shared decision is not the source of the gain.

Watch

Extended reading notes

Core claim

Routing Mamba (RoM) is a parameter-scaling scheme for Mamba layers in which the Conv, Gate, and Output projection matrices are replicated as $N$ linear experts, and one router per layer chooses the same top-$K$ experts for all three projections at each token. The smaller $x$ projection, $dt$ projection, and 1D convolution are shared across experts. With one of eight experts active per token, RoM models at 115M to 1.3B active parameters (0.71B to 10B total) beat dense Mamba models at every active-parameter count and training sequence length, reaching perplexity that dense Mamba needs roughly 2.0–2.3x more active parameters to match; at 1.3B active it produces 8.52 versus 9.07 validation PPL at 4K context. Applied to the Samba hybrid, RoM matches Samba (expand=4) perplexity while saving 23% FLOPs, and the shared routing generalizes to Mamba-2 and Gated DeltaNet. The paper also reports that adding an explicit load-balance loss does not help, and that extending expertization to the small internal projections slightly hurts.

Load-bearing premise

The paper's efficiency comparisons match dense and RoM models on active parameters, not total parameters; RoM carries 6–8x more total weights, so the perplexity gap it attributes to the sparse shared-routing design could in principle come from the larger pool of learned parameters.

Editorial extensions

If this is right

  • Dense Mamba needs about 2.0–2.3x more active parameters to match RoM's perplexity at the same training budget, so SSM scaling can be done sparsely rather than by widening layers.
  • RoM maintains its perplexity advantage consistently across 4K, 8K, and 16K training sequence lengths, and keeps lower PPL when evaluated at longer contexts than training.
  • Applying RoM to the Samba hybrid attains the perplexity of Samba (expand=4) at about 23% fewer FLOPs, giving a compute-efficient way to scale hybrid SSM-attention models.
  • RoM transfers to Mamba-2 and Gated DeltaNet with comprehensive expertization of major projections, indicating the recipe is not specific to the original Mamba block.
  • Combining RoM with FFN-MoE keeps downstream-task accuracy roughly level with FFN-MoE while using fewer or equal total parameters (e.g., 49.2% versus 49.5% average accuracy at 5.6B versus 5.7B).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gain is a total-parameter effect, RoM's real contribution would be a memory-compute trade-off rather than a sample-efficiency gain; this distinction matters for deciding when to use RoM instead of simply training a wider dense model.
  • Shared routing across projections formalizes a broader design principle: for any block with multiple coupled linear transforms, a single 'pathway' router may beat per-transform routers; this is testable in attention and convolution blocks beyond SSMs.
  • A natural stress test the paper does not run is varying the number of experts at fixed total parameters (e.g., 2 experts versus 16 experts with adjusted widths); the current 1-of-8 choice may not be the optimal sparsity level.
  • The absence of a load-balance loss, if it holds at larger scales with expert parallelism, would simplify MoE training pipelines, but the paper's evidence is limited to 20B-token runs at up to 1.7B total parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Routing Mamba (RoM), which applies sparse mixtures of linear projection experts to the Conv, Gate, and Output projections of Mamba layers, with a single router shared across these projections and with x_proj, dt_proj, and Conv1D parameters shared across experts. The authors report language-modeling perplexity on SlimPajama for Mamba and RoM at 115M, 353M, 765M, and 1.3B active parameters (up to 10B total), and for hybrid Samba models, comparing against MoE-Mamba, MoA, and SwitchHead. They claim that at 1.3B active parameters RoM is equivalent to a dense Mamba with over 2.3x more active parameters, and that RoM yields a 23% FLOPs saving for hybrid scaling.

Significance. If the empirical claims hold, RoM would be a useful recipe for scaling SSM capacity through projection-level MoE with shared routing. The paper's main strengths are direct perplexity measurements rather than fitted predictions, a comparison against several prior MoE designs, and ablations on which projections to expertize. However, the headline quantitative claim is not supported by the paper's own tables, and the absence of matched-total-parameter dense baselines leaves the mechanism attribution incomplete.

major comments (3)
  1. [Abstract; Section 5.2; Tables 7-9] The abstract's claim that at 1.3B active parameters and 16K training sequence length RoM matches a dense Mamba requiring "over 2.3x more active parameters" is contradicted by Tables 7-9. In Table 9 (16K training), RoM-1.3B has PPL 8.19 at 16K evaluation while Mamba-1.3B has 8.68 and Mamba-765M has 9.45; log-linear interpolation gives an equivalent dense size near 1.8B, i.e., about 1.4x the active parameters. The same calculation for 4K and 8K training (Tables 7 and 8) gives ratios near 1.5x. The measured ratio decreases with scale (2.3x at 115M, 2.0x at 353M, about 1.4x at 1.3B), so the abstract is extrapolating in the wrong direction. Please state scale-specific multipliers and remove the "over 2.3x" claim at 1.3B.
  2. [Section 5.2; Tables 7-9] The efficiency comparison is only active-parameter-matched. RoM stores 710M total parameters at the 115M active size and 10B total at the 1.3B active size, i.e., between about 6x and 8x more parameters than the corresponding dense Mamba. Because no dense Mamba with matched total parameters and no Mamba MoE baseline with independent routers at these scales is trained, the observed perplexity gains could in principle be due to the larger total parameter count rather than to shared-routing sparse projection experts. Please add a matched-total-parameter dense baseline (or otherwise control for total parameter count) before claiming parameter-efficient scaling.
  3. [Section 5.3; Table 10] The text states that Samba + RoM + FFN-MoE (16top1) "significantly outperforms" Samba + FFN-MoE (32top1), but Table 10 shows the opposite: PPL 8.98 vs 8.88 at 4K, 8.70 vs 8.60 at 8K, and 8.54 vs 8.45 at 16K. Moreover, the two rows are trained with different token budgets (20B vs 17B) and different hardware/batch configurations. This comparison does not support the claims in the paragraph, and the claim "with fewer parameters (5.6B vs 5.7B)" is also not a matched comparison because of the token-budget mismatch. Please correct the text or rerun the comparison under identical budgets.
minor comments (6)
  1. [Section 5.2] The text says "the 115M model has 12 layers" while Table 5 reports 24 layers; please align the numbers.
  2. [Abstract; Section 5.3] The abstract describes the 23% saving as "compared to dense Mamba scaling," but the experiment in Section 5.3 compares RoM to Samba (expand=4), a hybrid SSM-attention model; please use "Samba" or "hybrid model" for precision.
  3. [Section 4.2, Eqs. (9)-(13)] The routing weights R_i defined in Eq. (9) are used as multiplicative weights in Eq. (12), but Eqs. (10) and (11) use only the indicator 1_{i in TopK}; please clarify whether the softmax weights are intentionally omitted in the Gate and Conv projections and define the expert computation E_i consistently.
  4. [Section 5.4 / Table 6] The claim that RoM balances expert load "naturally" is inferred from equal perplexity with and without a load-balancing loss in Table 6; reporting actual expert utilization statistics would make the claim directly supported.
  5. [Table 10] The table's second block is labeled "17B training tokens," while Section 5.3 describes the same setup as the "20B token setup"; please reconcile the token count.
  6. [Figure 3] The red dashed line is described in the caption as "at least 2x" and in the body text as "up to 2.3x"; make the label and text consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the results are direct empirical perplexity measurements, and the paper's own text limits the 2.3x claim to the 115M scale, making the abstract's 1.3B extrapolation an overstatement rather than a circular derivation.

full rationale

The paper proposes an architecture (RoM) and reports measured validation perplexities; there is no quantity that is fitted to a target and then renamed as a prediction. The '2.3x' claim in the abstract is contradicted by the paper's own Section 5.2, which assigns 2.3x only to the 115M scale and 2.0x to the 353M scale, while Tables 7-9 imply a much smaller multiplier at 1.3B; however, that is an unsupported extrapolation or reporting inconsistency, not circularity. The experimental comparisons are active-parameter-matched, and the larger total parameter count of RoM is a design feature, not a fitted input. Self-citations to prior work by overlapping authors (Samba training settings, SparseMixer router training) are used as baselines or implementation tools and are not load-bearing evidence for the central claim that shared routing over projection experts improves scaling. No equation in the paper reduces to its own output by construction, and no 'uniqueness theorem' is imported from the authors' prior work. The central claims therefore stand or fall on external empirical evidence, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The ledger contains no new physical or mathematical entities. The main hidden costs are architectural choices (N=8, top-1, 20B tokens) and the unstated control assumption that active parameters, not total parameters, are the correct basis for the efficiency comparison.

free parameters (4)
  • Number of experts N per MoE layer = 8
    Chosen for the capacity/efficiency tradeoff; no RoM ablation sweeps N (FFN-MoE ablations use 16/32 experts, but RoM stays at 8).
  • Top-K router selection = 1
    Top-1 expert per token and per layer is a design choice; no K>1 experiments are reported for RoM.
  • Training token budget = 20B
    All scaling and hybrid comparisons use a fixed 20B-token budget, so the scaling claims are specific to this regime.
  • Model shape points for scaling comparison = hidden sizes 768/1024/1536/2048; layers 24/48; d_state=16
    These discrete points define the 2.3x active-parameter equivalence; the factor is interpolated between 115M and 353M results.
assumptions (3)
  • domain assumption Shared routing across Conv, Gate, and Output projections is more effective than independent routers because the projections are functionally coupled.
    Asserted via analogy in Section 5.3 ("Why RoM works better"); supported only by the empirical ablation in Table 4, not by a mechanism-level analysis.
  • ad hoc to paper Sharing x_proj, dt_proj, and Conv1D parameters across experts is at least as good as making them expert-specific.
    Based on Table 1 comparison of RoM (Conv, Gate, Out) vs RoM (Conv, Gate, dt, x, Out) on one Samba configuration.
  • domain assumption The router stays load-balanced without an auxiliary balancing loss.
    Supported by Table 6 at one scale; no expert utilization statistics are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection." pith.science (2026). https://pith.science/paper/PPGWDGKA

@misc{pith2026250618145,
  author       = {Pith},
  title        = {Pith review of: Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPGWDGKA}},
  note         = {Machine review of arXiv:2506.18145}
}
read the original abstract

Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3x more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance.

Figures

Figures reproduced from arXiv: 2506.18145 by the authors.

Figure 1
Figure 1. Our method overview. The red box indicates components that are shared across all experts. (Left) This section illustrates how our method applies the shared routing strategy to the Conv Proj and Out Proj layers. (Middle) This part presents the Mamba block incorporated in our method, where we apply the gating weights to multiply with the representations after Out Proj. (Right) This section demonstrates the Gate Proj u… view at source ↗
Figure 2
Figure 2. Perplexity (PPL) on the SlimPajama validation set (with 20B tokens pretrained) with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Validation Perplexity (PPL) of various scales of RoM and Mamba models with different [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Validation perplexity of various evaluation sequence lengths for models pre-trained with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Architectures overview of all configurations and baseline models. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 7 canonical work pages

  1. [1]

    Blackmamba: Mixture of experts for state-space models.arXiv preprint arXiv:2402.01771, 2024

    Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models.arXiv preprint arXiv:2402.01771, 2024

    arXiv 2024

  2. [2]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  3. [3]

    Xception: Deep learning with depthwise separable convolutions

    François Chollet. Xception: Deep learning with depthwise separable convolutions. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017

    2017

  4. [4]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  5. [5]

    Switchhead: Accelerating transformers with mixture-of-experts attention.ArXiv, abs/2312.07987, 2023

    R’obert Csord’as, Piotr Piekos, Kazuki Irie, and Jürgen Schmidhuber. Switchhead: Accelerating transformers with mixture-of-experts attention.ArXiv, abs/2312.07987, 2023

    arXiv 2023

  6. [6]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    arXiv 2024

  7. [7]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. InInternational conference on machine learning, pages 933–941. PMLR, 2017

    2017

  8. [8]

    Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficien...

    arXiv 2024

Show all 49 references
  1. [9]

    Deepseek-v3 technical report.arXiv preprint arXiv: 2412.19437, 2024

    DeepSeek-AI. Deepseek-v3 technical report.arXiv preprint arXiv: 2412.19437, 2024

    2024 arXiv

  2. [10]

    Xin Dong, Y . Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Mahabaleshwarkar, Shih- Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. ArXiv, abs/24...

    2024 arXiv

  3. [11]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

  4. [12]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.Neural networks, 107:3–11, 2018

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.Neural networks, 107:3–11, 2018

    2018

  5. [13]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  6. [14]

    MegaBlocks: Efficient Sparse Training with Mixture-of-Experts.Proceedings of Machine Learning and Systems, 5, 2023

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts.Proceedings of Machine Learning and Systems, 5, 2023

    2023

  7. [16]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 11

    2023 arXiv

  8. [17]

    On the parameterization and initialization of diagonal state space models.Advances in Neural Information Processing Systems, 35:35971–35983, 2022

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models.Advances in Neural Information Processing Systems, 35:35971–35983, 2022

    2022

  9. [18]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  10. [19]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021

    2021

  11. [20]

    Diagonal state spaces are as effective as structured state spaces.Advances in Neural Information Processing Systems, 35:22982–22994, 2022

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces.Advances in Neural Information Processing Systems, 35:22982–22994, 2022

    2022

  12. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  13. [22]

    Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts.Neural computation, 3(1):79–87, 1991

    1991

  14. [23]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  15. [24]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960

    1960

  16. [25]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  17. [26]

    Zettlemoyer, and Lili Yu

    Weixin Liang, Junhong Shen, Genghan Zhang, Ning Dong, Luke S. Zettlemoyer, and Lili Yu. Mixture-of-mamba: Enhancing multi-modal state-space models with modality-aware sparsity. 2025

    2025

  18. [27]

    Jamba: A hybrid transformer-mamba language model.ArXiv, abs/2403.19887, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Haim Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen,...

    2024 arXiv

  19. [28]

    Bridging discrete and backpropagation: Straight-through and beyond

    Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropagation: Straight-through and beyond. InarXiv:2304.08612 [cs], 2023

    2023 arXiv

  20. [29]

    Sparse backpropagation for moe training.arXiv preprint arXiv:2310.00811, 2023

    Liyuan Liu, Jianfeng Gao, and Weizhu Chen. Sparse backpropagation for moe training.arXiv preprint arXiv:2310.00811, 2023

    2023 arXiv

  21. [30]

    Megalodon: Efficient llm pretraining and inference with unlimited context length.Advances in Neural Information Processing Systems, 37:71831–71854, 2024

    Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length.Advances in Neural Information Processing Systems, 37:71831–71...

    2024

  22. [31]

    Long range language modeling via gated state spaces.arXiv preprint arXiv:2206.13947, 2022

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces.arXiv preprint arXiv:2206.13947, 2022

    2022 arXiv

  23. [32]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv: 2501.08313, 2025

    MiniMax. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv: 2501.08313, 2025

    2025 arXiv

  24. [33]

    Leave no context behind: Efficient infinite context transformers with infini-attention.arXiv preprint arXiv:2404.07143, 101, 2024

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention.arXiv preprint arXiv:2404.07143, 101, 2024

    2024 arXiv

  25. [34]

    Gpt-4 technical report.PREPRINT, 2023

    OpenAI. Gpt-4 technical report.PREPRINT, 2023. 12

    2023

  26. [35]

    The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  27. [36]

    Can mamba learn how to learn? a comparative study on in-context learning tasks.arXiv preprint arXiv:2402.04248, 2024

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks.arXiv preprint arXiv:2402.04248, 2024

    2024 arXiv

  28. [37]

    Moe- mamba: Efficient selective state space models with mixture of experts.ArXiv, abs/2401.04081, 2024

    Maciej Pi’oro, Kamil Ciebiera, Krystian Kr’ol, Jan Ludziejewski, and Sebastian Jaszczur. Moe- mamba: Efficient selective state space models with mixture of experts.ArXiv, abs/2401.04081, 2024

    2024 arXiv

  29. [38]

    Hyena hierarchy: Towards larger convolutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. InInternational Conference on Machine Learning, pages 28043–28078. PMLR, 2023

    2023

  30. [39]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522, 2024

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522, 2024

    2024 arXiv

  31. [40]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  32. [41]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  33. [42]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  34. [43]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023

    2023

  35. [44]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  36. [45]

    Selective structured state-spaces for long-form video understanding

    Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6387–6397, 2023

    2023

  37. [46]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. InInternational Conference on Machine Learning, pages 10524–10533. PMLR, 2020

    2020

  38. [47]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    2024 arXiv

  39. [48]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  40. [49]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  41. [50]

    Mixture of attention heads: Selecting attention heads per token

    Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. InConference on Empirical Methods in Natural Language Processing, 2022. 13 A Appendix A.1 Detailed Comparisons Results for Potential...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.