Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Rethinking the long-range dependency in Mamba/SSM and transformer models

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-space models provably forget the past exponentially; transformers need not—and a new interaction term lets SSMs do the same.

desk verdict The LRD comparison restates known linear-RNN behavior and the new interaction-based SSM is a plausible idea, but Theorem 4 does not prove stability and the abstract/conclusion overstate it. read the letter →

arxiv 2509.04226 v1 pith:YKM5AKY7 submitted 2025-09-04 cs.LG

Cong Ma , Kayvan Najarian This is my paper
classification cs.LG
keywords long-rangedependencystate-spacemodelsMambatransformersattentionexponentialdecayhidden-statestabilityGaussiantailbounds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain, from theory, why state-space models (SSMs) such as Mamba and transformers differ in long-range dependency. Using a definition of long-range dependency as the derivative of a later hidden state with respect to an earlier input, it proves that SSM/Mamba sensitivity decays exponentially with the time gap, because each step multiplies by a transition matrix whose eigenvalues are at most one. A single attention layer, by contrast, is not forced into exponential decay, since attention weights can grow or reorganize with distance. To combine SSM efficiency with transformer-like flexibility, the paper adds an input-dependent interaction term to the SSM recursion, shows empirically that the new recursion removes the exponential decay, and proves a probabilistic stability bound under Gaussian inputs for a special eigenvector-aligned case.

What carries the argument

The paper's operative object is the sensitivity matrix $\mathrm{LRD}(t+k,t) = \frac{d h_{t+k}}{d x_t}$ (Eq. 8), a discrete analog of the RNN memory function. The exponential-decay proof uses the identity that Mamba's transition matrices are matrix exponentials $\exp(\Delta_i A)$, so a time-ordered product collapses to $\exp((\sum \Delta_i) A)$, whose spectral norm is controlled by the largest eigenvalue $\lambda_1$ of $A$. The proposed architecture centers on the rank-one input-dependent update $\bar A_t + G x_t x_t^T W^T$ inserted into the SSM recursion; the stability analysis in Theorem 4 reduces to bounding products of scalar factors $\lambda_H + \gamma x_i^2$ for i.i.d. standard Gaussian $x_i$, using sub-exponential tail bounds on sums of $\log(\lambda_H + \gamma x_i^2)$.

What would settle it

Train or randomly instantiate a Mamba and compute the Jacobian norms $\left\| \frac{d h_{t+k}}{d x_t} \right\|$ over many random input sequences; if any sequence shows a non-exponential decay, the universal bound in Corollary 1 fails in practice. Then run the proposed update with $G$ and $W$ randomly drawn, not aligned with any eigenvector of $\bar A$, streaming standard-Gaussian scalar inputs with $\lambda_H + \gamma < 1$; if $\| h_t \|$ diverges, the stability certificate does not cover the generic configuration.

Watch

Extended reading notes

Core claim

The central claim is that long-range dependency, formalized as the norm of $\mathrm{LRD}(t+k,t) = \frac{d h_{t+k}}{d x_t}$, decays exponentially with $k$ for SSM/Mamba models but is not constrained to do so for transformers. Corollary 1 proves the decay: with $\lambda_1$ the largest eigenvalue of $A$, $\| \mathrm{LRD}(t+k+1,t) \|_2 \leq \exp(\lambda_1 \Delta_{t+k+1}) \| \mathrm{LRD}(t+k,t) \|_2$, so the memory of a past input is suppressed at a rate set by the transition matrix. For transformers, Theorem 1 expresses LRD as a sum of an attention-weight derivative term and the direct weight $w(t,t+k) W_V$, which can increase with distance when interactions strengthen, so no exponential cap applies. The paper's proposed recursion $h_t = (\bar A_t + G x_t x_t^T W^T)$ h_{

Load-bearing premise

The paper's comparison assumes long-range dependency is best measured by how much a later hidden state changes when an earlier input is nudged; if memory should instead be measured by stored information or output accuracy, the exponential-decay result and the flexibility comparison do not follow.

Editorial extensions

If this is right

  • A single SSM/Mamba layer has a provable ceiling on how far back an input can influence hidden states: every additional time step multiplies the influence by at most exp(λ_1 Δ), so long-range tasks cannot be solved by simply lengthening the memory of one layer.
  • Transformers can in principle express LRD that stays flat or grows with distance, so their advantage on long-range tasks is not just empirical; it is a structural difference in how attention weights can reorganize.
  • Hybrid transformer-SSM architectures are a natural consequence of the theory: SSM supplies fast, stable medium-range memory, and attention supplies unconstrained long-range interactions.
  • For the proposed interaction update, stability is not automatic; the paper's condition λ_H+γ<1 together with the Gaussian tail bound gives a high-probability guarantee that hidden states do not diverge, but only in the special case analyzed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exponential-decay bound is proved for the single linear recurrence; stacked SSMs with nonlinear layers could in principle combine layers to extend memory, so the bound should be read as a property of one transition, not of a full deep network.
  • Because the stability certificate requires G and W to be scalings of the same eigenvector of \bar A, a practical design would be to enforce or softly regularize that alignment during training, or to prove a bound for random G,W; neither is in the paper.
  • The derivative-based measure captures local sensitivity, not information storage; on tasks where recall means retrieving a specific stored token, a mutual-information or retrieval-accuracy measure might rank Mamba and transformers differently than the LRD norm does.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a derivative-based definition of long-range dependency (LRD), LR D(t+k,t)=dh_{t+k}/dx_t, and uses it to compare SSM/Mamba and transformer models. It claims SSM/Mamba LRD decays exponentially with time gap while transformer LRD is not constrained to exponential decay, proposes a new state update with an 'interaction' term (Eqs. 13-14), gives unrolled forms (Theorems 2-3), provides a simulation of LRD norms (Fig. 1), and attempts to prove stability of the new recursion under Gaussian inputs in a special case (Theorem 4). The conclusion states the new model 'addressed the fundamental limitation' of SSM/Mamba and that stability guarantees hidden states will not approach infinity.

Significance. If the results were correct, the paper would contribute a formal framework for comparing sequence-model memory and a new linear-time architecture with attention-like flexibility. The LRD definition is clear and the unrolling algebra in Theorem 2 is correct. The proposed interaction term is a reasonable idea worth exploring. However, the paper's central stability theorem does not establish hidden-state stability even under its own assumptions, the transformer LRD formula contains a missing term, the exponential-decay bound in Corollary 1 requires an unstated normality condition, and the claimed practical advantage is supported neither by theory nor by any downstream evaluation. The theoretical contribution is therefore not yet at the level claimed by the abstract and conclusion.

major comments (5)
  1. [Section 3.1, Corollary 1, Eq. (11)] The bound ||exp(τ A)||_2 ≤ exp(λ_1 τ) is not valid for an arbitrary matrix A. It holds only when A is normal or, more generally, when the eigenbasis is well conditioned. SSM/Mamba transition matrices, including HiPPO-initialized A, are not guaranteed to be normal. Thus the per-time-step exponential decay claim as stated is incorrect; at most an asymptotic spectral-radius statement can be made. The proof needs an explicit normality or diagonalizability assumption, or a different norm.
  2. [Section 4.2, Theorem 4 and Eq. (18)] Theorem 4 bounds only the product ∏_{i=1}^t (λ_H + γ x_i^2), a single diagonal factor of the transition product. It does not bound the full hidden state. Under the paper's own assumption λ_i ≤ 1, λ_1 = 1 is allowed. In that case the first component in Eq. (18) is h_{0,1} + Σ_{i=1}^t (P^T B)_1 x_i (plus similar terms), which, for generic B, is a random walk with unbounded variance. The theorem therefore does not prove that hidden states 'will not approach infinity'; indeed the stated conclusion is false in an allowed parameter regime. A valid stability proof must require λ_i < 1 for all i and must control all eigen-directions and the sum over inputs, not just one product.
  3. [Section 4.2, simultaneous diagonalizability assumption and abstract] The stability theorem is proved only under the assumption that G x_t x_t^T W^T is simultaneously diagonalizable with \bar A, which forces G and W to be scalings of the same eigenvector of \bar A. This is a very restrictive structural condition that no trained or randomly initialized model will generally satisfy. The abstract's phrase 'prove its stability under a standard Gaussian distribution of the input data' omits this condition entirely. Section 5 itself concedes that general stability is 'unknown'. The paper should either restrict the headline claim explicitly to the special case or provide a stability analysis that does not depend on eigenvector alignment.
  4. [Section 3.2, Theorem 1, Eq. (12)] The derivative formula for transformer LRD is missing a term. Since h_{t+k} = Σ_{i=1}^{t+k} w(i,t+k) W_V x_i, the derivative with respect to x_t includes Σ_{i=1}^{t+k} (∂w(i,t+k)/∂x_t) W_V x_i plus the direct term w(t,t+k)W_V. The theorem's summation stops at t+k−1, omitting the i=t+k term (∂w(t+k,t+k)/∂x_t) W_V x_{t+k}. This is a concrete error in a central formula; it should be corrected even though the qualitative flexibility claim may survive.
  5. [Introduction, Section 4.1, and Section 5] The introduction states that the proposed new model 'outperforms SSM', and the conclusion says the model 'addressed the fundamental limitation' of Mamba. However, the only supporting evidence is a single simulated LRD-norm curve in Figure 1 with no parameter details or error bars, and no task-based experiments. Section 5 explicitly acknowledges that 'evaluation using existing benchmarking datasets is needed'. The unsupported performance claim should be removed or substantiated with experiments.
minor comments (5)
  1. [Section 3.1] The notation 'λ_1, the largest eigenvalue of A' is ambiguous for non-symmetric A with complex eigenvalues; the relevant quantity is the largest real part. The phrase 'non-positive if the SSM is stable' should also be clarified for discrete vs. continuous stability.
  2. [Section 4, Theorem 3] The theorem is stated before the scalar-input assumption is introduced, but the derivative 2 x_t G W^T in Eq. (16) is only valid for scalar x_t. For vector inputs the derivative of G x_t x_t^T W^T is a third-order tensor. The scalar assumption should be stated before Theorem 3 or the theorem should be restricted accordingly.
  3. [Figure 1] The simulation in Figure 1 needs details: matrix dimensions, distributions for A, B, G, W, the value of Δ, the random seed, and error bars over multiple trials. As presented it is a single-instance illustration, not an empirical validation.
  4. [Appendix, Lemma 1 proof] The lower-tail argument for log(c+γX^2) contains a somewhat informal comparison involving the chi-square CDF and constants; it would be clearer to state the exact sub-exponential parameters and to avoid reusing 'c' for both a constant and the eigenvalue λ_H.
  5. [Throughout] There are several typographical errors ('fundemantal', 'srong', 'LRDtransformerpt+k,tq') and inconsistent use of Δ_t vs. Δ_t=1. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: definitions and theorems are self-contained; stability overclaim is a proof gap, not circularity.

full rationale

The paper's derivation chain is mathematically self-contained rather than circular. LRD is defined in Eq. (8) as a derivative; Corollary 1 then computes this derivative from the linear SSM recurrence (Eqs. (2), (10)) and applies sub-multiplicativity of the matrix exponential. This is a direct theorem, not a restatement of the definition. The transformer LRD formula (Theorem 1) is likewise obtained by differentiating Eq. (5). The new interaction update (Eqs. (13)-(14)) is an explicitly proposed ansatz; its ability to avoid exponential decay is explained in Sec. 4.1 as a consequence of adding a rank-one term to the transition matrix, and is illustrated by simulation (Fig. 1) rather than asserted as a prediction from a fit. There are no load-bearing self-citations (the references to memory-function decay [36] are external). The only concern is the stability claim in Sec. 4.2/Conclusion: Theorem 4 proves a tail bound on the scalar product ∏(λ_H+γ x_i^2) under restrictive assumptions (simultaneous diagonalizability, scalar inputs, Δ_t=1, λ_H strictly positive, λ_H+γ<1), but the statement that 'most of the terms in ht will diminish' would require controlling the sum over inputs and the other eigendirections in Eq. (18); the theorem alone does not establish this, and the paper's Future Work concedes that general stability is unknown. This is an unsupported inference/overclaim (a correctness risk), not a circular reduction: the theorem's bound is not defined in terms of the conclusion, and no fitted parameter is renamed as a prediction. Hence circularity score 0.

Assumptions & free parameters 3 free parameters · 7 assumptions · 1 invented entities

The results depend on (i) the derivative-based LRD definition, a domain choice about what 'capability' means; (ii) normality/non-positive-eigenvalue structure of A for the decay bound in Corollary 1 (satisfied by Mamba's diagonal A); (iii) the Section 4.2 special-case assumptions (simultaneous diagonalizability of G and W with A, scalar inputs, Δ=1, λ_H>0, λ_H+γ<1) that make the stability conclusion operative; and (iv) standard concentration inequalities. The hand-picked parameter pair λ_H+γ=0.999 is a free choice that the stability reading relies on.

free parameters (3)
  • λ_H and γ (eigenvalue offset pair) = λ_H=0.9, γ=0.099 and λ_H=0.5, γ=0.499 (both give λ_H+γ=0.999)
    Hand-chosen in Sec 4.2/Fig 2 so that λ_H+γ<1, the condition under which the product in Theorem 4 decays; this choice, not the theorem itself, is what makes the 'stable' reading go through.
  • Simulation matrices A, B and vectors G, W = Not specified (no seed)
    Figure 1's non-monotonic LRD depends on an unseeded random draw of the transition matrix, input map, and interaction vectors; different draws would give different peak locations.
  • Time step Δ_t = 1 (fixed)
    Theorem 4 and Fig 2 assume fixed Δ_t=1, excluding Mamba's input-dependent Δ_t from the stability analysis.
assumptions (7)
  • domain assumption LRD capability is measured by ||d h_{t+k}/d x_t|| (Eq. 8)
    The whole comparison rests on this definition; it measures input sensitivity, not trained memorization capacity, and alternative definitions would change the conclusions.
  • domain assumption All transition matrices exp(Δ_j A) commute, so the product equals exp((Σ Δ_j) A)
    Corollary 1's product-to-exponential step uses pairwise commutation; true for Mamba's diagonal A with input-dependent Δ_j, but stated as if generic.
  • domain assumption ||exp(Δ A)||_2 ≤ exp(λ_1 Δ) with λ_1 the largest eigenvalue of A
    Requires A to be normal (or a logarithmic-norm bound); holds for Mamba's diagonal A but not for general A as stated in Corollary 1.
  • ad hoc to paper G x_t x_t^T W^T is simultaneously diagonalizable with \bar A; G and W are scalings of the same eigenvector
    Section 4.2; the stability theorem only covers this configuration, which no randomly initialized or trained model will satisfy, so Theorem 4's scope is a narrow special case.
  • domain assumption Inputs x_1,...,x_T are i.i.d. standard Gaussian
    Needed for the chi-square/sub-exponential analysis in Lemma 1 and Theorem 4; real sequence data is not Gaussian i.i.d.
  • ad hoc to paper λ_H strictly positive and, for the decay conclusion, λ_H + γ < 1
    Theorem 4 is a tail bound for any λ_H>0, but the 'hidden states will not explode' reading requires the parameter condition the authors choose in Fig 2 (λ_H+γ=0.999).
  • standard math Standard chi-square and sub-exponential tail bounds (Vershynin, Wainwright)
    Appendix Lemma 1 and Theorem 4 rely on these concentration results; appropriately cited [40,41].
invented entities (1)
  • Interaction-modulated state transition (h^T W x) G x
    purpose: Adds attention-like input dependence to the SSM transition to break the exponential LRD decay (Eq 13-14)
    The only support is a synthetic unseeded simulation with hand-chosen matrices; the model is never trained or evaluated on any benchmark, so no independent evidence exists. Structurally resembles delta-rule linear RNN memory updates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking the long-range dependency in Mamba/SSM and transformer models." pith.science (2026). https://pith.science/paper/YKM5AKY7

@misc{pith2026250904226,
  author       = {Pith},
  title        = {Pith review of: Rethinking the long-range dependency in Mamba/SSM and transformer models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKM5AKY7}},
  note         = {Machine review of arXiv:2509.04226}
}
read the original abstract

Long-range dependency is one of the most desired properties of recent sequence models such as state-space models (particularly Mamba) and transformer models. New model architectures are being actively developed and benchmarked for prediction tasks requiring long-range dependency. However, the capability of modeling long-range dependencies of these models has not been investigated from a theoretical perspective, which hinders a systematic improvement on this aspect. In this work, we mathematically define long-range dependency using the derivative of hidden states with respect to past inputs and compare the capability of SSM and transformer models of modeling long-range dependency based on this definition. We showed that the long-range dependency of SSM decays exponentially with the sequence length, which aligns with the exponential decay of memory function in RNN. But the attention mechanism used in transformers is more flexible and is not constrained to exponential decay, which could in theory perform better at modeling long-range dependency with sufficient training data, computing resources, and proper training. To combine the flexibility of long-range dependency of attention mechanism and computation efficiency of SSM, we propose a new formulation for hidden state update in SSM and prove its stability under a standard Gaussian distribution of the input data.

Figures

Figures reproduced from arXiv: 2509.04226 by the authors.

Figure 1
Figure 1. LRD of new formula and SSM/Mamba using simulated matrices and inputs. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Histogram of 10000 samples of log ´śt i“1 pλH ` γx2 i q ¯ . (Left) λH “ 0.9, γ “ 0.099, t “ 10000. (Right) λH “ 0.5, γ “ 0.499, t “ 10000 Some future directions will further enhance the theoretical results. First, the stability condition only handles the special case when the weight vectors in the interaction term are scaling of the same eigenvector of the hidden state transition matrix, but the stability conditions… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    HST-HGN uses heterogeneous spatial-temporal hypergraph networks combined with bidirectional Mamba state space models to achieve state-of-the-art driver fatigue assessment from untrimmed videos while maintaining comput...

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [36]

    State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory

    Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. In Thirty-seventh Conference on Neural Information Processing Systems , November 2023

    work page 2023

  2. [1]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  3. [2]

    On the properties of neural machine translation: Encoder-decoder approaches

    Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014

    arXiv 2014

  4. [3]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    2017

  5. [4]

    A comprehensive survey on applications of transformers for deep learning tasks.Expert Systems with Applications, 241:122666, 2024

    Saidul Islam, Hanae Elmekki, Ahmed Elsebai, Jamal Bentahar, Nagat Drawel, Gaith Rjoub, and Witold Pedrycz. A comprehensive survey on applications of transformers for deep learning tasks.Expert Systems with Applications, 241:122666, 2024

    work page 2024

  6. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171–4186, 2019

    work page 2019

  7. [6]

    Cross-lingual language model pretraining.Advances in neural information processing systems, 32, 2019

    Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining.Advances in neural information processing systems, 32, 2019

    2019

  8. [7]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

Show all 41 references
  1. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  2. [9]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 8 Long-range dependency of Mamba and transformers

    2021 arXiv

  3. [10]

    Medical transformer: Gated axial-attention for medical image segmentation

    Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel. Medical transformer: Gated axial-attention for medical image segmentation. In International conference on medical image computing and computer-assisted intervention, pages 36–46. Springer, 2021

    2021

  4. [11]

    Ds-transunet: Dual swin transformer u-net for medical image segmentation

    Ailiang Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang, Guangming Lu, and David Zhang. Ds-transunet: Dual swin transformer u-net for medical image segmentation. IEEE Transactions on Instrumentation and Measurement , 71:1–15, 2022

    2022

  5. [12]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  6. [13]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022

    2022 arXiv

  7. [14]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021

    2021

  8. [15]

    scgpt: toward building a foundation model for single-cell multi-omics using generative ai

    Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scgpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature methods, 21(8):1470–1480, 2024

    2024

  9. [16]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  10. [17]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  11. [18]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156–5165. PMLR, 2020

    2020

  12. [19]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  13. [20]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  14. [21]

    Simplified state space layers for sequence modeling

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022

    2022 arXiv

  15. [22]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  16. [23]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  17. [24]

    U-mamba: Enhancing long-range dependency for biomedical image segmentation

    Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024

    2024 arXiv

  18. [25]

    Vm-unet: Vision mamba unet for medical image segmentation

    Jiacheng Ruan, Jincheng Li, and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402.02491, 2024

    2024 arXiv

  19. [26]

    Fd-vision mamba for endoscopic exposure correction

    Zhuoran Zheng and Jun Zhang. Fd-vision mamba for endoscopic exposure correction. arXiv preprint arXiv:2402.06378, 2024

    2024 arXiv

  20. [27]

    Semi-mamba-unet: Pixel-level contrastive and pixel-level cross-supervised visual mamba-based unet for semi-supervised medical image segmentation

    Chao Ma and Ziyang Wang. Semi-mamba-unet: Pixel-level contrastive and pixel-level cross-supervised visual mamba-based unet for semi-supervised medical image segmentation. arXiv preprint arXiv:2402.07245, 2024

    2024 arXiv

  21. [28]

    Medmamba: Vision mamba for medical image classification

    Yubiao Yue and Zhenzhang Li. Medmamba: Vision mamba for medical image classification. arXiv preprint arXiv:2403.03849, 2024

    2024 arXiv

  22. [29]

    Clinicalmamba: A generative clinical language model on longitudinal clinical notes

    Zhichao Yang, Avijit Mitra, Sunjae Kwon, and Hong Yu. Clinicalmamba: A generative clinical language model on longitudinal clinical notes. arXiv preprint arXiv:2403.05795, 2024

    2024 arXiv

  23. [30]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33:1474–1487, 2020

    2020

  24. [31]

    Block-biased mamba for long-range sequence processing

    Annan Yu and N Benjamin Erichson. Block-biased mamba for long-range sequence processing. arXiv preprint arXiv:2505.09022, 2025. 9 Long-range dependency of Mamba and transformers

    2025 arXiv

  25. [32]

    Remamba: Equip mamba with effective long-sequence modeling

    Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, and Dongyan Zhao. Remamba: Equip mamba with effective long-sequence modeling. arXiv preprint arXiv:2408.15496, 2024

    2024 arXiv

  26. [33]

    Spatial-mamba: Effective visual state space models via structure-aware state fusion

    Chaodong Xiao, Minghan Li, Zhengqiang Zhang, Deyu Meng, and Lei Zhang. Spatial-mamba: Effective visual state space models via structure-aware state fusion. arXiv [cs.CV], October 2024

    2024

  27. [34]

    SST: Multi-scale hybrid mamba-transformer experts for long-short range time series forecasting

    Xiongxiao Xu, Canyu Chen, Yueqing Liang, Baixiang Huang, Guangji Bai, Liang Zhao, and Kai Shu. SST: Multi-scale hybrid mamba-transformer experts for long-short range time series forecasting. arXiv [cs.LG], April 2024

    2024

  28. [35]

    Efficient hybrid long sequence modeling with state space augmented transformers

    Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis X Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient hybrid long sequence modeling with state space augmented transformers. In First Conference on Language Modeling, August 2024

    2024

  29. [37]

    Optimizing deeper transformers on small datasets

    Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon JD Prince, and Yanshuai Cao. Optimizing deeper transformers on small datasets. arXiv preprint arXiv:2012.15355, 2020

    2012 arXiv

  30. [38]

    Escaping the big data paradigm with compact transformers

    Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704, 2021

    2021 arXiv

  31. [39]

    Never train from scratch: Fair comparison of long-sequence models requires data-driven priors

    Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. arXiv preprint arXiv:2310.02980, 2023

    2023 arXiv

  32. [40]

    High-dimensional probability: An introduction with applications in data science , volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018

    2018

  33. [41]

    Wainwright

    Martin J. Wainwright. Basic tail and concentration bounds , page 21–57. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. 10 Long-range dependency of Mamba and transformers Appendix S1 Proof of Theorem 2 Proof. We prove this theor...

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.