Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing

Jiajun Zhu; Pan Li; Peihao Wang; Pragya Srivastava; Ruisi Cai; Yuehao Wang; Zhangyang Wang

REVIEW 3 major objections 4 minor 2 cited by

Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read State space models are inherently recency-biased, and depth scaling is blocked by over-smoothing; polarizing two transition channels to 0 and 1 mitigates both.

desk verdict The recency-bias result genuinely extends to selective SSMs and the polarization trick works empirically, but the over-smoothing theorem's proof relies on a false lemma, so the strong claims outrun the current math. read the letter →

arxiv 2501.00658 v2 pith:Z2RFIHFH submitted 2024-12-31 cs.LG

Peihao Wang , Ruisi Cai , Yuehao Wang , Jiajun Zhu , Pragya Srivastava , Zhangyang Wang , Pan Li This is my paper · ORCID

classification cs.LG

keywords statespacemodelsrecencybiasover-smoothinglong-rangedependenciesassociativerecallMambalinearattentionpolarization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structured state space models (SSMs) are often advertised as good at long sequences, but this paper argues they are intrinsically biased toward recent tokens. For any SSM whose state transition matrix is diagonal with entries in $(0,1)$, the influence of an input token on a later output decays exponentially with distance; Mamba and many linear-attention models fall in this class, so their long-range recall is limited regardless of selection mechanisms. The paper also proves that stacking SSM layers smooths token representations, with pairwise differences in memory states bounded by a factor smaller than one, which explains why depth scaling eventually stops helping. To address both limits at once, polarize two transition channels: one fixed at $1$ to preserve history and one fixed at $0$ to keep a fresh token channel, mitigating recency bias and over-smoothing simultaneously. Associative-recall experiments show the fix raises accuracy for distant key-value pairs and lets deeper models help instead of hurt.

What carries the argument

The workhorse is the closed-form parallel expansion of the recurrence: $h_t = \sum_{s<t} (\prod_{r=s+1}^{t} A_r)\Delta_s b_s(x_s) + \Delta_t b_t(x_t)$, which rewrites each SSM as a weighted sum over past tokens. When $A_t$ is diagonal with entries in $(0,1)$, the cumulative product decays as $A_{\max}^{t-s}$, producing the exponential influence bound of Theorem 3.1; the same recurrence under $A_t+\Delta_t \le 1$ produces the over-smoothing bound of Theorem 4.2. The proposed polarization mechanism places one diagonal entry at the constant value $1$ (an infinite-memory channel) and another at $0$ (a fresh-token-only channel), forcing the spectrum of $A_t$ to span both extremes and thereby avoiding both forgetting and over-mixing. In continuous-time S4, the same analysis identifies the layer as a low-pass filter, which is why repeated application flattens token differences.

What would settle it

Measure, on a trained Mamba, the empirical influential scores $\log|\partial y_t/\partial x_s|$ and check whether they follow the predicted slope $\log(A_{\max}^{-1})$; separately, record whether any channel has $(A_t)_{n,n}+\Delta_t > 1$. If the score does not decay exponentially with distance, or if over-smoothing fails to appear when many channels violate non-expansiveness, the central claims are wrong.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central result is Theorem 3.1: an SSM of the form $h_t = A_t h_{t-1} + \Delta_t b_t(x_t)$, $y_t = c_t(h_t)$, with diagonal $A_t \in (0,1)^{N\times N}$, has influential score $|\partial y_t/\partial x_s| = O(\exp(-\kappa(t-s)))$ with $\kappa = \Theta(\log(A_{\max}^{-1}))$, so distant inputs are exponentially under-weighted. Theorem 4.2 then shows that under a non-expansiveness condition and centered inputs, $\max_{t,s}\|h_t - h_s\|_\infty \le (1 - A_{\min}^{T-1}) \max_{t,s}\|b_t(x_t) - b_s(x_s)\|_\infty$, so deeper SSMs collapse token representations toward each other. The paper's proposed fix is polarization: pin one diagonal channel of $A_t$ to $1$ and one to $0$, leaving the rest learnable, which simultaneously preserves complete history and prevents over-mixing. Empirically, on associative recall, this raises accuracy at 256 key-value pairs from 33.52\% to 81.56\% for a four-layer Mamba, and makes deeper models strictly beneficial.

Load-bearing premise

The load-bearing premise is that real SSM recurrence operators are diagonal with entries in $(0,1)$ and satisfy the non-expansive condition $A_t+\Delta_t \le 1$; if actual trained models violate either, the theorems' conclusions are not guaranteed.

Editorial extensions

If this is right

If Theorem 3.1 holds, any practical SSM with sub-unit diagonal transitions, including Mamba, RetNet, RWKV, and GLA, will show exponential loss of retrieval accuracy once the sequence exceeds its effective memory, no matter how the selection mechanism is tuned.
If Theorem 4.2 holds, depth scaling alone cannot fix long-context modeling: beyond a depth threshold, added layers make token representations more alike, so validation performance plateaus and then declines, as the paper's scaling curves show.
Polarizing two channels is a cheap architectural change that does not alter gradient flow in the remaining channels, so it should transfer to any diagonal-SSM variant that wants long-range recall without over-smoothing.
Recency bias is a robustness liability: because trailing tokens dominate outputs, an attacker can achieve high success by corrupting only the last few tokens of an input sequence, while leaving leading tokens intact.
Because the theorems cover input-dependent $A_t$ and $b_t$, selection mechanisms such as Mamba's do not exempt a model from recency or over-smoothing; content-aware gating changes the constants but not the exponential form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not fit $\kappa$ from data; one could measure whether trained slopes of $\log|\partial y_t/\partial x_s|$ track $\log(A_{\max}^{-1})$ in practice.
The polarization trick suggests a general design rule for recurrent models: keep one integrator channel and one instantaneous readout channel even when other decay factors are learned.
A natural next test is whether the same two-channel polarization helps on natural-language long-context benchmarks, not just synthetic associative recall.
The theory implies that for robustness, end-of-context tokens should see stronger adversarial effects; this could be verified directly on instruction-following language models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper extends recency-bias analysis to selective/input-dependent SSMs (Mamba) and proposes a simple two-channel polarization trick that produces large gains on associative recall. Second, the over-smoothing theorem is not proven: its Lemma D.6 is false, and the authors' claim that Mamba easily satisfies the theorem's assumptions is unverified.

The genuinely new piece is Theorem 3.1, which gives an exponential upper bound on the influence score for SSMs with input-dependent transition matrices. That is a real extension of Wang and Xue's result for linear SSMs. The paper also includes careful empirical work: influence-score plots, needle-in-a-haystack tests on a 7B Mamba, depth-scaling pretraining experiments, and associative recall. The polarization method is simple and its empirical gains are substantial (e.g., average recall jumps from 72 to 93 on 256 key-value pairs). Code is released. Credit where due: this is a useful empirical paper with a plausible diagnosis.

The soft spots, in order. Theorem 3.1 is an upper bound on |∂y_t/∂x_s|; it shows influence cannot decay slower than some rate, but it does not prove it actually decays at that rate. The abstract's phrase "rigorously demonstrate strong recency bias" is stronger than the math. More seriously, Theorem 4.2's proof relies on Lemma D.6, and that lemma is false. Take α1=α2=0.5, Δ1=Δ2=0.5, z1=-1, z2=100. The lemma's upper bound for s2 is 49.5, but the recurrence gives 49.75. The base case of the induction ignores Δ1. So the "inevitable over-smoothing" claim lacks rigorous support. The theorem may be repairable, but as written it does not stand. Also, the assumption (At)_nn + Δ_t ≤ 1 is asserted but not checked for Mamba; Δ_t is softplus, which is unbounded, so that premise is not obviously satisfied.

The security section extrapolates from CIFAR-10 pixel attacks to LLM jailbreaks with no direct experiments; I would treat that as speculation.

Who should read this? Anyone working on SSM theory or architecture design. The empirical findings and the polarization trick are valuable even if the theory needs work. For a conference referee, I would send it to review but with major revision: fix or replace Lemma D.6, verify (or drop) the Mamba assumption, and soften the claims. The core diagnosis is probably right, but the paper's strongest language currently exceeds its proof.

Referee Report

3 major / 4 minor

Summary. The paper studies two purported bottlenecks of structured state space models (SSMs): recency bias and over-smoothing. Theorem 3.1 gives an exponential upper bound on the influence score |∂y_t/∂x_s| for diagonal state transition matrices with entries in (0,1), which the authors interpret as a rigorous demonstration that SSMs are recency-biased. Supporting experiments include a needle-in-a-haystack evaluation, adversarial attacks on CIFAR-10, and influence-score measurements. Scaling experiments in Section 4.1 show that deeper Mamba models first help and then hurt with increasing depth; the paper attributes this to over-smoothing. Theorem 4.2 claims a bound on the spread of hidden states that shrinks with the context length, formalizing the over-smoothing tendency. The authors then propose "polarization": forcing one channel of A_t to 1 and one channel to 0, and report improved associative-recall accuracy, especially in deeper models. The paper also discusses HiPPO theory, selection mechanisms, complex parameterizations, and related work on GNN and transformer over-smoothing.

Significance. If both theorems were sound, the paper would give a principled account of why SSMs underperform on long-range retrieval and how depth exacerbates representation collapse, together with a simple, practical mitigation. The work is also valuable empirically: it ships code, carefully documents the training settings, and provides a rich set of observations (influence-score decay, inverted needle retrieval curves, trailing-token vulnerabilities, and associative-recall gains from polarization) that are of independent interest. The connection to HiPPO and to prior over-smoothing literature is thoughtful. However, the two theoretical pillars are not on equal footing: the recency result is a valid upper bound but is over-interpreted as a proof of actual decay, and the over-smoothing theorem's proof contains a false lemma. Because the polarization method is justified through these theorems, the theoretical underpinning of the proposed mitigation is weakened, although the empirical results in Table 2 remain suggestive.

major comments (3)

[Section 3.1, Theorem 3.1] The abstract and Section 3.1 claim that the paper "rigorously demonstrates" a strong recency bias, but Theorem 3.1 is only an upper bound: |∂y_t/∂x_s| = O(exp(-κ(t-s))). An upper bound does not prove that influence actually decays; it only states that distant tokens cannot have arbitrarily large influence. A model in which all tokens have uniformly small influence also satisfies the bound. The empirical evidence in Figure 1 and the needle-in-a-haystack results do show actual decay, but the theorem itself does not. The authors should either prove a matching lower bound under suitable non-degeneracy assumptions (for instance, lower-bounding |∂(Δ_s b_s(x_s))/∂x_s| away from zero along some direction) or change the wording from "rigorously demonstrate strong recency bias" to a claim about an at-most-exponential decay of influence.
[Appendix D.2.2, Lemma D.6 and Theorems D.5/4.2] Lemma D.6, which is the key step in the proof of Theorem 4.2, is false as stated. Counterexample: take T=2, α_1=α_2=0.5, Δ_1=Δ_2=0.5, z_1=-1, z_2=100, and s_0=0. These values satisfy both assumption (i), since α_t+Δ_t=1, and assumption (ii), since min z=-1≤0≤100=max z. With m=-1, M=100, and p=1 (because z_1=m), the claimed upper bound for s_2 is A_min·p·m + (1-A_min·p)·M = 0.5·1·(-1) + (1-0.5·1)·100 = 49.5. The recurrence gives s_1 = -0.5 and s_2 = 0.5·(-0.5)+0.5·100 = 49.75, which exceeds the claimed bound. Moreover, the base case for t=1 already fails: the lemma's upper bound is z_1 = -1, while s_1 = -0.5. Consequently the induction in the proof of Lemma D.6 is invalid, and inequality (4) in Theorem 4.2 is not established. The over-smoothing conclusion in Section 4.2 therefore needs a corrected lemma with different hypotheses (for example, a separate treatment of the first time step) or a different proof strategy.
[Section 4.2, applicability to Mamba (Eq. (3))] The paper asserts that conditions (i) (A_t)_n,n + Δ_t ≤ 1 and (ii) centered b_t inputs are "easily achieved" by prevalent SSMs, including Mamba. This is not verified for Mamba. In Mamba, Δ_t = softplus(W_Δ x_t) is unbounded above, and (A_t)_n,n = exp(Δ_t A_n,n) with A_n,n<0. For any Δ_t>1, the sum exp(Δ_t A_n,n)+Δ_t can exceed 1; for instance, with A_n,n=-1 and Δ_t=2, exp(-2)+2 > 1. The centered-input condition min_t b_t(x_t)_n ≤ 0 ≤ max_t b_t(x_t)_n is also a property of the learned matrix W_B and the input distribution, not a design guarantee. If these conditions are not checked for the actual models in Figure 5, Theorem 4.2 does not connect to the empirical Mamba models, and the claim that Mamba "inevitably" over-smooths with depth remains unsupported by the theory. The authors should either verify these conditions empirically on the trained models or explicitly frame Theorem 4.2 as a conditional statement whose assumptions may or may not hold in practice.

minor comments (4)

[Section E.5.1] The sentence "We also study two variants: 0-polarized and 1-polarized Mamba by only pretending or appending one or zero to At" contains a typo: "pretending" should presumably be "prepending".
[Section 4.2, sharpness measure] In the definition of E(x), the normalization denominator and the index N appear to refer to the number of tokens, which is T in the notation of the paper; please use a consistent symbol to avoid confusion.
[Appendix D.1, Eq. (7)] In the proof of Theorem D.2, the bound on ∂u_t/∂x_s extracts a uniform constant C_1 that depends on A^{-1}_r and ∂A_r/∂x_s, but the subsequent geometric-series step is valid only if the same bound applies uniformly over all r; this is true under the compactness assumption but could be stated more explicitly.
[Appendix B, linear attention (LA)] The formulation (LA) sets A_t = I, which lies on the boundary (A_t)_n,n=1 rather than in (0,1); Theorem 3.1 requires strict inequality. The paper should clarify whether the theorem extends to this boundary case or whether linear attention requires a separate argument.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the recency and over-smoothing bounds are derived from the stated SSM recurrence under explicit structural assumptions, and the polarization method is an application of those bounds that is then empirically validated, not a fitted or self-referential prediction.

full rationale

The paper's two central theoretical claims are derived, rather than assumed. Theorem 3.1 starts from Eq. (1), uses Lemma D.1's parallel-form expansion of the recurrence, and bounds the cumulative products of diagonal At by Amax to obtain the exponential decay of |∂yt/∂xs|. The conclusion is a consequence of the stated assumption At ∈ (0,1)N×N, not an input to the proof; the recency behavior is not inserted as a premise. Theorem 4.2 is likewise a contraction estimate obtained in the proof of Theorem D.5 and Lemma D.6 from the recurrence st = αt st−1 + Δt zt under the stated non-expansiveness and centered-input conditions. Whether Lemma D.6 is actually correct is a separate mathematical-correctness question; a false or incomplete proof would make the over-smoothing claim unsupported, but it would not make the derivation circular. The polarization proposal in Section 6 is explicitly motivated by the bounds: one channel set to 1 keeps Amax near 1 to slow recency decay, and one channel set to 0 keeps Amin near 0 to slow the smoothing rate. This is a design informed by the theorems, and its effectiveness is then measured on associative recall in Table 2; the polarized constants are not fitted parameters and the improved accuracies are not forced by the theory. No fitted value is later relabeled as a prediction. Citations to prior work are used as context or comparison, including Wang & Xue (2024) for the linear-SSM decaying-memory result and Wang et al. (2022) for transformer over-smoothing; the overlapping-author citation (Wang et al. 2022) appears in a related-work sentence about transformers and is not load-bearing for any SSM-specific conclusion. The paper is self-contained in its derivation chain: the theorems reduce to algebra on Eq. (1), and the empirical claims are external benchmarks. The reviewer concern that Mamba's softplus Δt may violate (At)n,n+Δt ≤ 1 is an unverified-assumption issue, not circularity. Overall, no step equates the inputs with the conclusions by construction, so the circularity score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest primarily on two structural assumptions: the diagonal bounded form of A_t (for recency) and the non-expansive recurrence condition (A_t)_{n,n} + Δ_t ≤ 1 (for over-smoothing). The first is standard, the second is asserted without verification for Mamba. The polarization values 0 and 1 are an architectural hyperparameter chosen by hand. No new physical or mathematical entities are introduced.

free parameters (1)

polarization values = 0 and 1 for two channels of A_t
The paper sets one state-transition channel to 0 and another to 1 by hand, motivated by the bounds in Theorems 3.1 and 4.2. Other values could be chosen; the paper does not derive that these extremes are optimal.

assumptions (5)

domain assumption SSM recurrence of the form h_t = A_t h_{t-1} + Δ_t b_t(x_t) with diagonal A_t in (0,1)
Section 2 states this form for S4, Mamba, and many linear attention models. Theorem 3.1 relies on the diagonal and bounded structure of A_t.
standard math Compact input space and continuous differentiability of (A_t,b_t,c_t,Δ_t)
Required for Theorem 3.1 to ensure bounded derivatives and to justify the upper bound. These are standard regularity conditions.
domain assumption A_t diagonal with negative real parts for continuous-time S4
Used in Proposition 4.1 to show low-pass filtering. This is the standard S4 setup.
ad hoc to paper (A_t)_{n,n} + Δ_t ≤ 1 and centered b_t inputs for Theorem 4.2
The paper states Mamba can easily satisfy this but does not verify it. For A diagonal with values like -1 and Δ=1, exp(-1)+1 > 1, so the assumption may be violated. The over-smoothing bound depends on this condition.
domain assumption Class token placed at the end of the sequence for CIFAR-10 experiments
Necessary for causal SSMs; this placement is what makes trailing tokens more influential in the robustness experiments. The choice is explained in Section 3.3 and Appendix E.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing." pith.science (2026). https://pith.science/paper/Z2RFIHFH

@misc{pith2026250100658,
  author       = {Pith},
  title        = {Pith review of: Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2RFIHFH}},
  note         = {Machine review of arXiv:2501.00658}
}

read the original abstract

Structured State Space Models (SSMs) have emerged as alternatives to transformers. While SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously demonstrate that they are inherently limited by strong recency bias. Our empirical studies also reveal that this bias impairs the models' ability to recall distant information and introduces robustness issues. Our scaling experiments then discovered that deeper structures in SSMs can facilitate the learning of long contexts. However, subsequent theoretical analysis reveals that as SSMs increase in depth, they exhibit another inevitable tendency toward over-smoothing, e.g., token representations becoming increasingly indistinguishable. This fundamental dilemma between recency and over-smoothing hinders the scalability of existing SSMs. Inspired by our theoretical findings, we propose to polarize two channels of the state transition matrices in SSMs, setting them to zero and one, respectively, simultaneously addressing recency bias and over-smoothing. Experiments demonstrate that our polarization technique consistently enhances the associative recall accuracy of long-range tokens and unlocks SSMs to benefit further from deeper architectures. All source codes are released at https://github.com/VITA-Group/SSM-Bottleneck.

Figures

Figures reproduced from arXiv: 2501.00658 by the authors.

**Figure 1.** Visualization of log influential scores log |∂yt/∂xs| versus distance (t−s). The proof can be found in Appendix D.1. The first two assumptions are standard and always satisfied. The third assumption also holds for most of SSMs discussed in Sec. 2. Therefore, Theorem 3.1 applies to numerous SSMs including S4 (Gu et al., 2021a; 2022a), Mamba (Gu & Dao, 2023), and many LAMs (Sun et al., 2023; Peng et al., 2023; Yang et… view at source ↗

**Figure 2.** Comparison between SSM and Transformer on the “Needle in a Haystack" benchmark. The left figure shows the retrieval accuracy of the Mamba-Codestral-7B model, while the right figure presents the retrieval accuracy of the Mistral-7B model. We present a heatmap where "full context length" refers to the total length of the document, and "needle position" denotes the relative position of the statement to be retrieved wit… view at source ↗

**Figure 3.** Results of target attack experiments on CIFAR-10, where “horse” is the target class. (a) and (b) present target attack success rates under two attack ratios. Lower success rates suggest higher robustness in the corresponding attack regions. leading and trailing tokens are corrupted with Gaussian noise. Among all the SSM family methods compared, the performance drops caused by trailing token corruption are significan… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: We empirically observe that deeper models become increasingly advantageous as the context length grows. However, beyond a certain depth, the performance of SSMs begins to plateau and eventually declines. more experiment details in Appendix E.4. The validation loss vers…

**Figure 5.** Figure 5: Visualization of feature smoothness across layers in pre-trained Mamba and Pythia. The y-axis represents the average pairwise differences among tokens. Mixer outputs (b) solely consider the Mamba or attention module, while Block outputs (c) include all other components…

**Figure 6.** Figure 6: Cumulative histogram of (Amax − Amin). The height of each bin represents the cumulative proportion of (Amax − Amin) less than or equal to the corresponding value on the x-axis. Although Amin ≈ 0 and Amax ≈ 1 could theoretically occur simultaneously, our empirical fin…

**Figure 7.** Figure 7: Visualization of influential scores for Mamba and Transformer models under [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

**Figure 8.** Figure 8: An illustration of our synthetic data. 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% Needle Position 10240 12288 14336 16384 18432 20480 22528 24576 Full Context Length 0.54 0.54 0.55 0.56 0.56 0.53 0.51 0.51 0.52 0.5 0.48 0.49 0.52 0.54 0.…

**Figure 9.** Figure 9: Detailed comparison between SSM and Transformer on the “Needle in a Haystack" [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]

**Figure 10.** Figure 10: Overall success rate of our target attack experiments on CIFAR-10, calculated by averaging [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale
cs.AI 2026-07 conditional novelty 6.0 of 10

Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
cs.LG 2025-06 conditional novelty 5.0 of 10

On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.

Reference graph

Works this paper leans on

79 extracted references · 25 canonical work pages · cited by 2 Pith papers

[1]

The hidden attention of mamba models

Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590,

arXiv
[5]

Decimamba: Exploring the length extrapolation potential of mamba

Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. arXiv preprint arXiv:2406.14528,

arXiv
[6]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

1901
[7]

Learning phrase representations using rnn encoder-decoder for statistical machine translation

Kyunghyun Cho. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078,

arXiv
[9]

Rethinking attention with performers

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794,

arXiv 2009
[10]

Transformer-xl: Attentive language models beyond a fixed-length context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860,

arXiv 1901
[12]

Griffin: Mix- ing gated linear recurrences with local attention for efficient language models

Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427,

arXiv
[13]

Bert: Pre-training of deep bidirectional transformers for language understanding

11 Published as a conference paper at ICLR 2025 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL,

work page 2025

Show all 79 references

[15]

Were rnns all we needed? arXiv preprint arXiv:2410.01201,

Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, and Hossein Hajimirsadegh. Were rnns all we needed? arXiv preprint arXiv:2410.01201,

arXiv
[16]

Hungry hungry hippos: Towards language modeling with state space models

Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052,

arXiv
[19]

Efficiently modeling long sequences with structured state spaces

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021a. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, a...

arXiv
[20]

Long short-term memory

12 Published as a conference paper at ICLR 2025 Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780,

2025
[22]

Repeat after me: Trans- formers are better than state space models at copying

Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying. arXiv preprint arXiv:2402.01032,

arXiv
[23]

Mistral 7b

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

arXiv
[24]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

2001 arXiv
[25]

Semi-supervised classification with graph convolutional networks

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907,

arXiv
[28]

Jamba: A hybrid transformer- mamba language model

Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403.19887,

arXiv
[29]

Longhorn: State space models are amortized online learners

Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners. arXiv preprint arXiv:2407.14207,

arXiv
[30]

Mega: moving average equipped gated attention

Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: moving average equipped gated attention. arXiv preprint arXiv:2209.10655,

arXiv
[31]

Megalodon: Efficient llm pretraining and inference with unlimited context length

Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801,

arXiv
[32]

The illusion of state in state-space models

William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819,

arXiv
[33]

In-context learning and induction heads

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895,

arXiv
[34]

Graph neural networks exponentially lose expressive power for node classification

13 Published as a conference paper at ICLR 2025 Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations (ICLR),

2025
[35]

Can mamba learn how to learn? a comparative study on in-context learning tasks

Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kang- wook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04248,

arXiv
[36]

Rwkv: Reinventing rnns for the transformer era

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV , et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

arXiv
[37]

Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892,

arXiv
[38]

Ignore previous prompt: Attack techniques for language models

Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527,

arXiv
[39]

Mechanistic design and scaling of hybrid architectures

Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXiv:2403.17844,

arXiv
[40]

Train short, test long: Attention with linear biases enables input length extrapolation

Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409,

arXiv
[41]

Hgrn2: Gated linear rnns with state expansion

Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904,

arXiv
[42]

Revisiting over-smoothing in bert from the perspective of graph

Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08625,

arXiv
[43]

Learning to (learn at test time): Rnns with expressive hidden states

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620,

arXiv
[44]

A length-extrapolatable transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554,

arXiv
[45]

Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

arXiv
[46]

Long range arena: A benchmark for efficient transformers

14 Published as a conference paper at ICLR 2025 Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006,

2025 arXiv
[47]

Understanding over-squashing and bottlenecks on graphs via curvature

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXiv preprint arXiv:2111.14522,

arXiv
[48]

Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel

Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Transformer dissection: a unified understanding of transformer’s attention via the lens of kernel. arXiv preprint arXiv:1908.11775,

1908 arXiv
[49]

The unreasonable effectiveness of the forget gate

Jos Van Der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint arXiv:1804.04849,

arXiv
[50]

An empirical study of mamba- based language models

Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba- based language models. arXiv preprint arXiv:2406.07887,

arXiv
[51]

Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice

Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. arXiv preprint arXiv:2203.05962,

arXiv
[52]

A non-asymptotic analysis of oversmoothing in graph neural networks

Xinyi Wu, Zhengdao Chen, William Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. arXiv preprint arXiv:2212.10701,

arXiv
[53]

On the role of attention masks and layernorm in transformers

Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. arXiv preprint arXiv:2405.18781, 2024a. Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention- based gr...

arXiv
[54]

Gated linear attention transformers with hardware-efficient training

Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635,

arXiv
[55]

Parallelizing linear transformers with the delta rule over sequence length

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484,

arXiv
[56]

The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. arXiv preprint arXiv:2402.04347, 2024a. Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point could mamba: Poin...

arXiv
[57]

Universal and transferable adversarial attacks on aligned language models

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,

arXiv
[58]

Arora et al

15 Published as a conference paper at ICLR 2025 A O THER RELATED WORK Despite the empirical success of SSMs in various long-range applications (Lieber et al., 2024; Zhu et al., 2024; Zhang et al., 2024b), their theoretical properties and limitations remain underexplored. Arora...

2023
[59]

Linear Attention

This reformulation is expected to extend to models not explicitly covered below, including but not limited to Megalodon (Ma et al., 2022; 2024), Hyena (Poli et al., 2023), HGRN2 (Qin et al., 2024), TTT (Sun et al., 2024), Longhorn (Liu et al., 2024), xLSTM (Beck et al., 2024),...

2019
[60]

Each layer of RetNet consists of a key, 16 Published as a conference paper at ICLR 2025 query, and value transformation, akin to linear attention

is another variant of LAMs, proposed as a successor to transformers given its remarkable performance. Each layer of RetNet consists of a key, 16 Published as a conference paper at ICLR 2025 query, and value transformation, akin to linear attention. In addition, it imposes a ne...

2025
[61]

Similar to Mamba (Gu & Dao, 2023), RetNet shares bt and ct across channels while assigning distinct ∆t for each channel when handling multi-channel inputs

is a (learnable) scalar, k, q : R → RN , v : R → R are linear functions. Similar to Mamba (Gu & Dao, 2023), RetNet shares bt and ct across channels while assigning distinct ∆t for each channel when handling multi-channel inputs. Gated Linear Attention (GLA). GLA (Yang et al.,

2023
[62]

Its computational mechanism can be encompassed by our formulation in Eq

introduces gating mechanism, originally from RNNs (Van Der Westhuizen & Lasenby, 2018), to LAMs. Its computational mechanism can be encompassed by our formulation in Eq. 1: (GLA) At = diag(α(xt)), bt(xt) = k(xt), c t(ht) = q(xt)⊤ht, ∆t = v(xt), where α : R → (0, 1)N converts i...

2018
[63]

can be formulated as GLA with input-independent At (Liu et al., 2024). RWKV . RWKV is a series of models that linearize attention computation (Peng et al., 2023; 2024). We focus on RWKV-4 (Peng et al.,

2024
[64]

Later in RWKV-5 and RWKV-6 (Peng et al., 2024), the normalizer is removed for numerical stability. Griffin. The recurrent unit in Griffin (De et al.,

2024
[65]

In particular, the dimension of ht in Griffin is equal to the dimension of xt

can be re-formulated as a kind of SSMs: (Griffin) At = diag (α(xt)), bt(xt) = diag (i(xt)), c t(ht) = ht, ∆t = diag p 1 − α(xt)2 , where i(xt) = sigmoid( Wxxt + bx) is an input gate, α is computed in log-space: log α(xt) = −ξ softplus(Γ) ⊙ sigmoid(Waxt + ba), ⊙ is Hadamard pro...

2020
[66]

This design has quickly become a standard backbone for various SSMs (Gu & Dao, 2023; Beck et al., 2024)

lies in the introduction of self-gating connections and locally shifting convolutions to improve in-context recall for state space models (SSMs). This design has quickly become a standard backbone for various SSMs (Gu & Dao, 2023; Beck et al., 2024). However, we question its e...

2021
[67]

attention

Then we conclude the proof by induction. A remark of Lemma D.1. Lemma D.1 provides an alternative perspective on how SSMs compute the outputs. The predicted value for the t-th token is obtained via decoding a weighted aggregation over representations of all past tokens. The en...

2024
[68]

We consider ϵ >0 small enough, thus, it is sufficient to consider the scenario when |ω| > Amax ≜ maxn∈[N ] |An,n|

Then we can upper bound the magnitude of Z(ω) as: |Z(ω)| = NX n=1 cnbn iω − An,n ≤ NX n=1 cnbn iω − An,n ≤ NX n=1 |cnbn| ||ω| − |An,n|| . We consider ϵ >0 small enough, thus, it is sufficient to consider the scenario when |ω| > Amax ≜ maxn∈[N ] |An,n|. In this regime, we have:...

2023
[69]

Furthermore, let q = 1 − p

Suppose z1 = pm + (1 − p)M for some p ∈ [0, 1]. Furthermore, let q = 1 − p. Now we consider the following dynamics with s0 = 0: st = αtst−1 + ∆tzt, t ∈ [T ], (14) Next, we can show the result below (proved later): Lemma D.6. We have the following inequality for Eq. 14: (1 − At...

2025
[70]

less than

Now we can apply the above result to all memory channels. Assigning each channel with mn = mint∈[T ] bt(xt)n, Mn = maxt∈[T ] bt(xt)n and M ′ n, m′ n accordingly, where n ∈ [N ]. We can yield: max t,s∈[T ] ∥ht − hs∥∞ ≤ max n∈[N ] (M ′ n − m′ n) ≤ max n∈[N ] 1 − AT −1 min (Mn − ...

2024
[71]

Needle in a Haystack

E.3 CIFAR-10 I MAGE CLASSIFICATION Here we present experiment details in Sec. 3.3, where we conduct image classification on the CIFAR-10 dataset to study locality bias in SSMs. Specifically,32 × 32 RGB images in the dataset are 3https://github.com/gkamradt/LLMTest_NeedleInAHay...

2025
[72]

In SSMs, the class token must be positioned last to aggregate features from the entire sequence

places the class token at the first position of the input sequence, this design is incompatible with SSMs, which rely on causal sequence modeling. In SSMs, the class token must be positioned last to aggregate features from the entire sequence. We position the class token as th...

2025
[73]

In addition, our image classification setup differs from Tay et al

or placing the class token in the middle of the sequence (Zhu et al., 2024), work more robustly in general but do not fit the needs for our arguments on locality. In addition, our image classification setup differs from Tay et al. (2020), where an 8-bit pixel intensity lookup ...

2020
[74]

The models and training pipelines are built on Arora et al

Each model is trained for 100 epochs. The models and training pipelines are built on Arora et al. (2023). No perturbations are imposed on the input sequences in the training stage. Adversarial Attack. To introduce perturbations to test data for adversarial attack, we first def...

2023
[75]

For each curve in Fig

with Mamba and report the validation perplexity. For each curve in Fig. 4, we fix the depth of the model and vary the number of parameters from 140M to 550M by adjusting the hidden dimension accordingly. The depth is chosen from {16, 24, 32, 48, 64, 72}. All other model config...

2025
[76]

The evaluation set is created by holding out a subset of 10M tokens from the training data

as our training corpus for its high data quality. The evaluation set is created by holding out a subset of 10M tokens from the training data. We follow the training recipe provided in Appendix E.2.1 of Gu & Dao (2023), which roughly follows the Chinchilla scaling laws (Hoffman...

2023
[77]

We test two block sizes {2048, 8192}

to allocate the number of tokens for each model size. We test two block sizes {2048, 8192}. During training, we fix the number of tokens in one training batch (# sequences × sequence length) as 0.5M. The number of training steps is computed by the total number of tokens divide...

2023
[78]

0 A −1000 # , At ≈

and consistent with Gu & Dao (2023). E.5 SSM P OLARIZATION AND ASSOCIATIVE RECALL EXPERIMENTS In this section, we provide implementation details for our proposed polarization technique. Also, we introduce experiment details for the associative recall tasks we used to validate ...

2023
[79]

We consider 1-polarization mitigates locality most significantly, while deepening architecture only relieves recency mildly but deteriorates over-smoothing

We note the extent of locality and over-smoothing for each configuration. We consider 1-polarization mitigates locality most significantly, while deepening architecture only relieves recency mildly but deteriorates over-smoothing. 0-polarization alleviates over-smoothening and...

2023
[1997]

Training compute-optimal large language models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556,

arXiv
[1998]

Datacomp-lm: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794,

arXiv
[2014]

On the properties of neural machine translation: Encoder-decoder approaches

Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259,

arXiv
[2016]

Mamba: Linear-time sequence modeling with selective state spaces

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv
[2018]

What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298,

Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298,

arXiv
[2019]

Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060,

arXiv
[2020]

Zoology: Measuring and improving recall in efficient language models

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04927,

arXiv
[2021]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

2010 arXiv
[2022]

A mathematical perspec- tive on transformers

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspec- tive on transformers. arXiv preprint arXiv:2312.10794,

arXiv
[2023]

xlstm: Extended long short-term memory

Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517,

arXiv
[2024]

On the bottleneck of graph neural networks and its practical implications

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05205,

2006 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

The hidden attention of mamba models

Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590,

arXiv

[2] [5]

Decimamba: Exploring the length extrapolation potential of mamba

Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba. arXiv preprint arXiv:2406.14528,

arXiv

[3] [6]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

1901

[4] [7]

Learning phrase representations using rnn encoder-decoder for statistical machine translation

Kyunghyun Cho. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078,

arXiv

[5] [9]

Rethinking attention with performers

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794,

arXiv 2009

[6] [10]

Transformer-xl: Attentive language models beyond a fixed-length context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860,

arXiv 1901

[7] [12]

Griffin: Mix- ing gated linear recurrences with local attention for efficient language models

Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mix- ing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427,

arXiv

[8] [13]

Bert: Pre-training of deep bidirectional transformers for language understanding

11 Published as a conference paper at ICLR 2025 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL,

work page 2025