Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Eigenvalue spectra of attention and SSM dynamics show consistent signatures of memory retention and selective forgetting that align with task requirements.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:26 UTC pith:D2NNJNUN

load-bearing objection Extensive empirical spectral atlas for sequence models, but the central eigenvalue-memory claim rests on an unproven LTI-to-LPV transfer. the 3 major comments →

arxiv 2510.09379 v2 pith:D2NNJNUN submitted 2025-10-10 cs.LG cs.AIcs.SYeess.SY

Eigenvalues as a Metric for Memory Dynamics in Sequence Models

Rahel Rickenbach , Jelena Trisovic , Alexandre Didier , Jerome Sieber , Melanie N. Zeilinger This is my paper
classification cs.LG cs.AIcs.SYeess.SY
keywords modelsmemorymetricattentiondynamicssequencessmsanalysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Sequence models like transformers and state space models (SSMs) process information over time, but comparing how they remember is hard because their internals look different. This paper uses a recent "dynamical systems framework" that turns masked attention into a linear time-varying system, just like an SSM. Then it computes the eigenvalues of the state-transition matrix for several models (S4, LRU, Mamba-2, linear attention, softmax attention, norm attention) on tasks like image classification, text sentiment, list operations, associative recall, and language modeling.

The central observation is that eigenvalues near 1 appear when a task requires long-term memory, and eigenvalues near 0 appear when a task requires selectively forgetting old information. For example, on Long ListOps, where every token matters, even attention models avoid near-zero eigenvalues; on MQAR, where only specific key-value pairs matter, softmax attention and Mamba-2 show strong peaks near 1 (retention) and near 0 (gating). They also modify architectures: adding gating shifts spectra away from zero, adding convolution shifts spectra toward zero, and changing normalization in norm attention changes the memory-selectivity trade-off.

The authors conclude that eigenvalue analysis could be a principled metric for interpreting and possibly improving sequence models. However, the interpretation leans heavily on linear time-invariant system intuition applied to time-varying systems, and the attention models are trained with less effort than the SSMs, so the performance gaps might not be purely about eigenvalues.

Core claim

In Section 4.2, the authors state: "Eigenvalues capture essential aspects of stability, memory, and long-range dependency modeling, not only for LTI SSMs, but also LPV models and attention mechanisms. In other words, downstream task requirements are reflected in spectral signatures of eigenvalues." This is the load-bearing assertion: specific eigenvalue distributions (near one for memory retention, near zero for selective forgetting) correspond to task demands and explain model performance differences.

Load-bearing premise

The paper assumes that the memory semantics of eigenvalues from linear time-invariant (LTI) systems—eigenvalues near 1 mean slow decay/long memory, near 0 mean fast forgetting—carry over to linear parameter-varying (LPV) systems with input-dependent transition matrices Λ_i. This is used throughout to interpret spectra of attention and Mamba-2. However, for time-varying systems, instantaneous eigenvalues do not generally determine stability or memory decay; Lyapunov exponents or joint spectral radius do. The paper does not justify this transfer, and its comment that eigenvalues >1 "could potentially be problematic" (Section 4.2) is not rigorous. If this assumption fails, the entire interpretation collapses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes using eigenvalue spectra of state-transition matrices in a recently proposed dynamical-systems framework as a common metric for memory dynamics across SSMs and attention mechanisms. It recasts softmax, linear, and norm attention, as well as Mamba-2, as LPV systems (Eq. 1, Eq. 3), computes eigenvalues, and reports batch-averaged histograms on LRA, MQAR, and WikiText. The empirical study finds that tasks requiring long memory correlate with eigenvalue mass near one, tasks requiring selectivity correlate with mass near zero, and architectural modifications (gating, convolution, normalization changes, pseudo-LTI Mamba-2) shift spectra together with performance. The paper concludes that eigenvalues capture essential aspects of stability, memory, and long-range dependency modeling for both SSMs and attention, and that spectral signatures can guide architectural design.

Significance. If valid, the proposed spectral metric would provide a common quantitative language for comparing attention and SSM memory behavior, complementing memory-function analyses. The empirical corpus is extensive: six architectures, five tasks, multiple layers, heads, and seeds, with code released and standard benchmarks used. The paper also demonstrates a concrete use case by showing that architectural changes (gating, convolution, normalization, pseudo-LTI Mamba-2) produce interpretable spectral shifts. However, the central interpretive step—transferring LTI eigenvalue semantics to input-dependent LPV systems—is not established. The strength of the empirical observations therefore depends on an unvalidated assumption, and the paper's broad central claim in Section 4.2 is stronger than what the evidence currently supports.

major comments (3)
  1. [Section 3.2, Eq. (1), Eq. (3)] The memory interpretation 'eigenvalues near zero induce rapid forgetting, near one enable long-term memory' is stated for h_i = Λ_i h_{i-1} + B_i u_i, but Λ_i is input-dependent throughout (Eq. 1, Eq. 3). This statement is standard for LTI systems only; for LPV systems, the influence of input u_j on output y_i is governed by the ordered product C_i Λ_i ... Λ_{j+1} B_j, not by the instantaneous eigenvalues of Λ_i. Since all Λ_i considered here are scalar multiples of the identity, the relevant quantity is the cumulative product of the scalar eigenvalue sequence. A marginal, batch-averaged histogram of instantaneous eigenvalues discards the order in which eigenvalues occur, so systems with identical histograms can have very different memory behavior (e.g., alternating 0.1 and 10 yields a per-pair product near 1). The central claim in Section 4.2 and the gating/selectivity interpretations i
  2. [Section 4.2, eigenvalues > 1] The paper states that eigenvalues larger than one 'could potentially be problematic' because they cause unstable dynamics of LPV systems. For time-varying systems, this is not justified: a single instantaneous eigenvalue magnitude above one does not imply instability, and eigenvalues inside the unit circle do not guarantee bounded responses. The correct descriptors are Lyapunov exponents, joint spectral radius, or products of transition matrices. This issue affects the repeated interpretation of attention's high eigenvalues as 'unstable' or 'detrimental.' Please either provide a rigorous bound relating the histogram statistics to worst-case growth of the state, or explicitly downgrade such statements to heuristic observations.
  3. [Figure 1, Section 4.2] The empirical support for the central claim is correlational. The metric is a batch-averaged histogram of instantaneous eigenvalues, and the observed spectral differences between S4, Mamba-2, and attention models coincide with many other architectural differences (e.g., convolution, gating, parameterization). No experiment isolates the eigenvalue spectrum as the causal driver while holding other components fixed. A constructive remedy would be to include a controlled intervention—for example, spectral regularization on attention eigenvalues, or a direct comparison of the histogram-based predictions with actual convolution-kernel memory measures. Without such a test, the claim that 'downstream task requirements are reflected in spectral signatures' remains a post-hoc interpretation rather than a validated metric.
minor comments (5)
  1. [Section 4.2] The hedging in 'could potentially be problematic' and 'might further contribute' conflicts with the stronger claim made earlier in the same section. Please align the language with the level of support actually provided.
  2. [Figures 2 and 4] The labels 'with conv' and 'with gating' do not specify kernel sizes, gating projection dimensions, or insertion points in the main text; Appendix B provides some details but the main text should be self-contained enough to interpret the figures.
  3. [Figures 1 and 6] Notation is inconsistent: 'Sm Att' and 'SM Att' are both used for softmax attention. Please unify.
  4. [Section 5.5] The 'pseudo-LTI' Mamba-2 variant is described only in terms of changing the discretization sampling interval. Clarify exactly how the input-dependence is removed from Λ and how the input-dependent behavior is absorbed into B, since this is central to interpreting Figure 4.
  5. [Appendix C] The appendix is comprehensive but some panels are hard to read because six models share a single axis. Consider separating or enlarging the figures, or using facet labels instead of color alone.

Circularity Check

0 steps flagged

No significant circularity: the empirical eigenvalue–performance study is externally benchmarked, and the LTI-to-LPV interpretation is an unvalidated assumption rather than a circular reduction.

full rationale

The paper's central claim is an empirical correlation between eigenvalue distributions and task performance across several externally benchmarked tasks. The eigenvalue metric for attention is inherited from the authors' prior DSF work, but that prior work supplies a stated mathematical lemma with code, and the current paper performs controlled architectural experiments (gating, convolution, normalization, and a pseudo-LTI variant) whose performance is measured independently of the eigenvalue computation. No fitted parameter is later relabeled as a prediction, and no uniqueness theorem is invoked to force the choice of metric. Although for attention the DSF defines Λ_i through normalization ratios (Eq. 3/5a), the paper's conclusion is not equivalent to that definition: the computed spectra come from trained models, the normalization ratio alone does not determine the full attention memory kernel, and the task-alignment is an observed, falsifiable pattern rather than a tautology. The main weakness is the unproven transfer of instantaneous-eigenvalue memory semantics from LTI to LPV systems; that is a validity or correctness risk, not a circular derivation of the paper's results from its own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claim rests on transferring LTI eigenvalue-memory intuition to LPV systems without formal justification, and on the DSF representation from the authors' own prior work. No new model entities are introduced; the 'metric' is an analysis framework, not a physical or algorithmic invention.

free parameters (2)
  • Eigenvalue bin edges = 0-0.1, 0.1-0.5, 0.5-0.9, 0.9-1, 1-10, 10-100, >100
    The discretization of eigenvalue magnitudes into bins is chosen by hand and affects all reported distributions. Different binning could alter the perceived concentration near one vs zero, though the qualitative findings are likely robust.
  • Sampling interval for Mamba-2 pseudo-LTI = fixed (not input-dependent)
    In Section 5.5, the discretization step is modified to use a fixed sampling interval. This is an ad hoc design choice to make Mamba-2 LTI-like; it is not fitted to data but is a hand-chosen modification that influences the eigenvalue spectra and performance.
axioms (5)
  • standard math Eigenvalues near 1 imply long memory, near 0 imply rapid forgetting (LTI eigenvalue-memory semantics).
    Stated in Section 3.2 as a known result from linear system theory (Kailath, 1980).
  • ad hoc to paper The same eigenvalue-memory semantics apply to LPV systems with time-varying transition matrices.
    The paper interprets eigenvalues of input-dependent Λ_i using LTI intuition without proof. For LPV systems, instantaneous eigenvalues are not generally predictive of stability or memory length; this is the central unproven assumption.
  • domain assumption The DSF representation of masked attention is faithful, so eigenvalues of Λ_i in Eq. (3) are the correct spectral characterization.
    The paper relies on the DSF from Sieber et al. (2024) to obtain the transition dynamics. This is a modeling assumption from prior work, not independently verified here.
  • ad hoc to paper Batch-averaged eigenvalue distributions over test data adequately represent input-dependent dynamics.
    For attention and Mamba-2, eigenvalues vary per input; the paper averages over the test batch (Section 4.2). The resulting distribution may hide important time-varying behavior.
  • ad hoc to paper Eigenvalues with magnitude >1 indicate potential instability in LPV systems.
    In Section 4.2, the paper says eigenvalues >1 'could potentially be problematic' because of unstable dynamics, but this is not true in general for time-varying systems and is not derived.

pith-pipeline@v1.3.0-alltime-deepseek · 19685 in / 7670 out tokens · 77766 ms · 2026-08-04T10:26:17.237470+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Eigenvalues as a Metric for Memory Dynamics in Sequence Models." pith.science (2026). https://pith.science/paper/D2NNJNUN

@misc{pith2026251009379,
  author       = {Pith},
  title        = {Pith review of: Eigenvalues as a Metric for Memory Dynamics in Sequence Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2NNJNUN}},
  note         = {Machine review of arXiv:2510.09379}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While softmax attention drives state-of-the-art performance in sequence modeling, its quadratic complexity motivates linear alternatives such as state space models (SSMs). Structural differences between the two model classes, however, hinder direct comparisons of their memory dynamics, creating the need for a common metric to analyze, interpret, and improve their information processing capabilities. Inspired by recent advances in SSM performance driven by eigenvalue-guided insights, we leverage the dynamical systems framework to bring attention models into a unified analytical framework with SSMs. This allows us to perform a structured analysis, which investigates the applicability of an eigenvalue-spectrum memory dynamics metric to attention models. To this end, we first conduct an extensive empirical study across diverse attention-based models and SSMs on a range of benchmarks. We show that, for both model classes, eigenvalues influence key aspects of memory and long-range dependency modeling, revealing spectral signatures that align with task requirements. Building on these findings, we show how spectral signatures can motivate architectural modifications, how they can be guided through the training process, and how they can provide information about feature importance. The results thereby enable and emphasize the role of eigenvalue analysis as a principled metric for interpreting, explaining, and ultimately improving the capabilities of sequence models.

Figures

Figures reproduced from arXiv: 2510.09379 by Alexandre Didier, Jelena Trisovic, Jerome Sieber, Melanie N. Zeilinger, Rahel Rickenbach.

Figure 1
Figure 1. Figure 1: Eigenvalue distributions for one head, across models, selected layers, and tasks. Bars show the percentage of eigenvalues within discretized ranges (chosen to emphasize eigenvalues near zero and near one). Light and dark bars indicate the distribution at initialization and after training, respectively. Error bars denote standard deviation across input sequences. Model performance, measured as perplexity fo… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of the effects of gating and convolution on the eigenvalue spectra for one head, across selected tasks, models, and layers. Complete plots for all layers and tasks are provided in Appendix C. details for each modification and the conclusions we draw after investigating its eigenvalue spectra, which can be summarized in the following: Architectural changes are reflected not only in model performa… view at source ↗
Figure 3
Figure 3. Figure 3: Eigenvalue distribution comparison for single-layer Mamba-2 and softmax attention models with and without convolution on MQAR. Results for one out of four heads are shown. Given the observation suggesting that a layer-dependent task division for softmax attention is present in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Eigenvalue distributions on CIFAR-10 and ListOps for one of the heads for: (left, with white back￾ground) norm attention with convolution and different normalization functions; (right, with grey background) Mamba-2 as LTI. Complete plots for all layers and tasks are provided in Appendix C. In the norm attention model, we systematically replace the normalization function to test its influ￾ence on the distri… view at source ↗
Figure 6
Figure 6. Figure 6: contains all missing layers from Figure 1; [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Plot legend of all subsequent figures. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Eigenvalue distributions for one head, across all models, layers, and tasks (excluding MQAR). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Eigenvalue distributions across remaining heads and all layers for CIFAR-10. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Eigenvalue distributions across remaining heads and all layers for ListOps. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Eigenvalue distributions across remaining heads and all layers for IMDb. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Eigenvalue distributions across 4 additional heads and all layers for WikiText. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Eigenvalue distributions across remaining heads and all layers for WikiText. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Eigenvalue distributions across models, layers, and two out of three additional random seeds for CIFAR-10. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Eigenvalue distributions across models, layers, and one remaining additional random seed for CIFAR-10. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Eigenvalue distributions across models, layers, and two out of three additional random seeds for ListOps. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Eigenvalue distributions across models, layers, and one remaining additional random seed for ListOps. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Eigenvalue distributions across models, layers, and three additional random seeds for IMDb. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Eigenvalue distributions across models, layers, and three additional random seeds for MQAR. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Eigenvalue distributions across models, layers, and two out of three additional random seeds for WikiText. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Eigenvalue distributions across models, layers, and one remaining additional random seed for WikiText. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Comparison of the effects of gating and convolution on the eigenvalue spectra for one head, across CIFAR-10, IMDb, and ListOps, the three investigated attention models, and all layers. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Comparison of the effects of gating and convolution on the eigenvalue spectra for one head, Wiki￾Text, the three investigated attention models, and all layers [PITH_FULL_IMAGE:figures/full_fig_p033_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: The absolute change in performance after each architectural change across models and tasks. For gating and convolution, the difference is computed in comparison to the nominal models shown in [PITH_FULL_IMAGE:figures/full_fig_p033_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Eigenvalue distributions for all tasks and one of the heads for norm attention with convolution, using different normalization functions. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Eigenvalue spectra of Mamba-2 Pseudo LTI, one head, across all layers and LRA models. Standard binning is presented in the top row of plots, followed by a more fine-grained visualization around one. The corresponding legends are provided below the respective plots. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_24.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 2 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Zoology: Measuring and Improving Recall in Efficient Language Models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and Improving Recall in Efficient Language Models . arXiv:2312.04927, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Self-attention networks localize when qk-eigenspectrum concentrates

    Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention networks localize when qk-eigenspectrum concentrates. In Proceedings of the 41st International Conference on Machine Learning, pp.\ 2903--2922, 2024

    2024

  4. [4]

    xLSTM: Extended Long Short-Term Memory

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended Long Short-Term Memory . arXiv preprint arXiv:2405.04517, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Eigen analysis of self-attention and its reconstruction from partial computation

    Srinadh Bhojanapalli, Ayan Chakrabarti, Himanshu Jain, Sanjiv Kumar, Michal Lukasik, and Andreas Veit. Eigen analysis of self-attention and its reconstruction from partial computation. arXiv preprint arXiv:2106.08823, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    1901

  8. [8]

    Transformers are SSMs: Generalized Models and Efficient Algorithms with Structured State Space Duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms with Structured State Space Duality . In ICML 2024 , 2024

    2024

  9. [9]

    Fu, Tri Dao, Khaled K

    Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards Language Modeling with State Space Models , 2023. URL https://arxiv.org/abs/2212.14052

    Pith/arXiv arXiv 2023

  10. [10]

    Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rnns through negative eigenvalues, 2025. URL https://arxiv.org/abs/2411.12537

    Pith/arXiv arXiv 2025

  11. [11]

    HiPPO: Recurrent Memory with Optimal Polynomial Projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher R\' e . HiPPO: Recurrent Memory with Optimal Polynomial Projections . In Advances in Neural Information Processing Systems, volume 33, pp.\ 1474--1487. Curran Associates, Inc., 2020

    2020

  12. [12]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Efficiently Modeling Long Sequences with Structured State Spaces

    Albert Gu, Karan Goel, and Christopher R\'e. Efficiently Modeling Long Sequences with Structured State Spaces . In The International Conference on Learning Representations ( ICLR ) , 2022

    2022

  14. [14]

    Eigenvalue normalized recurrent neural networks for short term memory, 2019

    Kyle Helfrich and Qiang Ye. Eigenvalue normalized recurrent neural networks for short term memory, 2019. URL https://arxiv.org/abs/1911.07964

    Pith/arXiv arXiv 2019

  15. [15]

    Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks

    Cecilia Jarne. Different eigenvalue distributions encode the same temporal tasks in recurrent neural networks. Cognitive Neurodynamics, 17 0 (1): 0 257–275, April 2022. ISSN 1871-4099. doi:10.1007/s11571-022-09802-5. URL http://dx.doi.org/10.1007/s11571-022-09802-5

    work page doi:10.1007/s11571-022-09802-5 2022

  16. [16]

    Linear systems, volume 156

    Thomas Kailath. Linear systems, volume 156. Prentice-Hall Englewood Cliffs, NJ, 1980

    1980

  17. [17]

    Transformers are RNN s: fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are RNN s: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  18. [18]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization . In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  19. [19]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  20. [20]

    When recalling in-context, transformers are not ssms

    Destiny Okpekpe and Antonio Orvieto. When recalling in-context, transformers are not ssms. arXiv preprint arXiv:2508.19029, 2025

    arXiv 2025

  21. [21]

    Signals & systems

    Alan V Oppenheim, Alan S Willsky, and Syed Hamid Nawab. Signals & systems. Pearson Educaci \'o n, 1997

    1997

  22. [22]

    Resurrecting recurrent neural networks for long sequences

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp.\ 26670--26698. PMLR, 2023 a

    2023

  23. [23]

    Resurrecting Recurrent Neural Networks for Long Sequences

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting Recurrent Neural Networks for Long Sequences . In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp.\ 26670--26698. PMLR, 23--29 Jul 2023 b

    2023

  24. [24]

    Colin Raffel and Daniel P. W. Ellis. Feed-forward networks with attention can solve some long-term memory problems, 2016. URL https://arxiv.org/abs/1512.08756

    Pith/arXiv arXiv 2016

  25. [25]

    One-layer transformers fail to solve the induction heads task

    Clayton Sanford, Daniel Hsu, and Matus Telgarsky. One-layer transformers fail to solve the induction heads task. arXiv preprint arXiv:2408.14332, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and J \"u rgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pp.\ 9355--9366. PMLR, 2021

    2021

  27. [27]

    Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks

    Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie Zeilinger, and Antonio Orvieto. Understanding the differences in foundation models: Attention, state space models, and recurrent neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=iF7MnXnxRw

    2024

  28. [28]

    Smith, Andrew Warrington, and Scott Linderman

    Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified State Space Layers for Sequence Modeling . In The Eleventh International Conference on Learning Representations, 2023

    2023

  29. [29]

    Yuanhang Su and C.-C. Jay Kuo. On extended long short-term memory and dependent bidirectional recurrent neural network. Neurocomputing, 356: 0 151–161, September 2019. ISSN 0925-2312. doi:10.1016/j.neucom.2019.04.044. URL http://dx.doi.org/10.1016/j.neucom.2019.04.044

    work page doi:10.1016/j.neucom.2019.04.044 2019

  30. [30]

    Augmenting self-attention with persistent memory, 2019

    Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Augmenting self-attention with persistent memory, 2019. URL https://arxiv.org/abs/1907.01470

    Pith/arXiv arXiv 2019

  31. [31]

    Long Range Arena : A Benchmark for Efficient Transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena : A Benchmark for Efficient Transformers . In International Conference on Learning Representations ( ICLR ) , 2021

    2021

  32. [32]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All you Need . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    2017

  33. [33]

    Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization, 2024

    Shida Wang and Qianxiao Li. Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization, 2024. URL https://arxiv.org/abs/2311.14495

    Pith/arXiv arXiv 2024

  34. [34]

    State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory

    Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. Advances in Neural Information Processing Systems, 36: 0 74021--74038, 2023

    2023

  35. [35]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  37. [37]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  38. [38]

    ks V s Xi o/ θ e ]` moÁ, \1= 7vd ; ;7 z6'e ߷DvB |_ E?/ u z + 3R G Y_>2(/ B.6ɶe,Mm `q c? gĊ=t ?mZ ds@K B5޾

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2000