Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Uncovering the Spectral Bias in Diagonal State Space Models

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diagonal state-space models initialized directly in the discrete Fourier domain achieve uniform, alias-free spectral coverage independent of the discretization step, enabling from-scratch training on PathX-256.

desk verdict A genuinely useful discrete-domain pole initialization for diagonal SSMs, with a clean motivation and a strong PathX-256 result, but the universal-approximator claim and one proof step need fixing before the framing is trustworthy. read the letter →

arxiv 2508.20441 v1 pith:645EBDXS submitted 2025-08-28 cs.LG cs.AI

Ruben Solozabal , Velibor Bojkovic , Hilal AlQuabeh , Kentaro Inui , Martin Takáč This is my paper · ORCID
classification cs.LGcs.AI
keywords statespacemodelsspectralbiasinitializationdiscreteFouriertransformLongRangeArenasequencemodelingPathX-256
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

State space models (SSMs) are sequence models that represent long-range dependencies through a convolution kernel parameterized by a linear dynamical system, and diagonal SSMs restrict the state matrix to complex diagonal form. This paper tries to establish that the initialization of that diagonal matrix—not the architecture itself—is what biases these models toward local, short-range solutions, and that the bias can be removed by placing poles directly in the discrete domain. It shows that continuous-time initializations couple decay rate and oscillation frequency through the discretization step $\Delta$, so the frequency coverage depends on choosing $\Delta$ correctly, while the proposed S4D-DFouT initialization $\lambda_n = \exp(-\xi n/2 + i 2\pi n/N)$ covers the spectrum uniformly and alias-free regardless of $\Delta$. If the claim is right, diagonal SSMs no longer need task-specific timescale tuning, and the paper reports state-of-the-art Long Range Arena results including the first from-scratch training on PathX-256 at 87.89% accuracy.

What carries the argument

The central object is S4D-DFouT, an initialization for diagonal state-space models that fixes the discrete-time eigenvalues directly as $\lambda_n = \exp(-\xi n/2 + i 2\pi n/N)$ for $n = 0,\dots,N-1$, with $\xi$ a learnable damping factor. A layer of $H$ parallel SSMs is synchronized by assigning each machine a phase offset $\phi_h = 2\pi(h-1)/(NH)$, so the union of poles forms a uniform grid of $NH$ frequencies on $[0,2\pi)$. That grid is what does the work: it decouples decay from frequency selection, removes the discretization step $\Delta$ from the spectral-coverage question, and eliminates aliasing as long as every mode stays within its Nyquist band. In the $\xi = 0$ limit the state transition is unitary and the model reduces to a DFT basis, capable of representing any circular convolution kernel of length $N$; a half-plane variant for real inputs keeps only $N/2+1$ poles, halving the state dimension.

What would settle it

Train S4D-DFouT on a sequence-classification task constructed so the class label depends on the amplitude or phase of a single sinusoid whose frequency is deliberately off the N-point uniform grid (e.g., a repeating period of N+1). If the model still solves it, the universal-coverage claim would outrun the paper's circular-convolution argument; if it fails, the fixed-grid assumption is the load-bearing limitation.

Watch

Extended reading notes

Core claim

The central discovery is a diagnosis and a cure. In a diagonal SSM, each basis kernel is a damped complex exponential whose discrete frequency is $\Omega = \Delta\omega$; discretizing a continuous pole $\lambda = -\alpha + i\omega$ maps it to $e^{-\alpha\Delta + i\Delta\omega}$, so $\Delta$ shifts both decay and resonance. S4D-Lin and S4D-Inv initializations therefore cover different spectral regions depending on $\Delta$, and when $\Delta$ misses the task's intrinsic timescale, the learned kernels collapse to a local band—in sCIFAR the effective kernel has peaks only at row-stride offsets and almost no energy beyond. S4D-DFouT instead initializes $\lambda_n = \exp(-\xi n/2 + i 2\pi n/N)$ in the discrete domain, putting all poles on a uniform grid around the unit circle with a shared learnable damping; this gives complete, non-redundant coverage independent of $\Delta$ and, in the $\xi = 0$ limit, reduces the state transition to the DFT. On Long Range Arena the paper reports 87.89% accuracy on PathX-256 from scratch, the first such result without self-pretraining.

Load-bearing premise

The central premise is that the repeating pattern a task depends on has one of the evenly spaced periods the initialization provides; if the key information repeats at any other rate, the fixed grid supplies no mode to detect it.

Editorial extensions

If this is right

  • S4D-DFouT makes the initialization's frequency coverage independent of the discretization step $\Delta$, so the same hyperparameter settings transfer across LRA tasks without task-specific $\Delta$ tuning.
  • Fourier-initialized SSMs can match or exceed HiPPO-style diagonal baselines when their spectral support is uniform, overturning the earlier view that Fourier initialization underperforms on hard LRA tasks.
  • Far from requiring long-range receptive fields, LRA successes can be driven by local structure captured by a few spectral modes; in sCIFAR the learned kernels are essentially determined by the first 32 coefficients, the row stride.
  • The first from-scratch run on PathX-256 (87.89% accuracy) shows diagonal SSMs can scale to 65,536-token sequences without self-pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the uniform-grid argument suggests an adaptive initialization that places poles at the top-$K$ dominant frequencies estimated from the training data would need fewer modes than a full uniform grid for the same accuracy on local-structure tasks.
  • Editorial inference: because the universal-approximation claim in the paper holds for $N$-periodic circular kernels rather than arbitrary length-$L$ kernels, a direct comparison on a synthetic task with an off-grid informative frequency would delimit when the initialization helps versus when it is neutral.
  • Editorial inference: the observed inactivity of most SSM modes under the baseline suggests that the same $H_\infty$-norm analysis applied to S4D-DFouT after training could yield a pruning criterion, shrinking models with little accuracy loss.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper analyzes diagonal state-space model (SSM) initializations from a frequency-domain perspective. It argues that existing continuous-domain initializations such as S4D-Lin and S4D-Inv entangle decay and frequency through the discretization step Δ, producing non-uniform spectral coverage and sensitivity to the chosen Δ. The authors propose S4D-DFouT, which initializes the discrete-time poles directly on a uniform grid λ_n = exp(-ξn/2 + i2πn/N), optionally with layer-wise phase offsets, and claim this yields alias-free uniform spectral coverage independent of Δ. The paper provides a frequency-response analysis, a motivating delay task, and experiments on the Long Range Arena benchmark, including a from-scratch PathX-256 result at 87.89% accuracy, along with ablations and kernel visualizations suggesting that previously initialized SSMs learn local, row-periodic kernels on image tasks.

Significance. If the empirical results are reproducible, the from-scratch PathX-256 result is a noteworthy advance for diagonal SSMs, and the frequency-based design principle is simple and potentially useful. The experimental section is thorough, with per-task hyperparameters, multiple ablations, and an honest report of a failure on psCIFAR. However, the theoretical claims exceed what is proved: the 'universal approximator' statement in Section 4.2 is false as stated, and Proposition 1's proof contains a concrete algebraic error. These issues do not necessarily invalidate the empirical findings, but they must be corrected before the paper can be accepted.

major comments (2)
  1. [Section 4.2, Eq. (10)] The claim in Section 4.2 that S4D-DFouT 'enables the system to act as a universal approximator' is not supported by the preceding derivation. For ξ=0, the modes λ_n = e^{i2πn/N} yield basis kernels K_n[l] that are N-periodic in l, so the span of the N modes is confined to N-periodic sequences; a delay kernel δ_{l,τ} with τ>N is not representable. Moreover, for a sequence length L not a multiple of N, these exponentials are not orthogonal over l=0,...,L−1, so the 'complete, non-redundant basis' statement applies only to the N-point DFT/circular convolution. The layer-wise synchronization in Eq. (11) provides NH distinct frequencies, which may be the real source of dense coverage (for PathX-256, NH=32768 and L/2=32768), but no condition on H, N, L is stated under which the fixed grid covers the relevant task frequencies. The claim should be weakened to a statement about uniform coverage at the DFT grid, with the covering condition made explicit.
  2. [Appendix B, Proposition 1] Appendix B, Proposition 1 contains an algebraic error. For V_{l,n}=e^{i2πnl/τ} with l=0,...,τ, the off-diagonal entries of the Gram matrix are ∑_{l=0}^{τ} e^{i2π(n−m)l/τ} = 1 for n≠m, not 0. Hence V*V ≠ (τ+1)I, the condition number is not one, and the claimed one-step gradient descent convergence with η=1/(τ+1) does not follow. The proof also assumes continuous poles with no real part (w_n=iπn), whereas S4D-Lin is defined in Eq. (9) with poles −1/2+iπn, so the proposition does not accurately describe S4D-Lin under ZOH discretization. The spike-at-τ conclusion may remain true qualitatively, but the theorem and proof must be corrected or the proposition removed.
minor comments (4)
  1. [Section 4.2] There are typographical errors in the S4D-DFouT paragraph: 'propsoe' should be 'propose' and 'absis' should be 'basis'. Also, 'S4D-DfouT' appears in Section 5.5 and Figure 8 with inconsistent capitalization.
  2. [Section 3.1 / Proposition 1] The notation for the kernel index is inconsistent: the proposition uses ℓ while Eq. (5) uses l. Please unify.
  3. [Section 4.2 / Table 5] It is unclear whether the experimental results use the full grid of Eq. (10) or the half-plane variant described in Section 4.2; the hyperparameters in Table 5 do not specify. Please clarify this in the experimental setup.
  4. [Appendix A.1, Eq. (12)] The recurrence in Eq. (12) appears to contain a typo: 'h[l + 1] = Λx[l] + Bx[l]' should likely read 'h[l + 1] = Λh[l] + Bx[l]', and 'y[l] = Cx[l]' should likely be 'y[l] = C^T h[l]'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: S4D-DFouT's uniform spectral placement is a design choice, the LRA numbers are measured outcomes, and no load-bearing self-citation chain appears.

full rationale

The paper's central proposal, S4D-DFouT, places poles directly in the discrete domain as λ_n = exp(−ξn/2 + i2πn/N). The statement that this provides uniform, alias-free spectral coverage is a restatement of the construction rather than a fitted or externally predicted quantity; the paper presents it as the design rationale, not as an empirical prediction derived from fitted constants. The Long Range Arena accuracies in Table 1 are measured benchmark results and are not reconstructed from the initialization formula, and no parameter is fitted to a subset of LRA data and then reported as a prediction for that same data. The paper contains no self-citations; all cited prior work is by other authors, so there is no self-citation load-bearing chain. Two correctness concerns are noted but they are not circularity: the 'universal approximator' sentence in Section 4.2 overstates the DFT basis (for ξ=0 the basis is complete only for N-periodic circular kernels, not arbitrary length-L kernels), and the Gram-matrix identity in Proposition 1's proof in Appendix B is mathematically incorrect, since the off-diagonal sums are 1, not 0. Neither issue makes the derivation equivalent to its own inputs, so the circularity burden remains at zero.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central design rests on standard DSP facts (DFT basis, Nyquist aliasing) plus empirical assumptions about task spectral content. The learnable damping xi and hand-picked per-task ranges are the main free choices; no invented physical entities are introduced.

free parameters (3)
  • xi (learnable damping factor) = learned during training; per-task range (xi_min, xi_max)
    Controls pole magnitude exp(-xi/2) and hence kernel decay. The range is hand-chosen per task in Table 5, e.g., (0.001, 0.1) for most tasks and (0.0001, 0.1) for PathX, which weakens the 'no tuning' claim.
  • State size N and embedding size H = N in 64 to 256, H in 128 to 512 depending on task
    The grid resolution of the frequency coverage is set by N, the number of poles, and the layer synchronization uses H. These are architectural hyperparameters, not fitted constants.
  • Phase offsets phi_h = 2*pi*(h-1)/(N*H) for h = 1..H
    Deterministic offsets that spread the poles of parallel SSMs in a layer. This is a design choice, not fitted to data.
assumptions (5)
  • standard math Zeroth-order hold discretization maps continuous poles lambda to exp(Delta*lambda)
    Used in Eq. (3) for all schemes; standard in the SSM literature.
  • standard math DFT basis completeness: N unit-circle exponentials span the space of N-periodic sequences
    Underlies the claim in Section 4.2 that the model can exactly represent any circular convolutional kernel of length N.
  • domain assumption Taking the real part of the complex kernel preserves the ability to represent real-valued signals
    Used to justify the half-plane variant in Section 4.2, relying on Hermitian symmetry of real signals.
  • domain assumption The local-attention profile learned on sCIFAR transfers to the LRA image tasks
    Used to motivate the Delta prior for S4D-Lin on PathX in Appendix C.3 and the claim that LRA difficulty is overestimated.
  • ad hoc to paper Uniform alias-free spectral coverage at initialization is sufficient for good performance across tasks
    The central design assumption of S4D-DFouT; no formal result connects uniform pole placement to optimization success.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncovering the Spectral Bias in Diagonal State Space Models." pith.science (2026). https://pith.science/paper/645EBDXS

@misc{pith2026250820441,
  author       = {Pith},
  title        = {Pith review of: Uncovering the Spectral Bias in Diagonal State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/645EBDXS}},
  note         = {Machine review of arXiv:2508.20441}
}
read the original abstract

Current methods for initializing state space models (SSMs) parameters mainly rely on the \textit{HiPPO framework}, which is based on an online approximation of orthogonal polynomials. Recently, diagonal alternatives have shown to reach a similar level of performance while being significantly more efficient due to the simplification in the kernel computation. However, the \textit{HiPPO framework} does not explicitly study the role of its diagonal variants. In this paper, we take a further step to investigate the role of diagonal SSM initialization schemes from the frequency perspective. Our work seeks to systematically understand how to parameterize these models and uncover the learning biases inherent in such diagonal state-space models. Based on our observations, we propose a diagonal initialization on the discrete Fourier domain \textit{S4D-DFouT}. The insights in the role of pole placing in the initialization enable us to further scale them and achieve state-of-the-art results on the Long Range Arena benchmark, allowing us to train from scratch on very large datasets as PathX-256.

Figures

Figures reproduced from arXiv: 2508.20441 by the authors.

Figure 1
Figure 1. The kernels learned by S4D on sCIFAR present a —“local attention” pro￾file—. When unrolled, these kernels align with positions corresponding to the vicini￾ties of the pixel being attended. State space models (SSMs) have recently emerged as a principled and scalable means of modeling long se￾quences across diverse domains—including image process￾ing [1, 2, 3], time-series forecasting [4], and natural lan￾guage unders… view at source ↗
Figure 2
Figure 2. Frequency response of a diagonal SSM. Left: An example of pole configuration (i.e. state matrix entries λ1, . . . , λN ) for two discrete SSM systems of order N = 10 exhibiting conjugate symmetry. Following equation (8) we plot the corresponding frequency responses in dependence of the angular frequency of the corresponding discretized systems with ∆ = 0.1 (center and right figures). The system with ℜ(λ) = 0 (center… view at source ↗
Figure 3
Figure 3. Visualization of the poles configuration obtained on S4D initializations for a system of order [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Continuous copying or Delay task. Left: An example input x[l] and corresponding delayed output y[l]. Center: Reconstruction MSE for different initializations as a function of ∆. Right: The S4D-Lin kernel learned at the theoretically optimal ∆ = 0.002 (of similar perfor…
Figure 5
Figure 5. Figure 5: Pixel-level image classification on sCIFAR. Left: Ablation of the accuracy upon a reduced kernel length and number of trainable S4D-Lin layers. Right: Normalized H∞-norm for each individual SSM initialized under S4D￾Lin, sorted by its operating ∆. To quantify the impac…
Figure 6
Figure 6. Figure 6: Ablation experiment in LRA. Accuracy upon different initialization schemes for fixed [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Subplots (a)–(c) show the mean absolute value of the learned discrete kernel [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Ablation experiment in LRA. Accuracy upon different initialization schemes for fixed [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Polar plots of the learned complex poles on [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SHiPPO: Recurrent Memory with Transported Polynomial Projections

    cs.LG 2026-07 conditional novelty 7.0 of 10

    SHiPPO transports HiPPO coefficient memories via right actions into Sylvester dynamics, and diagnostics show this recovers order-sensitive memory changes that high-rank writes cannot.

Reference graph

Works this paper leans on

41 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML. OpenReview.net, 2024

    work page 2024

  2. [2]

    Fouriermamba: Fourier learning integration with state space models for image deraining

    Dong Li, Yidi Liu, Xueyang Fu, Senyan Xu, and Zheng-Jun Zha. Fouriermamba: Fourier learning integration with state space models for image deraining. CoRR, abs/2405.19450, 2024

    arXiv 2024

  3. [3]

    Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré

    Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4ND: modeling images and videos as multidimensional signals with state spaces. In NeurIPS, 2022

    work page 2022

  4. [4]

    Effectively modeling time series with simple discrete state spaces

    Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Ré. Effectively modeling time series with simple discrete state spaces. In ICLR. OpenReview.net, 2023

    work page 2023

  5. [5]

    Fu, Tri Dao, Khaled Kamal Saab, Armin W

    Daniel Y . Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In ICLR. OpenReview.net, 2023

    work page 2023

  6. [6]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752, 2023

    arXiv 2023

  7. [7]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In ICML. OpenReview.net, 2024

    work page 2024

  8. [8]

    Hippo: Recurrent memory with optimal polynomial projections

    Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In NeurIPS, 2020

    work page 2020

Show all 41 references
  1. [9]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In ICLR. OpenReview.net, 2022

    2022

  2. [10]

    How to train your HIPPO: state space models with generalized orthogonal basis projections

    Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your HIPPO: state space models with generalized orthogonal basis projections. In ICLR. OpenReview.net, 2023

    2023

  3. [11]

    Diagonal state spaces are as effective as structured state spaces

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In NeurIPS, 2022

    2022

  4. [12]

    On the parameterization and initialization of diagonal state space models

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In NeurIPS, 2022

    2022

  5. [13]

    Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De

    Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Çaglar Gülçehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 26670–26698. PMLR, 2023

    2023

  6. [14]

    Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. In ICLR. OpenReview.net, 2023

    2023

  7. [15]

    Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy T

    Rom N. Parnichkun, Stefano Massaroli, Alessandro Moro, Jimmy T. H. Smith, Ramin M. Hasani, Mathias Lechner, Qi An, Christopher Ré, Hajime Asama, Stefano Ermon, Taiji Suzuki, Michael Poli, and Atsushi Yamashita. State-free inference of state-space models: The *transfer function...

    2024

  8. [16]

    Layer-adaptive state pruning for deep state space models

    Minseon Gwak, Seongrok Moon, Joohwan Ko, and PooGyeon Park. Layer-adaptive state pruning for deep state space models. In NeurIPS, 2024

    2024

  9. [17]

    Long range arena : A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In ICLR. OpenReview.net, 2021

    2021

  10. [18]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  11. [19]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML (3), volume 28 of JMLR Workshop and Conference Proceedings, pages 1310–1318. JMLR.org, 2013

    2013

  12. [20]

    Mikhaeil, Zahra Monfared, and Daniel Durstewitz

    Jonas M. Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. In NeurIPS, 2022

    2022

  13. [21]

    Legendre memory units: Continuous-time representation in recurrent neural networks

    Aaron V oelker, Ivana Kajic, and Chris Eliasmith. Legendre memory units: Continuous-time representation in recurrent neural networks. In NeurIPS, pages 15544–15553, 2019

    2019

  14. [22]

    Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y . Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 28043–2...

    2023

  15. [23]

    Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus

    Ramin M. Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In ICLR. OpenReview.net, 2023

    2023

  16. [24]

    Autocorrelation matters: Understanding the role of initialization schemes for state space models

    Fusheng Liu and Qianxiao Li. Autocorrelation matters: Understanding the role of initialization schemes for state space models. CoRR, abs/2411.19455, 2024

    2024 arXiv

  17. [25]

    Mega: Moving average equipped gated attention

    Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In ICLR. OpenReview.net, 2023

    2023

  18. [26]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017

    2017

  19. [27]

    Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting

    Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, pages 5244–5254, 2019

    2019

  20. [28]

    Never train from scratch: Fair comparison of long-sequence models requires data-driven priors

    Ido Amos, Jonathan Berant, and Ankit Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In ICLR. OpenReview.net, 2024

    2024

  21. [29]

    Speech commands: A dataset for limited-vocabulary speech recognition

    Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. CoRR, abs/1804.03209, 2018

    2018 arXiv

  22. [30]

    Romero, Anna Kuzina, Erik J

    David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomczak, and Mark Hoogen- doorn. Ckconv: Continuous kernel convolution for sequential data. In ICLR. OpenReview.net, 2022

    2022

  23. [31]

    Observer-based asynchronous event-triggered robust h∞, adaptive switching control for nonlinear industrial cyber physical systems under data injection attacks

    Yanfei Qin and Ziwen Sun. Observer-based asynchronous event-triggered robust h∞, adaptive switching control for nonlinear industrial cyber physical systems under data injection attacks. International Journal of Control, Automation and Systems , 21(7):2175–2182, 2023

    2023

  24. [32]

    Chang Wei Tan, Christoph Bergmeir, François Petitjean, and Geoffrey I. Webb. BIDMC blood oxygen saturation dataset (32 seconds window) (version 2). https://doi.org/10.5281/ zenodo.4001464, June 2020. Accessed on YYYY-MM-DD

    2020

  25. [33]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020. 11

    2020

  26. [34]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In NeurIPS, 2020

    2020

  27. [35]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165. PMLR, 2020

    2020

  28. [36]

    Colwell, and Adrian Weller

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ...

    2021

  29. [37]

    For the sake of simplicity, let’s suppose poles of the continuous system have no decaywn =iπn; then, ZOH gives dn = exp(Ω) = exp(wn ∆) = exp(iπn∆)

    Discrete poles via ZOH. For the sake of simplicity, let’s suppose poles of the continuous system have no decaywn =iπn; then, ZOH gives dn = exp(Ω) = exp(wn ∆) = exp(iπn∆)

  30. [38]

    The first harmonic n = 1 has frequency Ω1 = π∆

    Fundamental discrete period. The first harmonic n = 1 has frequency Ω1 = π∆. A full 2π rotation requires Ω1T = 2π =⇒ T = 2/∆. Thus one cycle of n = 1 takes preciselyT = 2/∆ steps

  31. [39]

    SetT =τ, i.e

    MatchingT toτ. SetT =τ, i.e. ∆ = 2/τ. Then for anyn, dτ n = exp inπ ∆τ = exp(i 2πn ) = 1, so all modes align in phase at stepl =τ

  32. [40]

    The SSM kernel is K[l] =ℜ 2 N−1X n=0 Cndl n =ℜ 2 N−1X n=0 dl n

    Kernel evaluation. The SSM kernel is K[l] =ℜ 2 N−1X n=0 Cndl n =ℜ 2 N−1X n=0 dl n . Atl =τ, sincedτ n = 1 for alln, K[τ] = 2ℜ N−1X n=0 1 = 2N, whereas forl̸=τ, thedl n are not all unity and partially cancel, yielding|K[l]|<K [τ]

  33. [41]

    local attention

    Conditioning and GD. Fix an integer delayτ >N . Then for 0≤l≤τ and 0≤n≤N− 1, the Vandermonde matrix Vl,n =dl n = exp i 2πn τ l is exactly a (τ + 1)×N partial Fourier matrix. Concretely, its Gram matrix is V∗V = τX l=0 dl ndlm n,m = τX l=0 ei2π(n−m)l/τ = τ + 1, n =m, 0, n ̸=m ,...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.