Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Input-dependent discretization in Mamba's S6 layer can represent Haar-wavelet projections and exactly solve associative recall tasks, locating the source of Mamba's advantage over S4D in its selective memory clock.

desk verdict Solid constructive results on Mamba's MQAR capacity, but the headline wavelet claim needs the PE asterisk and the S4D comparison is shakier than the paper admits. read the letter →

arxiv 2506.11891 v1 pith:3SEVVEOD submitted 2025-06-13 cs.LG

Ningyuan Huang , Miguel Sarabia , Abhinav Moudgil , Pau Rodriguez , Luca Zappella , Federico Danieli This is my paper
classification cs.LG MSC 68T0742C4041A25
keywords MambaselectivestatespacemodelsinputselectivityHaarwaveletsassociativerecallMQARmemorydecay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain why Mamba's input-dependent state-space layer (S6) is more powerful than its predecessor S4D. It proves that the S6 layer can represent projections onto Haar wavelets when the input carries positional information, which yields exponentially faster approximation of discontinuous, piecewise-constant signals than the Fourier-type bases S4D provides. It also shows that input selectivity lets an S6 layer fight memory decay by effectively freezing its own clock, and that a single Mamba layer without gating can solve the multiple-query associative recall task exactly, with embedding size and state size that match empirical thresholds. Together these results locate the source of Mamba's practical advantage in the input-dependent discretization time step, and in the interaction of convolution, gating, and selectivity within the block.

What carries the argument

The central object is the input-dependent discretization time step $\Delta(x_t)$ inside the S6 recurrence. The paper treats $\Delta(x_t)$ as a freely tunable clock: pushing $\Delta\to\infty$ erases memory (decay factor $e^{-\lambda\Delta}\to 0$), while pushing $\Delta\to 0$ freezes the state ($e^{-\lambda\Delta}\to 1$). This one knob converts the layer's basis functions $e^{-\lambda\int_s^t \Delta(x_r)dr}$ into Heaviside functions, hence Haar wavelets, and simultaneously gives the layer a mechanism for selective memorization and forgetting. For associative recall, the machinery is the matrix-shaped hidden state $h_t\in\mathbb{R}^{d\times N}$ in Mamba's update, with $B_t$ selecting which key-column to write and $C_t$ selecting which column to read.

What would settle it

Inspect a successfully trained S6 layer on the KEEP FIFTH task: the paper's mechanism predicts a sharp step in the effective decay $e^{-\lambda\Delta(x_t)}$ over positions, roughly $\approx 0$ before the target position and $\approx 1$ after it. A model that solves the task without exhibiting this step, or that solves it without any positional information, would contradict the claimed role of input selectivity.

Watch

Extended reading notes

Core claim

The central claim is that the input-dependent discretization step $\Delta(x_t)$ is what gives Mamba its power, and that this power shows up in three concrete places. Theorem 1 states that, given time positional encoding, three S6 basis functions can approximate any Haar wavelet $\psi_{j,k}$ to arbitrary accuracy, because $\Delta(x_t)$ can be driven to $+\infty$ before a chosen time and to $0$ after it, making each basis a Heaviside function whose combination is a wavelet; Corollary 1 then gives $L^2$ error $O(2^{-N/(3m)})$ for piecewise-constant functions with $m$ jumps, versus $O(N^{-1})$ for S4D. Lemma 2 states that an S6 layer can preserve sensitivity to early inputs provided $\lambda\sum_r \Delta(x_r)$ stays bounded, i.e., by making the effective decay $e^{-\lambda\Delta}$ approach $1$; the paper shows trained models do exactly this on the KEEP n-th task. For associative recall, Theorem 2 constructs a gating-free 1-layer Mamba that solves MQAR with embedding size $d=O(\kappa+\log|V|)$ and state size $N=\kappa$ by using the hidden-state matrix columns as key-indexed value slots; Theorem 3 gives the Mamba-2 version with $d=O(\log\kappa+\log|V|)$ and $N=\log\kappa$; Theorem 4 shows even an S4D mixer solves MQAR using gating, with $d=O(\kappa\log|V|)$ and $N=1$. A variant that applies $\Delta$ along the state dimension instead of the embedding dimension solves INDUCTION HEADS with $d=2|V|$ and $N=|V|$.

Load-bearing premise

The wavelet-approximation result assumes the layer is told the absolute time of each token, through positional encoding or through an extra layer that computes time from a constant input; without that time information, a single standard Mamba layer is not shown to have the promised edge over S4D.

Editorial extensions

If this is right

  • On discontinuous or piecewise-constant signals, a single S6 layer with positional information achieves exponentially decaying approximation error $O(2^{-N/(3m)})$, whereas S4D is limited to $O(N^{-1})$.
  • Input selectivity provides a concrete memory mechanism: by keeping $\lambda\sum_r \Delta(x_r)$ bounded, an S6 layer can retain sensitivity to early tokens over arbitrarily long sequences, and trained models learn to set $e^{-\lambda\Delta}\approx 1$ for the tokens to remember and $\approx 0$ otherwise.
  • MQAR can be solved exactly by a 1-layer Mamba without gating using $d=O(\kappa+\log|V|)$ and $N=\kappa$; Mamba-2 achieves $d=O(\log\kappa+\log|V|)$ and $N=\log\kappa$; even an S4D mixer solves it with gating at $d=O(\kappa\log|V|)$ and $N=1$.
  • These model-size bounds are tight in practice: empirical accuracy on MQAR drops below 100% roughly when $d$ or $N$ falls under the theorized values.
  • Moving the input-dependent $\Delta$ from the embedding dimension to the state dimension (the Mamba-$\Delta^\top$ variant) improves performance on INDUCTION HEADS beyond standard Mamba at matched sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the $\Delta$-as-clock picture is right, then the parameterization of $\Delta$ matters more than the choice of state matrix $\Lambda$; a testable prediction is that constraining $\Delta$ to be a monotone function of a scalar feature such as position or saliency should reproduce most of Mamba's benefit on memorization tasks.
  • The paper gives existence constructions, not a proof that gradient descent finds them; probing trained Mamba models for the predicted piecewise structure of $\Delta$ over time on KEEP n-th would separate mechanism from mere expressivity.
  • The wavelet corollary suggests a direct application: initializing $\Delta$ from an explicit wavelet schedule could yield a 'wavelet SSM' with guaranteed approximation of discontinuities, potentially matching Transformers on needle-in-a-haystack tasks without attention.
  • Because the MQAR constructions rely on near-orthogonal embeddings via the Johnson-Lindenstrauss lemma, the logarithmic dimension bounds are likely robust to vocabulary growth but may degrade in adversarial input distributions where key correlations accumulate; the paper itself notes this failure mode for Mamba-2.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes the role of input selectivity in Mamba's S6 layer through three lenses: function approximation, long-range memory, and associative recall. It claims that (i) an S6 layer can represent projections onto Haar wavelets, giving a rate advantage over S4D for discontinuous functions; (ii) S6 can dynamically counteract memory decay; (iii) one-layer Mamba, Mamba-2, and Mamba-with-S4D-mixer solve the MQAR task with explicit model-size bounds; and (iv) a proposed Mamba-Δ⊤ variant solves the INDUCTION HEADS task. The theoretical constructions are complemented by experiments on KEEP n-TH, MQAR, and INDUCTION HEADS.

Significance. The MQAR constructions in Section 5 are explicit and concrete, with size bounds that are plausible and empirically well separated from failure regions; if the dimension-reduction issues noted below are fixed, these are valuable and novel contributions to the mechanistic understanding of SSMs. The sensitivity analysis in Section 4.2 is sound and connects cleanly to the KEEP n-TH experiments. The wavelet result, properly qualified, is an interesting explanation of input selectivity, but as written the abstract and Theorem 1 overstate what is proven for a standard Mamba layer. The paper also deserves credit for including detailed proofs and for validating the tightness of its constructions empirically.

major comments (4)
  1. [Section 4.1, Theorem 1 and Proposition 1] The theorem is proved only for the augmented input x̃_s = concat[x_s; s], and the abstract's unqualified claim that 'the S6 layer can represent projections onto Haar wavelets' is not supported for a standard Mamba layer without positional encoding. Proposition 1 shows only that a Mamba layer fed all-ones can output time t; it does not specify how the time signal and the original signal x are both supplied to the layer that computes the wavelet projection, since in a standard sequential stack the second layer's input is the first layer's output. The abstract and contribution (i) should be revised to state explicitly that the result requires time positional encoding (or a specified parallel/residual mechanism), and the claimed relaxation should be completed or removed.
  2. [Section 4.1, Theorem 1 and Corollary 1] The construction uses three different discretization functions Δ with different thresholds, but in the S6 formalization (14c), Δ_t ∈ R^d is shared by all N state columns for each embedding dimension, so with the stated augmented input x̃_s = concat[x_s;s] (d=2) a single S6 layer cannot implement three distinct Δ channels. The theorem should state the required embedding dimension (d≥3) or explicitly use three layers; otherwise the statement 'the S6 layer can represent' is ambiguous. In addition, Corollary 1's rate should be written as O(2^{-N/(3m)}) and should state that N Mamba basis functions yield N/3 Haar wavelets.
  3. [Section 5.1, Theorem 3 and Remark 7] In the JL-reduced construction, the query output is y_t = v_{j(i)} + Σ_{s: k_{s-1}≠k_i} O(ε) v_s, so cross-talk accumulates over all κ stored pairs. For the theorem to hold on arbitrary inputs, ε must be O(1/κ), which makes the projection dimension p=O(ε^{-2} log d) = O(κ^2 log d) and destroys the claimed O(log κ + log |V|) bound. Remark 7's statement that decreasing ε 'does not impact the logarithmic behavior' is incorrect; the theorem should either be restricted to a stochastic/generic setting or the construction must eliminate cross-key interference.
  4. [Section C.1, Theorem 4 Step 2] The proof asserts ⟨M e_i, 1_p⟩ ≤ ε because the last JL vector is parallel to 1_p, but if M e_{|V|+1} = 1_p/√p (unit norm), then ⟨M e_i, 1_p⟩ = √p ⟨M e_i, M e_{|V|+1}⟩ ≤ ε√p, which is not small. The output layer in (99) consequently needs a class-dependent bias to cancel the β⟨M e_l,1_p⟩ term, or a different treatment of the shift; as written, the d=O(κ log|V|) reduction is not justified.
minor comments (5)
  1. [Section 4.2, Lemma 2, Eq. (13)] The lower bound should evaluate the derivative factor at x_j, not x_s; B_n(x_s) should be B_n(x_j) in the displayed lower bound.
  2. [Corollary 1] The notation O(2^{-N/3m}) is ambiguous; please write O(2^{-N/(3m)}).
  3. [Section 4.2, Lemma 2, Eq. (12)] The limit in Eq. (12) may not exist for arbitrary input sequences; use lim sup or an explicit bounded-partial-sums assumption.
  4. [Figure 2] The caption says 'dashed curves denote our theoretical bounds,' but the displayed plot shows solid black lines; please align the caption with the figure.
  5. [Theorem 3] The state size N=log κ should specify the base of the logarithm, or note that the base is immaterial, to avoid confusion with natural logarithm versus base 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the S6 and MQAR constructions are explicit parameter assignments derived from the Mamba equations; the empirical tests are independent, and the only flagged issue is a non-circular scope gap concerning positional encoding.

full rationale

None of the paper's central claims reduces to its inputs by construction. Theorem 1 is an existence proof: the authors fix λ, B, and choose Δ(t)=SoftPlus(w t − w t̂) with w→∞ so that the basis (8) converges uniformly to a shifted Heaviside; three such bases are combined through (18) to realize a Haar wavelet. No fitted constant or measured quantity enters the construction. The MQAR theorems (Thm. 2-4) are likewise explicit: they specify embeddings, size-2 convolution weights, SSM parameters (e.g., Λ=0, Δ_t=1, B(x)=C(x)=W_k x in Thm. 2) and then apply the standard JL lemma; the resulting d and N bounds are read off the construction, not tuned to the experimental curves. The benchmarks in Fig. 2 and Fig. 3 are independent trained models, so the claimed 'tightness' is an external check rather than a relabeled fit. There are no load-bearing self-citations: foundational references are to Gu & Dao and Dao & Gu (the original Mamba/Mamba-2 papers by other authors), and the expressivity comparisons cite Orvieto et al., Vetterli, and Eckhoff. The one notable weakness is the wavelet theorem's scope: Thm. 1 requires x̃=concat[x_s;s], and the proposed relaxation (Prop. 1) only shows that a layer fed all-ones can output t; it does not show that the layer computing the projection in the standard sequential stack has simultaneous access to both x_s and s. This makes the abstract's unqualified claim ('the S6 layer can represent projections onto Haar wavelets') an overstatement, but it is a missing-support/scope issue, not a circular reduction.

Assumptions & free parameters 5 free parameters · 7 assumptions · 1 invented entities

The central claims rest on the exact Mamba recurrence, the ability to shape Delta through SoftPlus with extreme weights (an asymptotic assumption), and the reduction of constructions to implementable parameter choices. The JL lemma is standard. The S4D comparison imports an unproven 'at best Fourier' claim, and the SiLU-to-ReLU transfer is an asymptotic idealization. These are the main axioms beyond standard math.

free parameters (5)
  • w_Delta, b_Delta in wavelet construction = w_Delta to -inf, b_Delta = -w_Delta t_k
    Chosen to make Delta(s) approximate a Heaviside; hand-set, not learned.
  • lambda and B in wavelet bases = lambda=1, B=2^{j/2}
    Chosen for the 3-basis construction of each Haar wavelet in Thm 1.
  • Convolution parameters c0,c1,b and embedding scales k,v = Thm 2: v=1,k=2,c0=1,c1=2,b=1; JL step: k=10,c0=1,c1=10,b=3; Thm 4 similar
    Chosen by hand to satisfy inequalities (55) or (69) that produce desired convolution behavior.
  • JL epsilon and projection dimension p = p=O(epsilon^-2 log d), epsilon in (0,0.5), chosen small
    Controls near-orthogonality of projected embeddings; affects hidden constants in dimension bounds.
  • w_Delta in Mamba-delta-top = w_Delta to infinity
    Selective erasure of key columns in the state matrix; asymptotic construction.
assumptions (7)
  • domain assumption Mamba S6 discrete recurrence (14f) with ZOH/Euler discretization as in (5)
    The constructions operate on this exact definition of Mamba; the theory depends on it.
  • domain assumption Eigenvalues lambda_n have Re(-lambda_n) <= 0 for stability
    Inherited from SSM design; used to keep the hidden state bounded.
  • domain assumption Delta(x_t) is an arbitrary positive function of x_t via SoftPlus-Linear, so it can be shaped to approximate Heavisides with w_Delta to +/-inf, and with positional encoding can depend on time t
    The crux of Thm 1; requires PE augmentation or an extra all-ones layer (Prop 1).
  • ad hoc to paper S4D can at best project onto Fourier bases and achieves O(N^-1) for discontinuous functions
    Invoked in Cor 1 without a self-contained proof; imported from Orvieto et al. and Eckhoff; the 'at best' phrasing is an assertion.
  • ad hoc to paper SiLU can be approximated by ReLU for large magnitudes, making the convolution constructions (stated with ReLU) transfer to Mamba's SiLU
    Footnote in Thm 2 proof; relies on asymptotic scaling, not an exact equivalence.
  • standard math Johnson-Lindenstrauss lemma with +/-1 random projections gives near-orthogonal embeddings
    Used to reduce embedding dimensions in Thm 2-4.
  • ad hoc to paper Input tokens can be augmented with positional encoding or computed by an auxiliary layer (Prop 1)
    Required for Thm 1 and Cor 2; not part of the standard Mamba input.
invented entities (1)
  • Mamba-delta-top mixer independent evidence
    purpose: Variant of S6 where Delta varies along the state dimension instead of the embedding dimension; solves Induction Heads by selectively erasing outdated key columns.
    Lemma 3 gives an explicit construction with d=2|V|, N=|V|, and Fig 3/Fig A.4 show it empirically reaches 100% accuracy, providing a falsifiable handle outside the paper's own equations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity." pith.science (2026). https://pith.science/paper/3SEVVEOD

@misc{pith2026250611891,
  author       = {Pith},
  title        = {Pith review of: Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3SEVVEOD}},
  note         = {Machine review of arXiv:2506.11891}
}
read the original abstract

State-Space Models (SSMs), and particularly Mamba, have recently emerged as a promising alternative to Transformers. Mamba introduces input selectivity to its SSM layer (S6) and incorporates convolution and gating into its block definition. While these modifications do improve Mamba's performance over its SSM predecessors, it remains largely unclear how Mamba leverages the additional functionalities provided by input selectivity, and how these interact with the other operations in the Mamba architecture. In this work, we demystify the role of input selectivity in Mamba, investigating its impact on function approximation power, long-term memorization, and associative recall capabilities. In particular: (i) we prove that the S6 layer of Mamba can represent projections onto Haar wavelets, providing an edge over its Diagonal SSM (S4D) predecessor in approximating discontinuous functions commonly arising in practice; (ii) we show how the S6 layer can dynamically counteract memory decay; (iii) we provide analytical solutions to the MQAR associative recall task using the Mamba architecture with different mixers -- Mamba, Mamba-2, and S4D. We demonstrate the tightness of our theoretical constructions with empirical results on concrete tasks. Our findings offer a mechanistic understanding of Mamba and reveal opportunities for improvement.

Figures

Figures reproduced from arXiv: 2506.11891 by the authors.

Figure 1
Figure 1. Distribution of e −λ∆t computed on test inputs by models trained to successfully solve KEEP n-TH tasks for various sequence lengths T (same setup as Tab. 1). The left histogram confirms that the Mamba model must push e −λ∆t → 1 as T increases, as implied by Lem. 2, to retain information throughout the sequence. Also the S4D must behave similarly. Meanwhile, the right histogram shows that, compared to S4D, Mamba has … view at source ↗
Figure 2
Figure 2. Trained models accuracy on MQAR task (best of 7 seeds), varying κ and d. For S4D N = 4, for Mamba N = 2κ, and for Mamba-2 N = 8 ln κ. We use T = 100 and |V | = 128 for all runs. The theoretical bounds on model size for assembling the solutions proposed in Thm. 2 to 4 (black lines) separate reasonably well models that can achieve 100% accuracy (above black lines) from those that do not (below). In terms of model size… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages

  1. [1]

    Zoology: Measuring and Improving Recall in Efficient Language Models

    Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Re, C. Zoology: Measuring and Improving Recall in Efficient Language Models . In International Conference on Learning Representations , 2024 a

    work page 2024

  2. [2]

    Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff

    Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Zou, J., Rudra, A., and Re, C. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff . In International Conference on Machine Learning, volume 235, pp.\ 1763--1840, 2024 b

    work page 2024

  3. [3]

    Birth of a Transformer: A Memory Viewpoint

    Bietti, A., Cabannes, V., Bouchacourt, D., Jegou, H., and Bottou, L. Birth of a Transformer: A Memory Viewpoint . Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  4. [4]

    Theoretical Limitations of Multi-Layer Transformer

    Chen, L., Peng, B., and Wu, H. Theoretical Limitations of Multi-Layer Transformer . arXiv preprint, 2024. URL https://arxiv.org/abs/2412.02975

    arXiv 2024

  5. [5]

    and Cholak, P

    Chiang, D. and Cholak, P. Overcoming a Theoretical Limitation of Self-Attention . In Annual Meeting of the Association for Computational Linguistics, volume 1, pp.\ 7654--7664, 2022. doi:10.18653/v1/2022.acl-long.527

    doi:10.18653/v1/2022.acl-long.527 2022

  6. [6]

    M., Orvieto, A., Walker, B., Salvi, C., and Lyons, T

    Cirone, N. M., Orvieto, A., Walker, B., Salvi, C., and Lyons, T. Theoretical Foundations of Deep Selective State-Space Models . In Advances in Neural Information Processing Systems, volume 37, 2024

    work page 2024

  7. [7]

    A., and Verghese, G

    Dahleh, M., Dahleh, M. A., and Verghese, G. Lectures on Dynamic Systems and Control . MIT OpenCourseWare, 2011. URL https://ocw.mit.edu/courses/6-241j-dynamic-systems-and-control-spring-2011/996025f6db0d90b00f11c44fc49b85f9_MIT6_241JS11_textbook.pdf

    work page 2011

  8. [8]

    and Gu, A

    Dao, T. and Gu, A. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality . In International Conference on Machine Learning, volume 235, pp.\ 10041--10071, 2024

    work page 2024

Show all 32 references
  1. [9]

    Eckhoff, K. S. Accurate and Efficient Reconstruction of Discontinuous Functions from Truncated Series Expansions . Mathematics of Computation , 61 0 (204): 0 745--763, 1993. doi:10.1090/S0025-5718-1993-1195430-1

    1993 doi

  2. [10]

    and Inglese, G

    Gautschi, W. and Inglese, G. Lower Bounds for the Condition Number of Vandermonde Matrices . Numerische Mathematik , 52: 0 241--250, 1987. doi:10.1007/BF01398878

    1987 doi

  3. [11]

    K., Zela, A., Hutter, F., and Pontil, M

    Grazzi, R., Siems, J., Franke, J. K., Zela, A., Hutter, F., and Pontil, M. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues . In International Conference on Learning Representations, 2025

    2025

  4. [12]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces . arXiv preprint, 2023. URL https://arxiv.org/abs/2312.00752

    2023 arXiv

  5. [13]

    HiPPO: Recurrent Memory with Optimal Polynomial Projections

    Gu, A., Dao, T., Ermon, S., Rudra, A., and R \'e , C. HiPPO: Recurrent Memory with Optimal Polynomial Projections . Advances in Neural Information Processing Systems, 33: 0 1474--1487, 2020

    2020

  6. [14]

    On the Parameterization and Initialization of Diagonal State Space Models

    Gu, A., Goel, K., Gupta, A., and R \'e , C. On the Parameterization and Initialization of Diagonal State Space Models . Advances in Neural Information Processing Systems, 35: 0 35971--35983, 2022 a

    2022

  7. [15]

    Efficiently Modeling Long Sequences with Structured State Spaces

    Gu, A., Goel, K., and R\'e, C. Efficiently Modeling Long Sequences with Structured State Spaces . In International Conference on Learning Representations, 2022 b

    2022

  8. [16]

    Needle In A Haystack - Pressure Testing LLMs

    Kamradt, G. Needle In A Haystack - Pressure Testing LLMs . Github, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack. [Accessed 2025-05-29]

    2023

  9. [17]

    Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention . In International Conference on Machine Learning, pp.\ 5156--5165, 2020

    2020

  10. [18]

    Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization . arXiv preprint, 2014. URL https://arxiv.org/abs/1412.6980

    2014 arXiv

  11. [19]

    On the Power of Convolution Augmented Transformer

    Li, M., Zhang, X., Huang, Y., and Oymak, S. On the Power of Convolution Augmented Transformer . In Conference on Artifical Intelligence, number 17, pp.\ 18393--18402, 2025. doi:10.1609/aaai.v39i17.34024

    2025 doi

  12. [20]

    Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks

    Li, Z., Han, J., Weinan, E., and Li, Q. Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks . Journal of Machine Learning Research, 23 0 (42): 0 1--85, 2022

    2022

  13. [21]

    Group Invariant Scattering

    Mallat, S. Group Invariant Scattering . Communications on Pure and Applied Mathematics, 65 0 (10): 0 1331--1398, 2012. doi:10.1002/cpa.21413

    2012 doi

  14. [22]

    The Illusion of State in State-Space Models

    Merrill, W., Petty, J., and Sabharwal, A. The Illusion of State in State-Space Models . In International Conference on Machine Learning, 2024

    2024

  15. [23]

    In-context Learning and Induction Heads

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022

  16. [24]

    Orvieto, A., De, S., Gulcehre, C., Pascanu, R., and Smith, S. L. Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues . In International Conference on Machine Learning, 2024

    2024

  17. [25]

    One-Layer Transformers Fail to Solve the Induction Heads Task

    Sanford, C., Hsu, D., and Telgarsky, M. One-Layer Transformers Fail to Solve the Induction Heads Task . arXiv preprint, 2024 a . URL https://arxiv.org/abs/2408.14332

    2024 arXiv

  18. [26]

    Transformers, Parallel Computation, and Logarithmic Depth

    Sanford, C., Hsu, D., and Telgarsky, M. Transformers, Parallel Computation, and Logarithmic Depth . In International Conference on Machine Learning, 2024 b

    2024

  19. [27]

    The Expressive Capacity of State Space Models: A Formal Language Perspective

    Sarrof, Y., Veitsman, Y., and Hahn, M. The Expressive Capacity of State Space Models: A Formal Language Perspective . Advances in Neural Information Processing Systems, 37: 0 41202--41241, 2024

    2024

  20. [28]

    Crucial Aspects of Zero-Order Hold LPV State-Space System Discretization

    T \'o th, R., Felici, F., Heuberger, P., and Van den Hof, P. Crucial Aspects of Zero-Order Hold LPV State-Space System Discretization . IFAC Proceedings Volumes, 41 0 (2): 0 4952--4957, 2008

    2008

  21. [29]

    High-Dimensional Probability: An Introduction with Applications in Data Science

    Vershynin, R. High-Dimensional Probability: An Introduction with Applications in Data Science . 2018. doi:10.1017/9781108231596

    2018 doi

  22. [30]

    Wavelets, Approximation, and Compression

    Vetterli, M. Wavelets, Approximation, and Compression . IEEE Signal Processing Magazine, 18 0 (5): 0 59--73, 2001. doi:10.1109/79.952805

    2001 doi

  23. [31]

    Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

    Wang, S., Li, Z., and Li, Q. Inverse Approximation Theory for Nonlinear Recurrent Neural Networks . In International Conference on Learning Representations, 2024

    2024

  24. [32]

    Pointer Value Retrieval: A New Benchmark for Understanding the Limits of Neural Network Generalization

    Zhang, C., Raghu, M., Kleinberg, J., and Bengio, S. Pointer Value Retrieval: A New Benchmark for Understanding the Limits of Neural Network Generalization . arXiv preprint, 2021. URL https://arxiv.org/abs/2107.12580

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.