Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that synthetic sequence targets built from four parametric memory functions—exponential, polynomial, impulse, and Airy—produce a continuum of controllable tasks whose temporal structure dictates which architectures can…

desk verdict A useful synthetic benchmark for probing memory structure, but the central claim that training-loss minima measure approximation error is not established. read the letter →

arxiv 2506.05678 v3 pith:NGLOILZO submitted 2025-06-06 cs.LG

Haotian Jiang , Zeyu Bao , Shida Wang , Qianxiao Li This is my paper
classification cs.LG
keywords memoryfunctionssequencemodelingapproximationtheorytemporalconvolutionalnetworkstransformerslong-rangedependenciessyntheticbenchmarkstailenergy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the temporal structure of a sequence target, not just its overall difficulty, determines which architectures can approximate it. To do this, it generates synthetic targets from four memory functions—exponential decay, polynomial decay, a single distant impulse, and an Airy pattern—each with a parameter $\alpha$ that continuously tunes the strength of the temporal dependency. Training LSTM, S4D (a diagonal state-space model), TCN, and Transformer models on these targets produces loss-versus-$\alpha$ curves that confirm parts of existing theory and reveal new behaviors, including a tail-energy complexity measure that tracks TCN approximation error and a memory-dependent trade-off between the number of attention heads and per-head dimension. If correct, the framework turns approximation theory into a measurable benchmark: given a task's memory structure, the curves indicate which architecture will fit it and where capacity limits appear.

What carries the argument

The load-bearing object is the memory function $\rho(s,\alpha)$, a weight that assigns to each past input its influence on the current output, with the parameter $\alpha$ controlling how strong or distant the dependency is. Targets are built from the rule $y(t)=\sigma_1(\sum_{s=0}^{t}\rho(s,\alpha)\sigma_2(x(t-s)))$, which reduces to a convolution with $\rho$ as the impulse response when the activations are linear. Scaling functions $\mu_{\mathrm{exp}},\mu_{\mathrm{poly}},\mu_{\delta},\mu_{\mathrm{Ai}}$ map $\alpha$ to comparable regimes so that different memory functions are tested at matched strengths. The paper's new diagnostic is tail energy $C(\rho,s)=\sum_{t=s}^{T}|\rho(\pi(t),\alpha)|^2$, where $\pi$ reorders the memory function by decreasing magnitude; Theorem C.1 bounds the approximation error of a temporal convolutional network by this tail after the largest values are covered by channels. For Transformers, the corresponding mechanism is the effective rank of the attention matrix, whose spectrum decays more slowly with $\alpha$ for the exponential, polynomial, and Airy memories but stays concentrated for the impulse memory.

What would settle it

Regenerate the Airy targets exactly as written in Eq. (6) rather than with the scaling stated in Section 3.2, and compare the TCN loss curves; if the upward trend with $\alpha$ disappears, Observation 3 is an artifact of the implemented task rather than of sparsity. Separately, retrain every model with the minimum loss pushed to a plateau for all $\alpha$ (longer runs or more seeds); if large-$\alpha$ losses keep decreasing, the observed degradations reflect optimization, not approximation.

Watch

Extended reading notes

Core claim

The central claim is that a target of the form $y(t)=\sigma_1(\sum_{s=0}^t \rho(s,\alpha)\sigma_2(x(t-s)))$ acts as a controllable probe of sequence-model approximation, with the memory function $\rho(s,\alpha)$ fixing the temporal structure and $\alpha$ fixing its strength. For recurrent models, loss stays flat under exponential decay but climbs with $\alpha$ under polynomial decay and grows sharply as an impulse moves farther into the past. For TCNs, loss is insensitive to the impulse's location but rises when the Airy target becomes less sparse, a rise the paper quantifies with tail energy $C(\rho,s)=\sum_{t=s}^{T}|\rho(\pi(t),\alpha)|^2$ after sorting $|\rho|$ in decreasing order; a bound in Theorem C.1 connects this quantity to TCN approximation error. Transformers display TCN-like trends, and with hidden dimension fixed, the loss as a function of the number of heads is non-monotonic for the decay and impulse memories at large $\alpha$, while remaining nearly monotonic for Airy. The paper takes the minimum-over-seeds final training loss, after a train-test consistency check, as an upper bound on approximation error, so each observation is an assertion about approximation capacity rather than about training alone.

Load-bearing premise

The load-bearing premise is that the minimum-over-seeds final training loss, after checking train-test consistency, sits close enough to each model's true approximation floor at every $\alpha$ that the loss-versus-$\alpha$ trends describe approximation capacity rather than optimization shortfalls or memorization.

Editorial extensions

If this is right

  • Recurrent architectures (LSTM and S4D) approximate exponentially decaying targets well at every tested $\alpha$, but their error grows with $\alpha$ for polynomial decay and rises sharply when a target depends on a single distant impulse.
  • Temporal convolutional networks can track an impulse anywhere in the receptive field without added approximation error, yet their error rises when the Airy target's high-magnitude region widens; the tail-energy measure $C(\rho,s)$ follows this rise and is the quantity bounded in Theorem C.1.
  • Transformers with a fixed number of heads behave like TCNs on these targets: loss increases with $\alpha$ for exponential, polynomial, and Airy memories and stays flat for the impulse memory.
  • At a fixed total hidden dimension, the optimal number of attention heads is not universal; for decay memories the loss becomes non-monotonic in heads at large $\alpha$, for the impulse memory it is non-monotonic everywhere, and for Airy it remains nearly monotonic.
  • TCN optimization is unusually seed-sensitive, especially on impulse and Airy targets and with AdamW, so the stable-looking TCN trends in approximation require the multiple-seed minimum to be interpreted carefully.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We would extend the tail-energy diagnostic beyond TCNs: the same computation on any fixed-filter architecture with a receptive field should predict where approximation error concentrates, giving a cheap architecture-independent complexity fingerprint.
  • The framework's reliance on minimum-over-seeds training loss suggests a sharper cross-check: compare the measured curves to explicit approximation bounds computed from the hypothesis space (e.g., widths, ranks, or filter counts) on the very same targets, which would separate capacity effects from optimization shortfalls.
  • The non-monotonic heads-versus-dimension trade-off implies that Transformer hyperparameter selection cannot ignore the target's memory structure; a testable prediction is that on tasks with known long-range sparse dependencies, fewer heads with larger per-head dimension should win, while dense slow-decay tasks should favor more heads.
  • The Airy target contains an internal inconsistency between the definition in Eq. (6) and the scaling stated in Section 3.2; until the authors fix which one generated the data, the upward TCN loss trend for $\rho_{\mathrm{Ai}}$ should be treated as tentative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a synthetic benchmarking framework for sequence models. Targets are generated by y(t)=σ1( (1/t) Σ_{s=0}^{t} ρ(s,α) σ2(x(t−s)) ) with four parametric memory functions (exponential, polynomial, impulse, Airy) controlled by a strength parameter α. The authors evaluate LSTM, S4D, TCN, and Transformer architectures and report six observations about how loss varies with α across memory functions, including a proposed tail-energy complexity measure C(ρ,s) for TCNs and a non-monotonic head-count trade-off for Transformers. The central premise is that the minimum final training loss over seeds, after checking train/test consistency, serves as an upper bound on the approximation error.

Significance. If the identification between training loss and approximation error were solid, this would be a valuable controlled benchmark for studying sequence-model approximation behavior, complementing prior theoretical work by the same group. The paper's strengths are its explicit target definitions, the breadth of architectures and memory structures tested, and the fact that the empirical trends could in principle refute the cited theories, so the benchmark is not circular at the equation level. However, the quantitative support for the central claims is currently weak: the approximation-error interpretation is not validated against optimization failures, the Airy target definition is internally inconsistent, and the proposed TCN approximation theorem is only sketched and appears incomplete.

major comments (3)
  1. [Section 3.1 / Appendix B.1] The claim that the minimum final training loss over seeds estimates the approximation error e=inf_{ŷ∈H} ||y−ŷ|| requires that training reaches the approximation floor for every (architecture, α) pair, but the paper only checks train/test consistency, which excludes overfitting, not underoptimization. The paper itself documents large seed-to-seed variance for TCNs (Appendix E.1, Figs. 8-9) and optimizer-dependent behavior (Observation 4), so the upward loss trends in Figures 1-4 for large α, and especially the non-monotonic head trade-off in Figure 4, may reflect optimization difficulty rather than approximation capacity. Without an independent check that the training loss has converged to a global minimum (e.g., convergence to a plateau across many seeds and optimizers, or comparison with a directly computed approximation bound), the central conclusion that these curves reveal approximation behavior is not established.
  2. [Section 3.2 / Eq. (6)] The Airy memory function is defined inconsistently. Eq. (6) defines ρAi(s,α)=Aitr(μAi(s−c·α)) with μAi:[0,1]→R≥0 and c>0 left unspecified, while Section 3.2 states that μAi is chosen so that ρAi(s,α)=Aitr(40(s/T − α/2)). These two forms are not equivalent: the first applies a function μAi to (s−cα), while the second applies a fixed affine map 40(s/T − α/2) that depends on T and is not a function of α alone. The truncation at s=20 in Eq. (5) also interacts with this argument transformation in a way that is never specified for the reported T values. Since the Airy experiments are used in Observations 3 and 5, the actual tasks used in those experiments must be clarified or re-run under a single consistent definition.
  3. [Appendix C, Theorem C.1 / Eq. (9)] The proposed tail-energy complexity measure is not well-defined, and the theorem is not proved. In Eq. (9), C(ρ,s) is written without an α argument, yet the right-hand side sums |ρ(π(t),α)|^2; since ρ is a function of both s and α, the measure must either be defined for each α or the notation must be changed. The permutation π is described only informally, and it is not stated whether π depends on α or on the model class. In the proof of Theorem C.1, the equality in Eq. (19) is asserted without justification: a permutation of the summation index cannot turn the total variation over s=0,...,lK−1 into the tail sum C(ρ,M+1), and the statement 'We use the M channels to cover the largest M values of ρ' is not a proof that the remainder is bounded by C(ρ,M+1). Since Theorem C.1 is cited in Observation 3 as the theoretical basis for the tail-energy bound, this missing support weakens the central TCN claim.
minor comments (5)
  1. [Observation 3] The figure citations are swapped: the text says 'As shown in Figure 2c, TCNs exhibit increased approximation error under the Airy memory function ρAi' but the Airy panel is Figure 2d, while the impulse panel is Figure 2c; the following sentence also cites Figure 2d for the impulse result.
  2. [Section 3, Eq. (9) and Figure 6] The statement that C(ρδ,s) is 'constant zero' is imprecise: for an impulse memory function with one nonzero value, the monotone rearrangement gives C(ρδ,0)=1 and C(ρδ,s)=0 for s≥1, so the statement requires qualification.
  3. [Figures 1-4] The main loss-versus-α figures do not show error bars or seed distributions, despite the text claiming experiments over multiple seeds; adding min-max or percentile bands would greatly help assess the reliability of the reported trends.
  4. [Appendix D.1] The text refers to 'LSTM, S4, and TCN' but the main text uses S4D; please harmonize the notation.
  5. [Section 3.2, Eq. (8)] The claim that the RMSE denominator is 'always positive' is not obviously true for arbitrary input x and nonlinear activations σ1,σ2; if zero-energy outputs are possible in principle, this needs clarification.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found: targets are explicit equations, loss curves are empirical and falsifiable, and self-citations are independent theoretical support rather than load-bearing definitions.

full rationale

The derivation chain is not circular. The targets are generated from explicit closed-form memory functions (Eqs. (1)-(6)) and prescribed scaling functions (Section 3.2), with no parameter fitted to the measured losses; the loss-vs-alpha curves (Figs. 1-4) are empirical and could have come out differently, e.g., TCN on rho_delta could have degraded but is reported stable and Transformers could have differed from TCN but are reported similar. The tail-energy measure C(rho,s) in Eq. (9) is computed solely from rho by a decreasing reordering; it contains no loss data or fitted constants, so its correlation with loss (Figs. 6-7) is a post hoc but testable empirical relation, not a quantity defined from the loss it is said to predict. Theorem C.1 is a stated consequence of Jiang et al. (2021, Eq. (18)) and does not import the empirical observations. The theoretical interpretations cite several prior works by the authors (Jiang et al. 2021; Jiang & Li 2024; Li et al. 2022; Wang et al. 2023; Wang & Li 2024), but these are parameter-free theorems with stated assumptions and the present experiments could refute them, so the citations are not load-bearing in a circular sense. Appendix D.5 discloses limitations (single-layer, single-filter, restricted memory forms) and Appendix E.1 discloses TCN seed variance; these are robustness caveats, not circularity. Non-circular correctness risks for the reviewer: Appendix B.1 uses the minimum-seed training loss as an approximation-error upper bound, which presumes optimization reaches the approximation floor; and the Airy memory function is internally inconsistent (Eq. (6) writes rho_Ai(s,alpha)=Aitr(mu_Ai(s-c alpha)) while Section 3.2 defines rho_Ai(s,alpha)=Aitr(40(s/T-alpha/2))), affecting reproducibility of the Airy experiments but not making any prediction equivalent to its input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim rests on hand-chosen scaling and truncation constants, on the convolution representation theorem for LTI targets, and on the assumption that training loss proxies approximation error. The tail-energy measure is a new construct whose predictive value is demonstrated only on the same four memory functions used to motivate it.

free parameters (3)
  • alpha_max for polynomial scaling (mu_poly) = 0.3
    Chosen by hand in Appendix D.1 so that the tail sum of rho_poly beyond T is at the 1e-8 scale; this controls the range of temporal strengths tested and affects all polynomial-target loss curves.
  • Airy truncation threshold = 20
    The Airy function is truncated at s = 20 in Eq (5) because the authors judge it negligible beyond this point; this approximation is used to define the Airy target.
  • Airy scaling factor (40/T and shift speed) = 40/T implied
    Section 3.2 fixes the Airy argument as 40(s/T - alpha/2), which effectively sets the shift speed and sparsity rate; this is a hand-chosen scaling that determines how alpha changes the target.
assumptions (5)
  • domain assumption A linear, continuous, causal, time-homogeneous target has a convolution representation with memory function rho (Lemma 15 of Li et al. 2022).
    Invoked in Appendix B to justify using memory functions as the defining structure of the benchmark; it restricts the target class to LTI-like systems.
  • domain assumption The final training loss, minimized over random seeds, is an upper bound estimate of the approximation error when no overfitting occurs.
    Stated in Appendix B.1 and used throughout Section 4 to interpret loss curves as approximation error; it assumes optimization reaches the model's capacity and that the train-test gap is negligible.
  • domain assumption Nonlinear targets with tanh activations and iid Gaussian inputs behave similarly to linear convolution targets for the purposes of approximation theory.
    The paper extends linear theoretical results to the nonlinear targets of Eq (1) without proving that the memory-function structure is preserved; this is an unstated transfer assumption.
  • standard math The prior approximation bounds of Jiang et al. 2021 and Li et al. 2022 are correct and applicable to the architectures used here.
    The cited theorems are used to interpret observations and to derive Theorem C.1; the paper does not re-derive them.
  • ad hoc to paper The tail-energy complexity measure C(rho,s) with a monotone rearrangement pi is a meaningful measure of TCN approximation difficulty.
    Equation (9) is introduced post hoc, and no theorem proves it is the unique or correct complexity measure; the correlation with loss is empirical.
invented entities (1)
  • Tail-energy complexity measure C(rho,s) independent evidence
    purpose: Quantifies the decay of sorted memory-function values to predict TCN approximation error.
    It is a new measure introduced in Eq (9); it makes a falsifiable prediction that larger C corresponds to larger TCN loss, which the paper checks on four memory functions, but it has not been tested on novel targets outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions." pith.science (2026). https://pith.science/paper/NGLOILZO

@misc{pith2026250605678,
  author       = {Pith},
  title        = {Pith review of: Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGLOILZO}},
  note         = {Machine review of arXiv:2506.05678}
}
abstract

The evolution of sequence modeling architectures, from recurrent neural networks and convolutional models to Transformers and structured state-space models, reflects ongoing efforts to address the diverse temporal dependencies inherent in sequential data. Despite this progress, systematically characterizing the strengths and limitations of these architectures remains a fundamental challenge. In this work, we propose a synthetic benchmarking framework to evaluate how effectively different sequence models capture distinct temporal structures. The core of this approach is to generate synthetic targets, each characterized by a parametric memory function $\rho(s, \alpha)$ and a controllable parameter $\alpha$ that determines the temporal strength. This setup allows us to produce a continuum of tasks that vary in temporal complexity, enabling fine-grained analysis of model behavior with respect to specific memory properties. We focus on four representative memory functions, each corresponding to a distinct class of temporal structures: exponential and polynomial functions for decay dynamics, impulse functions for long-range dependencies, and Airy functions for sparsity patterns. Experiments on several sequence modeling architectures confirm existing theoretical insights and reveal new findings regarding approximation capabilities, optimization dynamics, and architectural trade-offs. These results demonstrate the effectiveness of the proposed method in advancing theoretical understanding and highlight the importance of using controllable targets with clearly defined structures for evaluating sequence modeling architectures.

Figures

Figures reproduced from arXiv: 2506.05678 by the authors.

Figure 1
Figure 1. Loss versus α for LSTM and S4D on ρexp, ρpoly, and ρδ, with varying m. Observation 1. Consistent with existing theoretical understanding, the approximation behavior of the recurrent architectures is strongly influenced by the decay properties of ρexp and ρpoly, as well as the dependency range of the impulse memory function ρδ. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Loss versus α for TCN on ρexp, ρpoly, ρδ and ρAi, with varying m. Observation 2. The approximation behavior of TCN is not affected by the dependency range of ρδ. In Figure 2d, TCNs maintain stable performance across varying α under the impulse memory function ρδ, where the output depends on a single input at a distant time step. This demonstrates TCNs’ ability to capture long-range dependencies through dilated convo… view at source ↗
Figure 3
Figure 3. Loss versus α for Transformer on ρexp, ρpoly, ρδ and ρAi, with fixed nh = 4 and varying m. From an approximation theory perspective, Jiang & Li (2024) provides a more consistent interpreta￾tion. It shows that the approximation difficulty of Transformers is closely related to the effective rank of the target function. When applied to our setting, this implies that the effective rank of the attention matrix increases … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Loss versus nh for Transformers with fixed m = 128 and varying α. 5 Conclusion This work introduces a synthetic framework for systematically analyzing the theoretical properties of sequence models. By designing targets with controlled memory patterns, we isolate key fa…
Figure 5
Figure 5. Figure 5: Plot of individual memory functions for T = 512, illustrating variations with respect to the parameter α. The colorbar indicates the corresponding α values. The scaling functions µδ and µAi used for these plots are the same as those defined in Section 3.1, while µexp a…
Figure 6
Figure 6. Figure 6: The plot of complexity measures defined in Equation (9) for different memory functions. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: The plot of loss vs complexity measures defined in Equation (9) models with [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Loss versus α for TCN with Adam optimizer on ρexp, ρpoly, ρimpulse, and ρAi. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Loss versus α for TCN with AdamW optimizer on ρexp, ρpoly, ρimpulse, and ρAi. 0.25 0.50 0.75 1.00 2 4 Loss 1e 9 (a) LSTM, ρexp (Adam) 0.25 0.50 0.75 1.00 2 4 Loss 1e 9 (b) LSTM, ρpoly (Adam) 0.0 0.5 1.0 10 7 10 5 10 3 10 1 Loss (c) LSTM, ρδ (Adam) 0.0 0.5 1.0 10 7 10 5…
Figure 10
Figure 10. Figure 10: Loss versus α for LSTM with Adam optimizer on ρexp, ρpoly, ρimpulse, and ρAi. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: The plot of the spectrum of the attention matrix [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Plot of loss vs α for Transformers with different type of positional encoding on memory function ρδ. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: The number of training steps to reach a certain training error [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: The number of training steps to reach a certain training error [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Cosine similarity between gradients computed with varying batch sizes and the full dataset [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 25 canonical work pages

  1. [1]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  2. [2]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  3. [3]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  4. [4]

    Quality over Quantity in Attention Layers : When Adding More Heads Hurts

    Noah Amsel, Gilad Yehudai, and Joan Bruna. Quality over Quantity in Attention Layers : When Adding More Heads Hurts . In The Thirteenth International Conference on Learning Representations , October 2024

    work page 2024

  5. [5]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , April 2018

    work page 2018

  6. [6]

    LongBench : A Bilingual , Multitask Benchmark for Long Context Understanding , June 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench : A Bilingual , Multitask Benchmark for Long Context Understanding , June 2024

    work page 2024

  7. [7]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 0 (2): 0 157--166, March 1994. ISSN 1941-0093. doi:10.1109/72.279181

    doi:10.1109/72.279181 1994

  8. [8]

    On the Relationship between Self-Attention and Convolutional Layers

    Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the Relationship between Self-Attention and Convolutional Layers . In arXiv :1911.03584 [Cs, Stat] , January 2020

    arXiv 1911

Show all 35 references
  1. [9]

    BAMBOO : A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models , March 2024

    Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. BAMBOO : A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models , March 2024

    2024

  2. [10]

    The Pile : An 800GB Dataset of Diverse Text for Language Modeling , December 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile : An 800GB Dataset of Diverse Text for Language Modeling , December 2020

    2020

  3. [11]

    Efficiently Modeling Long Sequences with Structured State Spaces , August 2022 a

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces , August 2022 a

    2022

  4. [12]

    On the Parameterization and Initialization of Diagonal State Space Models , August 2022 b

    Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the Parameterization and Initialization of Diagonal State Space Models , August 2022 b

    2022

  5. [13]

    Long Short-Term Memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory . Neural Computation, 9 0 (8): 0 1735--1780, November 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.8.1735

    1997 doi

  6. [14]

    RULER : What 's the Real Context Size of Your Long-Context Language Models ?, August 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER : What 's the Real Context Size of Your Long-Context Language Models ?, August 2024

    2024

  7. [15]

    Approximation Rate of the Transformer Architecture for Sequence Modeling

    Haotian Jiang and Qianxiao Li. Approximation Rate of the Transformer Architecture for Sequence Modeling . In The Thirty-eighth Annual Conference on Neural Information Processing Systems , November 2024

    2024

  8. [16]

    Approximation Theory of Convolutional Architectures for Time Series Modelling

    Haotian Jiang, Zhong Li, and Qianxiao Li. Approximation Theory of Convolutional Architectures for Time Series Modelling . In Proceedings of the 38th International Conference on Machine Learning , pp.\ 4961--4970. PMLR, July 2021

    2021

  9. [17]

    Krizhevsky

    A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images . In Technical Report 0 , 2009

    2009

  10. [18]

    Can Vision Transformers Perform Convolution ?, November 2021

    Shanda Li, Xiangning Chen, Di He, and Cho-Jui Hsieh. Can Vision Transformers Perform Convolution ?, November 2021

    2021

  11. [19]

    Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks

    Zhong Li, Jiequn Han, Weinan E, and Qianxiao Li. Approximation and Optimization Theory for Linear Continuous-Time Recurrent Neural Networks . Journal of Machine Learning Research, 23 0 (42): 0 1--85, 2022. ISSN 1533-7928

    2022

  12. [20]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis . In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational L...

    2011

  13. [21]

    Pointer Sentinel Mixture Models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models . In International Conference on Learning Representations , February 2017

    2017

  14. [22]

    Stable Recurrent Models , March 2019

    John Miller and Moritz Hardt. Stable Recurrent Models , March 2019

    2019

  15. [23]

    Alan Oppenheim, Alan Willsky, and S. Nawab. Signals and Systems . Pearson, Upper Saddle River, NJ, 2nd edition edition, October 1996. ISBN 978-0-13-814757-0

    1996

  16. [24]

    Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De

    Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting Recurrent Neural Networks for Long Sequences , March 2023

    2023

  17. [25]

    The LAMBADA dataset: Word prediction requiring a broad discourse context, June 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context, June 2016

    2016

  18. [26]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323 0 (6088): 0 533--536, October 1986. ISSN 1476-4687. doi:10.1038/323533a0

    1986 doi

  19. [27]

    Self- Attention with Relative Position Representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self- Attention with Relative Position Representations . arXiv:1803.02155 [cs], April 2018

    2018 arXiv

  20. [28]

    RoFormer : Enhanced Transformer with Rotary Position Embedding , November 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer : Enhanced Transformer with Rotary Position Embedding , November 2023

    2023

  21. [29]

    Long Range Arena : A Benchmark for Efficient Transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena : A Benchmark for Efficient Transformers . In International Conference on Learning Representations , October 2020

    2020

  22. [30]

    WaveNet : A Generative Model for Raw Audio

    Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet : A Generative Model for Raw Audio . arXiv:1609.03499 [cs], September 2016

    2016 arXiv

  23. [31]

    Attention is All you Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017

    2017

  24. [32]

    StableSSM : Alleviating the Curse of Memory in State-space Models through Stable Reparameterization

    Shida Wang and Qianxiao Li. StableSSM : Alleviating the Curse of Memory in State-space Models through Stable Reparameterization . In Proceedings of the 41st International Conference on Machine Learning , pp.\ 50766--50793. PMLR, July 2024

    2024

  25. [33]

    State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory

    Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory. In Thirty-Seventh Conference on Neural Information Processing Systems , November 2023

    2023

  26. [34]

    Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

    Shida Wang, Zhong Li, and Qianxiao Li. Inverse Approximation Theory for Nonlinear Recurrent Neural Networks . In The Twelfth International Conference on Learning Representations , October 2023

    2023

  27. [35]

    Do RNN and LSTM have Long Memory ? In Proceedings of the 37th International Conference on Machine Learning , pp.\ 11365--11375

    Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do RNN and LSTM have Long Memory ? In Proceedings of the 37th International Conference on Machine Learning , pp.\ 11365--11375. PMLR, November 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.