Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a learnable two-sided short-time Laplace transform, with tunable decay rates, frequencies, and window bandwidth, can replace self-attention and process sequences of over 100,000 tokens in linear time and…

desk verdict The O(NS) scalability claim is contradicted by the paper's own primary forward pass; an interesting Laplace-basis idea, but as written this should not go to review. read the letter →

arxiv 2506.15714 v1 pith:VZIHUDZQ submitted 2025-06-01 cs.LG cs.CL

Andrew Kiruluta This is my paper
classification cs.LGcs.CL
keywords two-sidedshort-timeLaplacetransformlearnabledecayratesself-attentionreplacementlinearcomplexityefficienttransformerslong-contextlanguagemodelingadaptivenodeallocationinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that self-attention can be replaced by a two-sided short-time Laplace transform with learnable decay rates, oscillation frequencies, and window bandwidth, so that each token is represented by a small bank of S complex-valued coefficients. The intended payoff is a transformer that mixes global context in O(N S) time and O(S) memory instead of O(N²), making sequences of 100,000 tokens or more practical on a single GPU. The learned parameters are claimed to be interpretable, with half-lives t_{1/2,k} = ln 2/σ_k describing how far each node reaches and frequencies ω_k describing recurrent patterns. The paper reports that this STLT layer matches or beats several efficient-transformer baselines on four standard benchmarks and that ablations confirm the learnable parameters and adaptive node count are what drive the gains.

What carries the argument

The load-bearing object is the exponential recurrence for the two-sided short-time Laplace coefficient. Each Laplace node s_k defines a sliding windowed complex exponential $e^{{-s_k mΔ}}$; the identity $e^{{-s_k(m+1)Δ}}$ = r_k $e^{{-s_k mΔ}}$ with r_k = $e^{{-s_kΔ}}$ lets the contribution of node k be updated from token to token in O(1) per position, giving the two linear passes that avoid an N×N buffer. The paper pairs this with an adaptive gating mechanism that learns how many of the S candidate nodes are actually needed per input, so the effective node count S_eff can be smaller than the configured maximum. The relevance matrix R_{n,m} = Σ_k L_{n,k} L_{m,k} and the softmax output Z = softmax(R/√S)V are the attention-surrogate definitions on top of these coefficients.

What would settle it

Measure wall-clock time and peak GPU memory on sequences of length N = 1,024, 2,048, 4,096, 8,192, ... with fixed S: if either quantity grows faster than linearly in N (or superlinearly in the streaming regime), the O(N S) claim fails. A direct check is to instrument the STLT forward pass to raise an error whenever an N×N array would be allocated; the claimed streaming formulation must never trigger it. Alternatively, verify whether the learned window can be updated by the recurrence for a non-exponential w(u−τ;T); a Hann or other finite-support window requires a convolution of cost O(T_win) per position, which changes the stated complexity.

Watch

Extended reading notes

Core claim

The paper's central claim is that a bank of S learnable Laplace nodes s_k = σ_k + jω_k, each carrying its own decay and oscillation, can act as the mixing operator that self-attention provides. For every position n, the layer computes L_{n,k} = Σ_{m=1}^{N} x_m w((m−n)Δ; T) $e^{{-s_k mΔ}}$, then defines token-to-token relevance as R_{n,m} = Σ_k L_{n,k} L_{m,k} and produces outputs Z = softmax(R/√S)V. Because the exponential $e^{{-s_k mΔ}}$ satisfies the first-order recurrence $e^{{-s_k(m+1)Δ}}$ = $e^{{-s_kΔ}}$ $e^{{-s_k mΔ}}$, the paper argues the coefficients can be accumulated in one forward and one backward streaming pass without materializing any N×N array, yielding O(N S) time and O(S) memory with S ≪ N. On top of this, a differentiable gating mechanism learns an effective node count S_eff per input, and a reconstruction-error bound is offered showing that S = O(log N) nodes suffice to keep the approximation error below any ε. The paper then presents experiments on language modeling, translation, and long-document QA in which the STLT transformer is at parity with or better than Linformer, FNet, Performer, Longformer, and Mamba, and scales to contexts over 100k tokens.

Load-bearing premise

The linear-cost claim stands on the assumption that the forward pass can be completed by the exponential recurrence without ever constructing the N×N relevance matrix R_{n,m} = Σ_k L_{n,k}L_{m,k}; the paper's own primary formulation defines Z = softmax(R/√S)V with exactly that matrix, and the recurrence as written applies to the exponential factor, not to a general window function.

Editorial extensions

If this is right

  • If the O(N S) streaming recurrence works as stated, transformer-style models can ingest entire books or multi-hour audio in one pass, and chunked pipelines become unnecessary for many long-document tasks.
  • The learned half-lives t_{1/2,k} = ln 2/σ_k give each node a concrete temporal meaning, so a trained STLT layer can be audited: shallow layers with short half-lives track local syntax, deeper layers with long half-lives track discourse structure.
  • Because node allocation is adaptive, the model can spend more compute on hard inputs and near-zero compute on easy ones, giving a controllable accuracy–speed trade-off per instance rather than per architecture.
  • With S = O(log N) nodes and the paper's error bound, the method claims O(N log N) approximation with provable convergence, making it a candidate for settings where linear-attention approximations need a priori guarantees.
  • The hybrid encoder–decoder design (bilateral STLT in the encoder, unilateral in the decoder) provides a drop-in replacement that preserves autoregressive causality without a causal mask computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the quadratic relevance matrix R_{n,m} = Σ_k L_{n,k}L_{m,k} is what is actually used in the primary formulation, then the empirically reported linear scaling (Section 4.6) cannot come from that formulation unless the softmax is approximated or the matrix is never fully materialized; reconciling this is the key to testing the paper's central efficiency claim.
  • A natural extension would be to replace the N×N softmax with a streaming softmax over the per-node coefficients, or to parameterize the window itself as a decaying exponential so the whole layer becomes a bank of IIR filters; that would turn the STLT layer into a close cousin of state-space models and let it reuse their optimized kernels.
  • The paper's error bound suggests an empirical test: measure the spectral decay of the learned relevance matrix R on real text; if its effective rank stays small, the relevance can in principle be computed through a low-rank factorization, which would justify the linear claim even if the current implementation is quadratic.
  • If the learned frequencies ω_k cluster at a few characteristic periods, those periods could serve as a compact diagnostic summary of a model's attention patterns, analogous to a periodogram of the sequence dynamics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing self-attention in transformers with a learnable two-sided short-time Laplace transform (STLT). Each Laplace node has trainable decay σ_k, frequency ω_k, and window bandwidth T, and the number of active nodes can be adapted per input. The authors claim O(N S) time and O(S) memory complexity, interpretable parameters, and competitive results on WikiText-103, Project Gutenberg, WMT'14 En-De, and NarrativeQA. The central contribution is the scalability claim: that the STLT relevance computation can be done without materializing an N×N matrix. The paper also presents a theoretical error bound claiming S = O(log N) suffices for arbitrary accuracy.

Significance. If the O(N S) time and O(S) memory claims were established, the proposed STLT would be a useful contribution: it combines a learnable and interpretable time-frequency representation with linear scaling, and the adaptive node allocation is a sensible mechanism for trading off cost and capacity. The empirical results, however, are presented for a forward pass that the paper itself defines as quadratic, and the theoretical error-bound section explicitly defers the key derivation to future work. The paper provides no code, no machine-checked proofs, and no reproducible implementation details for the streaming algorithm. Thus the central claim of the paper—linear-time replacement for self-attention—is not supported by the manuscript as written.

major comments (4)
  1. [§3.4, Figure 1] The primary forward pass is defined as R_{n,m} = Σ_{k=1}^S L_{n,k} L_{m,k} and Z = softmax(R/√S)V, with R an N×N relevance matrix. Forming R requires Θ(N^2 S) operations and storing R requires Θ(N^2) memory, and the row-wise softmax is Θ(N^2). This directly contradicts the abstract's claim of O(N S) time and O(S) memory. The linear variant Z_n = Σ_k L_{n,k} V'_k is mentioned only as an alternative ('similar to attention heads'), and the paper does not state that the experiments used it. The scalability claim is therefore unsupported for the formulation actually used in the experiments.
  2. [§3.3 and Eq. (3)–(4)] The streaming recurrence only updates the factor e^{-s_k mΔ}: e^{-s_k(m+1)Δ} = r_k e^{-s_k mΔ}. It does not update the window w((m−n)Δ; T) as n advances. For a finite-support window such as a Hann window, each output L_{n,k} requires contributions from O(T_win) shifted window values, and the O(1) per-node update is only valid if the window is itself exponential. The paper uses a symmetric Hann-type window in Section 3.1, so the claimed O(S d) memory and two-pass streaming algorithm do not follow from the recurrence as written.
  3. [§3.7] The total reconstruction error bound ∥x(τ) − x̂(τ)∥ ≤ C_1 e^{-Bτ} + C_2 B (1/S^p) + C_3 e^{-T σ_min} is presented as a theorem-like result, but the section immediately states that 'future theoretical work could refine these bounds' and that analyzing the impact of learned, data-dependent distributions of s_k is left for the future. The choice B ∝ S, T ∝ ln S, and S = O(log N) is asserted without proof, so the claim that the error can be driven below any ε > 0 is not actually established. The downstream claim that ∥ΔR∥ ≲ 10^{-2} implies less than 0.2 perplexity/BLEU change is supported only by an informal empirical observation, not by a derived bound.
  4. [§4.6] The wall-clock and memory benchmark claims (e.g., N = 131072 fitting in 32GB) are attributed to the streaming recurrence in Section 3.3, but, as noted above, that recurrence does not compute the primary softmax-based forward pass of Section 3.4. If the benchmark used the linear variant Z_n = Σ_k L_{n,k} V'_k instead, the paper must say so explicitly and must identify which results in Tables 1–3 were produced with which forward pass. Without that specification, the efficiency numbers cannot be reproduced or verified, and they do not validate the complexity of the primary formulation.
minor comments (5)
  1. [Abstract and §1] The abstract uses 'O(N S) in time and O(S) memory' but the introduction also refers to 'constant memory overhead O(S)' and later 'O(N Seff)' without clarifying whether Seff is the same as S; please define all complexity symbols consistently.
  2. [§2] There is a typo 'F ourier mixing(FNet [12]))' with a double closing parenthesis, and the bullet list formatting is inconsistent.
  3. [§3.2] The sentence 'Both variants cost O(N Sd) (assuming the window has support proportional to N, or O(N ST′d) if window support T′ is explicit and T′ ≪ N)' is confusing because the direct summation in Eq. (3) costs O(N S d) only if the sum over m is over all N; for a finite-support window the cost is O(N S T_win d). Please write the complexity in terms of the window support explicitly.
  4. [§4.4, Table 4] The ablation row 'No node regularization (Rmask with λmask = 0)' reports perplexity 23.7 and Seff ≈ 55, which is better than the full model's 23.8 with Seff ≈ 28; the text claims removing regularization gives 'minimal performance gain,' but the numbers show a small gain at higher cost, which warrants a more precise discussion.
  5. [References] Reference [23] lists the author as 'Sinxin Wang,' which should be 'Sinong Wang'; please verify the spelling.

Circularity Check

1 steps flagged · score 6.0 of 10

Central O(NS) complexity claim is definitional for the linear variant, while the primary formulation is quadratic (R = LL^T, Z = softmax(R)V).

  1. other [Abstract; Section 3.3; Section 3.4; Figure 1; Section 4.6]
    "By selecting S ≪ N learnable nodes and leveraging fast recursive convolution, we achieve an effective complexity of O(N S) in time and O(S) memory. ... Token-to-token relevance is uniformly defined by R_{n,m} = Σ_{k=1}^S L_{n,k} L_{m,k} ... In our primary formulation, Z = softmax(R)V is used as per the figure description. ... The streaming recurrence (Section 3.3) is critical for these results, avoiding instantiation of any N × N matrices."

    The linear scalability claim is carried by the streaming recurrence for the coefficients L_{n,k} and by the secondary context-vector rule Z_n = Σ_k L_{n,k} V'_k. But the primary formulation, as specified in Section 3.4 and Figure 1, explicitly forms the N×N relevance matrix R_{n,m} = Σ_k L_{n,k} L_{m,k} and computes Z = softmax(R/√S)V. Forming R and applying a row-wise softmax costs Ω(N^2) for fixed S; no streaming or approximate softmax for R is provided. The O(NS) 'prediction' is therefore not derived from the model as defined; it is true only for a different output rule, which is substituted into the argument. The scalability result reduces to a definitional choice of the linear variant, while the reported primary and experimental model remains quadratic.

full rationale

The paper contains one central equivocation rather than a clean derivation chain. The abstract and Section 5 claim O(NS) time and O(S) memory for the STLT layer, and Section 4.6 reports wall-clock linear scaling. However, Section 3.4 and Figure 1 define the primary forward pass as R = LL^T followed by Z = softmax(R/√S)V, which is quadratic in the sequence length regardless of S. The streaming recurrence in Section 3.3 only computes the coefficients L_{n,k}; it does not compute the N×N softmax without materializing R. Thus the central scalability claim is not a consequence of the primary formulation but of a different linear context-vector rule that is presented as an alternative in the same section. This is a definitional substitution: the claimed result is built into the linear variant and then ascribed to the quadratic primary model. The theoretical error-bound section is explicitly incomplete, deferring refinements to 'future theoretical work', and the empirical “validation” of the bounds is a post-hoc observation without supporting data, but these are correctness/completeness concerns rather than additional circular steps. There is one self-citation (reference [10], same first author) used to frame the STLT as a strict generalization of the LMWT; this is a novelty claim and is not load-bearing for the recurrence, the error bound, or the empirical results. Because the central complexity result reduces by construction to a chosen linear output rule while the model as defined and used is quadratic, a score of 6 is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central scalability claim rests on assumptions about the kernel form and the recurrence that are not proven for general windows, and the empirical validation is not backed by released artifacts. The derived error bound is asserted with unspecified constants. The learnable parameters σk, ωk, and T are the core free parameters of the method, but the paper does not report their learned values, so their role cannot be independently assessed.

free parameters (4)
  • σk (per-node decay rates) = learned, values not reported
    Central to the learnability claim; updated by gradient descent during training; no learned values are reported in the paper.
  • ωk (per-node oscillation frequencies) = learned, values not reported
    Trained end-to-end; the paper reports only qualitative observations of learned frequencies.
  • T (window bandwidth) = learned, values not reported
    Trained end-to-end; initial values are given but learned values are only described qualitatively.
  • Wα, bα (node gating network weights) = learned, values not reported
    Parameters of the adaptive node allocation mechanism; trained with the rest of the model.
assumptions (4)
  • domain assumption Token-to-token relevance can be well-approximated by a low-rank sum of exponentially damped oscillations.
    This is the core modeling premise for replacing attention with STLT nodes; it is not justified beyond an analogy to physical systems.
  • ad hoc to paper The discrete exponential recurrence e^{-s_k(m+1)Δ} = r_k e^{-s_k mΔ} is sufficient to compute the windowed STLT in O(1) per node.
    The recurrence applies to the exponential only; a general window w((m−n)Δ; T) breaks the simple first-order recurrence unless the window is itself exponential or the filter is implemented as a finite-impulse-response convolution with cost O(T_win).
  • ad hoc to paper Choosing B ∝ S, T ∝ ln S, and S = O(log N) drives the total reconstruction error below any ε > 0.
    Stated in Section 3.7 without proof; constants C1, C2, C3 are not derived and the quadrature bound is asserted.
  • ad hoc to paper Relevance matrix error ∥ΔR∥ ≲ 10^{-2} implies downstream perplexity or BLEU degradation below 0.2 points.
    Claimed as empirically observed in Section 3.7 with no accompanying data or analysis; used to validate the theoretical bounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention." pith.science (2026). https://pith.science/paper/VZIHUDZQ

@misc{pith2026250615714,
  author       = {Pith},
  title        = {Pith review of: Adaptive Two Sided Laplace Transforms: A Learnable, Interpretable, and Scalable Replacement for Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZIHUDZQ}},
  note         = {Machine review of arXiv:2506.15714}
}
read the original abstract

We propose an innovative, learnable two-sided short-time Laplace transform (STLT) mechanism to supplant the traditional self attention in transformer-based LLMs. Our STLT introduces trainable parameters for each Laplace node, enabling end-to-end learning of decay rates , oscillatory frequencies, and window bandwidth T. This flexibility allows the model to dynamically adapt token relevance half lives and frequency responses during training. By selecting S learnable nodes and leveraging fast recursive convolution, we achieve an effective complexity of in time and memory. We further incorporate an efficient FFT-based computation of the relevance matrix and an adaptive node allocation mechanism to dynamically adjust the number of active Laplace nodes. Empirical results on language modeling (WikiText\-103, Project Gutenberg), machine translation (WMT'14 En\-De), and long document question answering (NarrativeQA) demonstrate that our learnable STLT achieves perplexities and scores on par with or better than existing efficient transformers while naturally extending to context lengths exceeding 100k tokens or more limited only by available hardware. Ablation studies confirm the importance of learnable parameters and adaptive node allocation. The proposed approach combines interpretability, through explicit decay and frequency parameters, with scalability and robustness, offering a pathway towards ultra-long-sequence language modeling without the computational bottleneck of self-attention.

Figures

Figures reproduced from arXiv: 2506.15714 by the authors.

Figure 1
Figure 1. STLT Transformer Architecture. Encoder: Given input token embeddings X ∈ R N×d , we add positional encodings P to form Xe = X + P. Each of the Le encoder layers then applies a two-sided short-time Laplace transform (STLT) block. The STLT block computes, for each position n and Laplace node k: Ln,k = PN m=1 xm w [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 23 canonical work pages

  1. [1]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8055–8060, 2020

    work page 2020

  2. [2]

    Findings of the 2014 workshop on statistical ma- chine translation

    Ondˇ rej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Jo- hannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Sori- cut, Lucia Specia, and Aleˇ s Tamchyna. Findings of the 2014 workshop on statistical ma- chine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pag...

    work page 2014

  3. [3]

    The Fourier Transform and Its Applications

    Ronald N Bracewell. The Fourier Transform and Its Applications . McGraw-Hill, 2000

    work page 2000

  4. [4]

    Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J

    Krzysztof M. Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Performer: Linear attention via random fourier features. In International Conference on Learning Representations (ICLR), 2021. 15

    work page 2021

  5. [5]

    Ten Lectures on Wavelets

    Ingrid Daubechies. Ten Lectures on Wavelets . CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM, 1992

    work page 1992

  6. [6]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In International Conference on Learning Representations (ICLR) , 2024

    work page 2024

  7. [7]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022

    work page 2022

  8. [8]

    HyMap: eliciting hypotheses in early-stage software startups using cognitive mapping

    Xu Guo et al. Wavelet transformer for long-sequence modeling. arXiv preprint arXiv:2102.09387, 2021

    work page Pith review arXiv 2021

Show all 24 references
  1. [9]

    Nicholas J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2008

    2008

  2. [10]

    Learnable multi-scale wavelet transformer: A novel alternative to self-attention

    Andrew Kiruluta, Priscilla Burity, and Samantha Williams. Learnable multi-scale wavelet transformer: A novel alternative to self-attention. arXiv:2504.03821, 2025

    2025 arXiv

  3. [11]

    The narrativeqa reading comprehension challenge

    Tom´ aˇ s Koˇ cisk´ y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G´ abor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. In Transactions of the Association for Computational Linguistics, volume 6, pages 317–328, 2018

    2018

  4. [12]

    Fnet: Mixing to- kens with fourier transforms

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing to- kens with fourier transforms. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), p...

    2022

  5. [13]

    C. Liu, F. Zhang, and Y. Wang. Wavelet-based convolutional neural networks for texture classification. Neural Computing and Applications, vol. 32, no. 9, 2020

    2020

  6. [14]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A con- tinuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017

    2017

  7. [15]

    A Wavelet Tour of Signal Processing

    Stephane Mallat. A Wavelet Tour of Signal Processing . Academic Press, 3 edition, 2009

    2009

  8. [16]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR) , 2017

    2017

  9. [17]

    A call for clarity in reporting bleu scores

    Aur´ elie N´ ev´ eol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Ver- spoor. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 186–191, Brussels, Belgium, Oct 2018. Associati...

    2018

  10. [18]

    Oppenheim, Ronald W

    Alan V. Oppenheim, Ronald W. Schafer, and John R. Buck. Discrete-Time Signal Pro- cessing. Prentice Hall, 2nd edition, 1999

    1999

  11. [19]

    The Fourier Integral and Its Applications

    Athanasios Papoulis. The Fourier Integral and Its Applications . McGraw-Hill, 1987

    1987

  12. [20]

    Simplified state space layers for sequence modeling

    Scott Smith, Albert Gu, Harsh Mehta, Li Kevin, and Christopher R´ e. Simplified state space layers for sequence modeling. In International Conference on Machine Learning (ICML) , pages 32038–32057, 2023. 16

    2023

  13. [21]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations (ICLR) , 2021

    2021

  14. [22]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS 2017) , pages 5998–6008, 2017

    2017

  15. [23]

    Linformer: Self-attention with linear complexity

    Sinxin Wang, Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, and Jingjing Liu. Linformer: Self-attention with linear complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings (EMNLP Findings) , pages 1047–1058, 2020

    2020

  16. [24]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020) ...

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.