Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By learning K bandpass filters in the graph Fourier domain of a dependency parse, the Graph Wavelet Transformer replaces quadratic self-attention with a linear-time spectral mixing operator, reporting 0.8 BLEU gain, 7% fewer parameters…

desk verdict The GWT paper is not ready for review: the directed-graph Laplacian eigendecomposition is invalid and the text mixes in another model's contributions. read the letter →

arxiv 2505.07862 v1 pith:5ZFT4JNG submitted 2025-05-09 cs.CL

Andrew Kiruluta , Eric Lundy , Priscilla Burity This is my paper · ORCID
classification cs.CL
keywords graphwavelettransformerspectralLaplacianlearnablebandpassfiltersself-attentionalternativemachinetranslationmulti-scaledecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing the quadratic dot-product self-attention of a Transformer with a bank of learnable bandpass filters applied in the eigenbasis of a graph Laplacian built from syntactic dependency parses. The central claim is that $K \ll N$ such filters, each a small neural network on the graph-frequency variable $\lambda$, can simultaneously capture local syntax and global semantics at linear time and memory cost. On WMT14 English–German translation, the resulting Graph Wavelet Transformer reports a 0.8 BLEU gain over a Graph Transformer baseline while using 7% fewer parameters and speeding inference by about 15%. The authors read these results as evidence that multi-scale spectral decomposition is an interpretable, efficient alternative to quadratic attention for graph-structured sequences.

What carries the argument

The load-bearing object is the normalized graph Laplacian $L = I - D^{-1/2} A D^{-1/2}$ of the dependency graph and its eigendecomposition $L = U \Lambda U^\top$, which supplies the graph Fourier basis $U$. The mechanism is functional calculus: a spectral filter $g$ acts as $g(L) = U g(\Lambda) U^\top$, so a bank of $K$ scalar functions $g_k$, implemented as small multilayer perceptrons on $\lambda$, carves the spectrum into interpretable frequency bands. Each band's node-domain signal is multiplied by a learned per-filter mixing vector $\alpha^{(k)}$ and summed. This replaces the pairwise dot products of attention with a weighted sum of spectral projections, which is what makes the claimed linear-time behavior and the low-frequency/high-frequency interpretation possible.

What would settle it

Take a small directed dependency graph with an arc from node $i$ to $j$ and no reverse arc, form $A$, and compute $L = I - D^{-1/2} A D^{-1/2}$; the entries $(i,j)$ and $(j,i)$ differ, so $L \neq L^\top$. Since $L$ is not symmetric, the claimed real orthonormal eigendecomposition $L = U \Lambda U^\top$ cannot exist, and the spectral filters as defined are not well-posed on that graph.

Watch

Extended reading notes

Core claim

The central discovery is that token mixing in a sequence model can be done by a multi-scale wavelet transform on the normalized graph Laplacian of a dependency parse, with no attention matrix at all. The model forms $L = I - D^{-1/2} A D^{-1/2}$, writes $L = U \Lambda U^\top$, and applies $K$ learned filters $g_k(\lambda)$ to the spectrum, producing filtered embeddings $\hat{X}^{(k)} = U g_k(\Lambda) U^\top X$. These subband signals are combined as $Y = \sum_{k=1}^K \hat{X}^{(k)} \mathrm{diag}(\alpha^{(k)})$ with learned mixing vectors $\alpha^{(k)}$, then passed through a standard feed-forward block. Because the filters touch only the diagonal of the spectrum, the operation can be truncated to the top $M$ eigenpairs or approximated by Chebyshev polynomials, reducing the nominal $O(N^2 d)$ cost to $O(MNd)$ or $O(|E|d)$. Empirically the model outperforms the Graph Transformer baseline by 0.8 BLEU, uses 60M versus 65M parameters, and raises throughput from 155 to 178 sentences per second.

Load-bearing premise

The paper defines the dependency graph as directed but builds a symmetric normalized Laplacian from it; the whole method, including the eigendecomposition and the linear-time truncation, presupposes that this Laplacian really is symmetric and positive semidefinite, which a directed adjacency matrix does not guarantee.

Editorial extensions

If this is right

  • Sequence models could drop the quadratic self-attention matrix for a compact filter bank, opening longer contexts within the same memory budget.
  • The ablation, where BLEU rises from 27.2 at a single filter to 28.1 at four filters, indicates the number of spectral bands is a direct capacity control for syntax versus semantics.
  • A 7% parameter cut and roughly 15% faster decoding make the architecture a candidate for on-device or real-time translation.
  • Because the module only needs a graph Laplacian, the same block can be dropped into any graph-structured Transformer layer beyond translation, such as semantic parsing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the directed-graph symmetry gap is closed by symmetrizing the adjacency, the same filter bank should transfer directly to other parse-graph tasks such as AMR-to-text and semantic parsing.
  • A length-scaling experiment beyond the tested 256 tokens, say 512, 1024, and 2048, would show whether the linear-time claim holds in practice or only at short-sequence settings.
  • Comparing GWT against a Graph Transformer that keeps the adjacency but uses identity filters would isolate how much of the BLEU gain comes from the spectral mechanism itself rather than from the graph structure alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Graph Wavelet Transformer (GWT), a sequence-to-sequence architecture that replaces quadratic self-attention with a learnable multi-scale spectral filter bank defined over a graph Laplacian derived from dependency parses. The authors define a directed dependency graph, form a normalized graph Laplacian, apply K learned filters in the graph Fourier domain, and mix the subband signals with per-dimension coefficients. They claim that this yields a linear-time mixing operator that improves BLEU by 0.8 over a Graph Transformer baseline on WMT14 En–De while reducing parameters and inference latency. The paper includes a theoretical formulation in Section 3, experiments in Section 4, and a discussion of limitations in Section 5.

Significance. If the construction were mathematically sound, the idea of replacing self-attention with learned spectral filters on linguistic graphs would be a potentially interesting contribution to efficient sequence modeling, with the reported gains of 0.8 BLEU and 15% speedup being practically meaningful. The paper also deserves credit for including an ablation over the number of filters and for comparing against FNet and Linformer baselines. However, as written, the central mathematical construction is undefined for the very graphs the paper specifies, and the complexity and wavelet claims are not empirically or theoretically substantiated. The manuscript also appears to conflate the proposed GWT with a different prior model (SDGM, reference [22]), which undermines the novelty presentation. The significance is therefore limited by the correctness and clarity gaps; the core idea may have merit, but the current form does not establish a valid method or a rigorous evaluation.

major comments (4)
  1. [Section 3, 'Graph Signal and Laplacian'] The paper defines G=(V,E) as a directed graph with adjacency A and then forms L = I - D^{-1/2} A D^{-1/2}, stating that 'As L is symmetric positive semidefinite, we can write L=UΛU^T' with orthonormal U. For a directed adjacency matrix, D^{-1/2} A D^{-1/2} is not symmetric in general, so L is not symmetric and need not have real eigenvalues or an orthonormal eigenbasis. Every subsequent formula - h(L)=U h(Λ) U^T, the filter bank outputs, and the mixing sum - relies on this eigendecomposition. No symmetrization step is stated anywhere, and Section 4 confirms that the edges are directed dependency arcs. This is an internal mathematical inconsistency that invalidates the core construction of the method.
  2. [Section 3, 'Multi-Scale Learnable Wavelet Bank'] The filters g_k(λ) are implemented by MLPs that take a scalar λ and output a nonnegative weight, with no bandpass constraint, no partition-of-unity property, and no localization requirement. Calling these functions 'bandpass filters' or 'wavelet filters' is therefore unjustified; an arbitrary nonnegative MLP applied in the spectral domain does not define a wavelet transform in any standard sense. This matters because the central claim of the paper is that GWT captures local syntactic and global semantic context through multi-scale spectral decomposition, and that interpretability claim rests entirely on the filters acting as bandpass filters.
  3. [Section 3, 'Spectral Mixing vs. Self-Attention'] The paper acknowledges the worst-case complexity O(KN^2d) and states that truncation to top M eigenpairs or Chebyshev polynomial approximations can reduce this to O(MNd) or O(|E|d). However, neither the value of M nor the Chebyshev degree is ever reported, and the experiments in Section 4 do not identify which approximation was actually used. The abstract and conclusion claim 'linear-time complexity' and 'linear-time mixing operator' without empirical or theoretical support, because the actual complexity of the implemented method is never stated. This is a load-bearing gap for the efficiency claim.
  4. [Section 1, 'Introduction' and reference [22]] The first three paragraphs of Section 1, including the enumerated contributions, describe a 'Spectral Dictionary Generative Model (SDGM)' with a complex-valued Fourier dictionary, a convolutional encoder, a GMM, and Penn Treebank/WikiText-2 experiments; reference [22] is titled 'Learnable Multi-scale Wavelet Transformer', not SDGM. The paper then abruptly pivots to the Graph Wavelet Transformer in a way that never clearly separates the prior SDGM contribution from the new GWT contribution. This conflates two different models and makes it impossible to assess what is genuinely new in this manuscript relative to the authors' prior work.
minor comments (5)
  1. [Abstract and keywords] There are several typographical errors: 'incursO(N 2)' should be 'incurs O(N^2)', and the keyword 'equence-to-sequence modeling' should be 'sequence-to-sequence modeling'.
  2. [Section 4, 'Experiments'] The sentence 'peak GPU memory at inference drops by 15' is incomplete; the intended value appears to be 'by 15%' based on the abstract.
  3. [Table 1] The table reports BLEU, parameters, and memory for GWT and baselines, but there is no standard Transformer (non-graph) baseline, and the statistical test behind 'p<0.01' is not described.
  4. [References] Several references contain incorrect author lists or titles: reference [7] (GFNet), reference [16] (AFNO), reference [38] (Nyströmformer), and reference [22] (title mismatch with the SDGM description). These errors should be corrected.
  5. [Section 3, 'Connection to Laplace-Domain Intuition'] The extended discussion of the classical Laplace transform in Section 3 is not used elsewhere in the paper and does not contribute to the method; it could be removed or substantially shortened.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported 0.8 BLEU gain is measured on held-out WMT14 data, and the Laplacian wavelet construction is a standard spectral filter bank rather than a quantity defined in terms of its own empirical output.

full rationale

The paper's central empirical claim is a measured outcome of training on WMT14 English-German, not a prediction derived from a fitted parameter renamed as a result: the mixing coefficients alpha^(k) and filter MLPs g_k are learned on training data, and BLEU, parameter count, and latency are reported from held-out evaluation and throughput measurement. The mathematical chain in Section 3 (L = I - D^{-1/2} A D^{-1/2}, L = U Lambda U^T, X^(k) = U g_k(Lambda) U^T X, Y = sum_k X^(k) diag(alpha^(k))) is an architectural definition; no step of that chain is defined in terms of the 0.8 BLEU improvement or the 15% speedup, so nothing reduces to its own input by construction. The only self-citation, [22], is used as motivational background (More recently, we merged these ideas into a sequence model...) and is not invoked as a theorem, uniqueness proof, or fitted input that forces the present result; it is therefore a novelty or overlap concern, not load-bearing circularity. The directed-graph Laplacian issue (A need not be symmetric, so an orthonormal eigendecomposition is not guaranteed) is an internal mathematical inconsistency that blocks the method as written, but it is a correctness problem rather than a circularity, because the reported outputs are not equal to the inputs by definition. Accordingly, no circular step meeting the evidence standard is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several unverified premises: the spectral decomposition of a directed graph Laplacian, the wavelet interpretation of unconstrained filter MLPs, and the top-M truncation whose parameter is unreported. There are no invented physical entities. The model's learned filter and mixing weights are fitted to WMT14 data, so the reported gains are not predictions from a parameter-free theory.

free parameters (3)
  • K (number of spectral filters) = 4 (ablation tested 1 and 2)
    Hand-chosen hyperparameter; the ablation shows translation quality and memory both depend on K, so it is not derived from first principles.
  • M (number of top eigenpairs for truncation) = not reported
    The linear-time and memory claims depend on truncating to the top M eigenpairs or using Chebyshev approximation, but M is never stated or varied.
  • Filter-MLP weights and mixing coefficients alpha^(k) = learned on WMT14
    These are the model's trainable parameters, fitted to data. They are the intended mechanism rather than ad hoc corrections, but the empirical gains are not predictions from a parameter-free theory.
assumptions (5)
  • standard math The normalized Laplacian L = I - D^(-1/2) A D^(-1/2) is symmetric positive semidefinite with eigenvalues in [0,2].
    Invoked in Section 3 to justify L = U Lambda U^T. The statement is true only if A is symmetric; the paper defines a directed graph, so the premise is not met as stated.
  • domain assumption Graph Laplacian eigenvectors form a meaningful Fourier basis for token embeddings.
    Spectral graph methods assume smoothness over graph edges corresponds to useful linguistic structure. The paper offers no theoretical or empirical support beyond analogy.
  • ad hoc to paper The MLP-implemented filters g_k are bandpass wavelet filters.
    Section 3 says each g_k is a nonnegative MLP output and calls it a bandpass filter, but no bandpass constraints, scale parameters, or wavelet admissibility conditions are defined.
  • ad hoc to paper Truncating to top M eigenpairs or using Chebyshev approximation preserves translation quality.
    The linear-complexity claim in Section 3 depends on this. No M value, approximation degree, error bound, or ablation isolating truncation error is reported.
  • domain assumption Dependency parse edges provide a graph structure that improves or preserves NMT quality.
    Section 4 builds G from CoreNLP dependency arcs without demonstrating that parse structure is a reliable inductive bias for translation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition." pith.science (2026). https://pith.science/paper/5ZFT4JNG

@misc{pith2026250507862,
  author       = {Pith},
  title        = {Pith review of: Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZFT4JNG}},
  note         = {Machine review of arXiv:2505.07862}
}
read the original abstract

Existing sequence to sequence models for structured language tasks rely heavily on the dot product self attention mechanism, which incurs quadratic complexity in both computation and memory for input length N. We introduce the Graph Wavelet Transformer (GWT), a novel architecture that replaces this bottleneck with a learnable, multi scale wavelet transform defined over an explicit graph Laplacian derived from syntactic or semantic parses. Our analysis shows that multi scale spectral decomposition offers an interpretable, efficient, and expressive alternative to quadratic self attention for graph structured sequence modeling.

Figures

Figures reproduced from arXiv: 2505.07862 by the authors.

Figure 1
Figure 1. Graph Wavelet Transformer Architecture. Starting from input embeddings X ∈ R N×d (top left), we first build a directed graph G = (V, E) from a syntactic or semantic parse. From G we compute the normalized graph Laplacian L = I − D−1/2A D−1/2 , whose eigen￾decomposition L = U ΛU ⊤, Λ = diag(λ1, . . . , λN ) yields the graph Fourier basis U. In the multi-scale wavelet block (shaded), K learnable bandpass filters gk(λ)… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Pixels and Words to Waves: A Unified Framework for Spectral Dictionary vLLMs

    cs.CV 2025-06 reject novelty 4.0 of 10

    A spectral-dictionary VLM claims to match mid-scale baselines without convolutions or self-attention, but its O(L log L) complexity argument is not supported by the equations as written.

Reference graph

Works this paper leans on

40 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [22]

    Learnable multi-scale wavelet transformer: A novel alternative to self-attention

    Andrew Kiruluta, Priscilla Burity, and Samantha Williams. Learnable multi-scale wavelet transformer: A novel alternative to self-attention. arXiv:2504.03821, 2025

    arXiv 2025

  2. [1]

    K-svd: An algorithm for designing overcomplete dictionaries for sparse representation.IEEE Transactions on Signal Process- ing, 54(11):4311–4322, 2006

    Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation.IEEE Transactions on Signal Process- ing, 54(11):4311–4322, 2006

    work page 2006

  3. [2]

    Allen and Lawrence R

    James B. Allen and Lawrence R. Rabiner. Short time spectral analysis, synthesis, and modification by discrete fourier transform.IEEE Transactions on Acoustics, Speech, and Signal Processing, 25(3):235–238, 1977

    work page 1977

  4. [3]

    Graph convolutional encoders for syntax-aware neural machine translation

    Jasmijn Bastings, Wilker Aziz, Trevor Cohn, Lucy Martin, Karin Verspoor, and Phil Blun- som. Graph convolutional encoders for syntax-aware neural machine translation. In EMNLP, pages 1957–1967, 2017

    work page 1957

  5. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document trans- former. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2010–2022, 2020

    work page 2020

  6. [5]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  7. [6]

    Spectral networks and locally connected networks on graphs

    Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. InarXiv preprint arXiv:1312.6203, 2013. 8

    arXiv 2013

  8. [7]

    Gfnet: Global filter networks for image classification.arXiv preprint arXiv:2107.00645, 2021

    Hengrui Chi, Wenhai Huang, Xinggang Wang, Song Bai, Jing Liu, Jianhuang Shi, and Liu Qi. Gfnet: Global filter networks for image classification.arXiv preprint arXiv:2107.00645, 2021

    arXiv 2021

Show all 40 references
  1. [8]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. InAdvances in Neural Information Processing Systems, vol- ume 32, pages 1179–1188, 2019

    2019

  2. [9]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. InarXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  3. [10]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Luke Hawkins, Jakub Davis, Sanjiv Mohiuddin, Łukasz Kaiser, David Belanger, and Ilya Sutskever. Rethinking attention with performers. InInternational Con- ference on Learning R...

    2021

  4. [11]

    Coifman and Mauro Maggioni

    Ronald R. Coifman and Mauro Maggioni. Diffusion wavelets.Applied and Computational Harmonic Analysis, 21(1):53–94, 2006

    2006

  5. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL–HLT, pages 4171–4186, 2019

    2019

  6. [13]

    Generalization of transformer networks to graphs

    Vijay Prakash Dwivedi and Xavier Bresson. Generalization of transformer networks to graphs. InICML, pages 2081–2090, 2020

    2020

  7. [14]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.arXiv preprint arXiv:2101.03961, 2021

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.arXiv preprint arXiv:2101.03961, 2021

    2021 arXiv

  8. [15]

    Griffin and Jae S

    Daniel W. Griffin and Jae S. Lim. Signal estimation from modified short-time fourier transform.IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(2):236– 243, 1984

    1984

  9. [16]

    Afno: Adaptive fourier neural operator for long-range sequence modeling

    Zongyi Guo, Zongyi Li, Zheng Sun, Pratik Bhattacharya, and Weinan E. Afno: Adaptive fourier neural operator for long-range sequence modeling. InNeurIPS, 2022

    2022

  10. [17]

    Hammond, Pierre Vandergheynst, and Rémi Gribonval

    David K. Hammond, Pierre Vandergheynst, and Rémi Gribonval. Wavelets on graphs via spectral graph theory.Applied and Computational Harmonic Analysis, 30(2):129–150, 2011

    2011

  11. [18]

    Statistical methods for speech recognition.MIT Press, 1977

    Frederick Jelinek. Statistical methods for speech recognition.MIT Press, 1977

    1977

  12. [19]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Panos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InProceedings of the International Conference on Machine Learning, pages 5156–5165, 2020

    2020

  13. [20]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 [cs.LG], 2014. 9

    2014 arXiv

  14. [21]

    Kipf and Max Welling

    Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. InICLR, 2017

    2017

  15. [23]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. InProceedings of the International Conference on Learning Representations, 2020

    2020

  16. [24]

    Fnet: Mixing to- kens with fourier transforms

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing to- kens with fourier transforms. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pa...

    2022

  17. [25]

    Academic Press, 1999

    Stephane Mallat.A Wavelet Tour of Signal Processing. Academic Press, 1999

    1999

  18. [26]

    Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky

    Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. InACL System Demonstrations, pages 55–60, 2014

    2014

  19. [27]

    Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini

    Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. InComputational Linguistics, volume 19, pages 313–330, 1993

    1993

  20. [28]

    Regularizing and optimizing lstm language models.arXiv preprint arXiv:1708.02182, 2017

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Regularizing and optimizing lstm language models.arXiv preprint arXiv:1708.02182, 2017

    2017 arXiv

  21. [29]

    A call for clarity in reporting bleu scores

    Matt Post. A call for clarity in reporting bleu scores. InProceedings of the Third Confer- ence on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, Oct

  22. [30]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018

    2018

  23. [31]

    Language models are unsupervised multitask learners.OpenAI Blog, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI Blog, 2019. https://openai.com/blog/better-language-models

    2019

  24. [32]

    Reynolds

    Douglas A. Reynolds. Gaussian mixture models.Encyclopedia of Biometrics, 2009

    2009

  25. [33]

    Hybrid fourier-transformer for efficient sequence modeling

    Minghao Su, Xinyun Zhang, Jia Zhou, and Zhiyuan Liu. Hybrid fourier-transformer for efficient sequence modeling. InACL, 2021

    2021

  26. [34]

    Efficient transformers: A survey.arXiv preprint arXiv:2009.06732, 2020

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey.arXiv preprint arXiv:2009.06732, 2020

    2009 arXiv

  27. [35]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, pages 5998–6008, 2017. 10

    2017

  28. [36]

    Graph attention networks

    Petar Veli ˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. InICLR, 2018

    2018

  29. [37]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity. arXiv:2006.04768 [cs.CL], 2020

    2006 arXiv

  30. [38]

    Le, and Yuxin Lu

    Zihang Xiong, Zihang Dai, Qingyan Hager, Soham Ramteke, Fady Khaled, Mike Johnson, Quoc V . Le, and Yuxin Lu. Nyströmformer: A nyström-based algorithm for approximating self-attention. InICML, 2021

    2021

  31. [39]

    Big bird: Trans- formers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Karan Dubey, Joshua Ainslie, Chris Alberti, Saurabh Joshi, Tristan Pham, Kanad Ravula, Shaowei Wang, Li Yang, and Others. Big bird: Trans- formers for longer sequences. InAdvances in Neural Information Processing Systems, volume 33, pages 17283–...

    2020

  32. [2018]

    Association for Computational Linguistics

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.