Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

Hyowon Wi; Jeongwhan Choi; Noseong Park

REVIEW 4 major objections 5 minor 55 references

Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-attention is a low-pass graph filter; AGF replaces it with a learned singular-value-domain filter that passes high frequencies and runs in linear time.

desk verdict A plausible linear-attention variant with honest benchmark gains, but the theoretical story has two load-bearing errors and needs a rewrite. read the letter →

arxiv 2505.08516 v1 pith:X5DHSEHX submitted 2025-05-13 cs.LG cs.AI

Hyowon Wi , Jeongwhan Choi , Noseong Park This is my paper · ORCID

classification cs.LGcs.AI

keywords attentivegraphfilterlineartransformersingularvaluedomainsignalprocessinglow-passJacobipolynomiallongrangearenatimeseriesclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-attention in a Transformer can be read as a graph filter: multiplying value vectors by a softmax-normalized attention matrix is a first-order, low-pass operation that smooths tokens and suppresses high-frequency information. The paper proposes AGF (Attentive Graph Filter), a linear-complexity attention layer that instead learns the filtering directly in the singular value domain: small networks generate singular vectors and singular values from the input, the singular values are shaped by a Jacobi-polynomial filter with learnable coefficients, and the result multiplies the value matrix with cost $\mathcal{O}(nd^2)$ in sequence length $n$ and hidden dimension $d$. The paper claims this filter can pass both low and high frequencies, and reports state-of-the-art average accuracy of 75.1 on UEA time-series classification and 60.1 on the Long Range Arena benchmark, beating the vanilla Transformer and prior linear methods. If correct, AGF is a drop-in replacement for softmax attention that avoids quadratic cost without giving up frequency information.

What carries the argument

The object doing the work is the learned singular-value-domain factorization $\mathbf{H}(\mathbf{X}) = \mathbf{U}(\mathbf{X}) \Sigma(\mathbf{X}) \mathbf{V}(\mathbf{X})^\top$, where $\mathbf{U}$ and $\mathbf{V}^\top$ are softmax-normalized linear projections of the input and $\Sigma$ is a per-token singular-value matrix filtered through a truncated Jacobi polynomial expansion. The Jacobi basis is an orthogonal polynomial family on $[-1,1]$, so the learnable coefficients $\theta_k$ give a stable filter instead of an ill-conditioned monomial expansion. The argument uses this factorization to avoid ever forming the $n \times n$ attention map: the multiplication with the value matrix is reordered to contract the $d \times d$ part first, which is what yields $\mathcal{O}(nd^2)$ complexity and makes the layer a practical linear-attention replacement.

What would settle it

A direct frequency-response measurement would settle the central claim: take a trained AGF layer, feed value signals that are pure sinusoids at known frequencies, and compare output amplitudes; the paper's story predicts that learned $\theta_k$ produce bands that amplify high frequencies. The paper's own $\mathbf{H}_{UV^\top}$ ablation, which fixes $\Sigma$ to the identity, already drops UEA average accuracy from 75.1 to 71.7, showing that the filter term itself carries the effect.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a linear transformer does not have to approximate the softmax attention map; it can instead learn a graph filter. The paper proves that the standard softmax attention matrix is a low-pass filter (Theorem 1) because repeated multiplication by the row-stochastic matrix drives signals to the dominant eigenvector. It then constructs AGF, which parameterizes $\mathbf{U}(\mathbf{X}) = \mathrm{softmax}(\mathbf{X}\mathbf{W}_U)$, singular values $\Sigma(\mathbf{X}) = \sum_{k=0}^K \theta_k T_k(\mathrm{diag}(\sigma(\mathbf{X}\mathbf{W}_\Sigma)))$ with $T_k$ a Jacobi polynomial, and $\mathbf{V}(\mathbf{X})^\top = \mathrm{softmax}((\mathbf{X}\mathbf{W}_V)^\top)$, and applies the filter as $\mathbf{H} = \mathbf{U} \Sigma \mathbf{V}^\top$ to the value matrix. Because the coefficients $\theta_k$ can be negative, the filter can emphasize high-frequency components, and because the singular values differ per token, each token gets its own frequency response. Empirically AGF reports average accuracy 75.1 on the ten UEA datasets versus 71.9 for the vanilla Transformer, 60.1 on LRA versus 58.8, marginal ImageNet gains when plugged into DeiT-small, and reduced over-smoothing: token cosine similarity stays near 0.5 instead of rising to about 0.9 in a 12-layer vision transformer.

Load-bearing premise

The load-bearing premise is that the learned matrices $\mathbf{U}$, $\Sigma$, and $\mathbf{V}$ behave as a genuine singular-value decomposition with $\Sigma$ diagonal and $\mathbf{U}$, $\mathbf{V}$ orthogonal, so that the proven low- and high-pass behavior of spectral graph filters transfers to AGF; the orthogonality is only softly regularized, and the regularizer as written mismatches dimensions.

Editorial extensions

If this is right

AGF can be dropped into existing Transformer layers in place of softmax attention with the same input and output shapes, changing only the attention computation and training objective.
Because it learns coefficients that can go negative, AGF need not behave as a low-pass filter; it can preserve high-frequency components of value vectors, which the paper argues is why it outperforms linear attention methods that merely approximate softmax attention.
The token-specific singular values mean different tokens can filter their own frequency content, an ability the paper ties to better handling of complex dynamics in hidden representations.
In deep Transformers, the paper reports that AGF mitigates over-smoothing: token embeddings remain more distinguishable across layers than with vanilla attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to apply AGF to tasks where sharp local features matter, such as anomaly detection, boundary detection, or punctuation-heavy text, where preserving high frequencies should matter more than in the benchmarks reported.
The complexity claim hides a $d^2$ factor: AGF is most attractive for long sequences with moderate hidden dimension, and for very high-dimensional embeddings the $d \times d$ contraction could dominate; the paper does not report this crossover.
Because the paper's SVD interpretation is only softly enforced, a direct diagnostic would be to measure $\|\mathbf{U}^\top\mathbf{U}-\mathbf{I}\|$ and $\|\mathbf{V}\mathbf{V}^\top-\mathbf{I}\|$ on trained models; if the learned factors are far from orthogonal, the benefits may come from the parameterization rather than the spectral story.
The graph-filter framing suggests variants not explored here: other orthogonal bases or explicit band-pass coefficient designs, or applying AGF inside state-space sequence models, which the paper names as future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about arXiv:2505.08516. First, the empirical claim looks real: AGF, a learnable-SVD linear attention with a Jacobi polynomial filter, gets 60.1 on LRA (vs 59.0 for Nyströmformer) and 75.1 on UEA, with consistent margins across datasets and sensible ablations. Second, the theory in the paper does not hold as written: Theorem 1 is about the limit of repeated application M^t, whereas self-attention applies the filter once per layer; Theorem 2's proof imports undirected spectral facts (λ1=1, |λi|<1) that do not apply to singular values of directed row-stochastic matrices; and Eq. (11) compares V(X)V(X)^T, which is n×n under their own definitions, to the d×d identity. These are not cosmetic. The paper's central motivation — that AGF leverages high-frequency information because sign-changing coefficients yield high-pass behavior — depends on Theorem 2, and the proof as given does not establish it.

What is genuinely new: the specific combination of parameterized SVD with a learnable polynomial filter in the singular value domain. Singularformer uses parameterized SVD but no graph filter; the authors' prior NeurIPS paper has the graph-filter view of attention but not the singular-value parameterization. The empirical work is the real contribution: benchmark coverage includes UEA, LRA, and ImageNet, with runtime and memory numbers, ablations on polynomial order, Jacobi parameters, and activation, plus a Wilcoxon signed-rank test against the UEA baselines. That is more than most linear-attention submissions do.

The soft spots, in order of severity. First, the theoretical scaffold needs repair or removal. Theorem 1 can be fixed by stating it as a property of the attention operator's stationary behavior, but as written it does not prove that one application of attention is low-pass. Theorem 2's proof applies Von Luxburg's undirected spectral facts to singular values of a directed matrix; a simple counterexample — [[0.5,0.5],[0.9,0.1]] has largest singular value about 1.089 — shows the premise fails. The paper itself requires ||Ā||≤1 to invoke Maskey et al., and that condition is never verified. Second, Eq. (11) has a dimension mismatch: with V(X)^T in R^{d×n}, V(X) in R^{n×d}, V V^T is n×n, but the identity is I_d; the regularization should be on V^T V or the shapes corrected. This is minor to fix but needs stating. Third, no code is released, which matters for a method whose value is mainly empirical. The paper acknowledges the state-space model comparison gap in its conclusion; that is a fair limitation but not the main issue.

Who is this for? People working on efficient attention who want a concrete alternative to Nyströmformer or Primalformer that reads as a drop-in replacement. The paper deserves a serious referee — the empirical claims are extensive and reproducible in principle — but it should come back with the theory either fixed or explicitly downgraded to intuition. I would accept it as a conditional, not reject it.

Referee Report

4 major / 5 minor

Summary. The paper proposes Attentive Graph Filter (AGF), a linear-complexity self-attention layer that abandons the explicit softmax attention map and instead generates, via neural networks, matrices U(X), Sigma(X), and V(X) intended to play the roles of singular vectors and singular values. The singular values are modulated by a learnable Jacobi-polynomial graph filter, and an orthogonality regularizer is added to encourage U and V to behave like SVD factors. The authors motivate the method by two theoretical claims: that softmax self-attention is a low-pass graph filter (Theorem 1), and that a polynomial filter with sign-changing coefficients can pass high-frequency information in the singular-value domain (Theorem 2). The empirical sections report results on UEA time-series classification, Long Range Arena, ImageNet classification with DeiT-small, ablation studies, runtime/memory comparisons, and an analysis of over-smoothing in deep transformers.

Significance. If the method and its motivation were sound, AGF would be a useful drop-in linear attention layer, and the paper would contribute a new GSP-based design space for efficient transformers. The empirical evaluation is a genuine strength: it covers multiple benchmarks (UEA, LRA, ImageNet), includes ablations on the polynomial type, activation function, filter variant, hyperparameters, and report runtime/memory and Wilcoxon signed-rank tests. The paper states a reproducibility appendix, although no code is provided. However, the two theoretical pillars of the paper have concrete errors, and one equation is dimensionally invalid; the central 'low/high-frequency' motivation is therefore not established as written. The empirical results may still stand, but the supporting theory needs substantial correction or reframing.

major comments (4)

[Section 3.1, Theorem 1 (Appendix D)] Theorem 1 states that softmax self-attention acts as a low-pass filter, but the statement and proof concern lim_{t->infinity} M^t x, i.e., the repeated application of the row-stochastic matrix M. In a Transformer layer, the graph filter H = \bar A is applied once to the value signal (Eqs. (1) and (3)), not infinitely many times. The asymptotic dominance of the Perron eigenvalue under repeated application does not imply that a single application attenuates high-frequency components relative to low-frequency components; for a single application the HFC/LFC ratio can behave differently. This theorem is the stated motivation for the entire method, so the claim that vanilla self-attention suppresses high-frequency information is not supported.
[Section 3.6, Theorem 2 (Appendix E, Eq. (17))] The proof of Theorem 2 assumes that the singular values of the attention matrix satisfy \lambda_1 = 1 and |\lambda_i| < 1 for i >= 2, citing Von Luxburg's undirected spectral-clustering results. These facts hold for eigenvalues of a row-stochastic matrix (spectral radius 1, all eigenvalues in the unit disk), but they do not hold for singular values of a non-symmetric row-stochastic matrix. For example, the row-stochastic matrix [[0.5, 0.5], [0.9, 0.1]] has largest singular value approximately 1.09. Softmax attention matrices are generally non-symmetric, and Section 3.2 itself requires ||\bar A|| <= 1 for the Maskey et al. SVD framework, a condition that is neither guaranteed for softmax attention nor verified in the paper. Consequently, the low-pass and high-pass conclusions in Theorem 2, including the ratio |(1+\alpha)/(1+\alpha \lambda_i)| > 1 in Eq. (21), are not justified for the singular values of a directed attention matrix. The paper's central claim that AGF 'incorporates both low and high-frequency information' lacks theoretical support.
[Section 3.4, Eq. (11)] The orthogonality regularizer is dimensionally invalid. With V(X) in R^{n x d} as defined in Eq. (10), the product V(X) V(X)^T is an n x n matrix, but Eq. (11) compares it to I in R^{d x d}. The analogous term should be V(X)^T V(X) to yield a d x d matrix, or the shapes must be corrected consistently. As written, the regularizer does not enforce the intended row-orthogonality of V and breaks the SVD interpretation that U, Sigma, and V are factors of a single attention matrix.
[Appendix H.1/H.2 vs Tables 10/11] The reported hyperparameter search ranges do not cover the hyperparameters listed as best. Appendix H.1 states that K is grid-searched over {2, 3, 4, 5} for UEA, but Table 10 reports best values K=6 (EthanolConcentration), K=6 (Heartbeat), K=9 (FaceDetection, SelfRegulationSCP1), and K=10 (UWaveGestureLibrary). Similarly, Appendix H.2 states K in {2, 3, 4, 5} for LRA, but Table 11 reports K=7 for Image. This inconsistency makes it impossible to reproduce the reported results from the stated protocol and needs to be corrected or explained.

minor comments (5)

[Theorem 1 statement] The theorem uses both N and n for the ambient dimension ('For all x in R^N' vs. M in R^{n x n}), and the ratio ||HFC[M^t(x)]||_2 / ||LFC[M^t(x)]||_2 is written with the limit placed slightly ambiguously; the definitions of LFC and HFC should be restated in the appendix to match the statement.
[Appendix L.2] The text says the polynomial-type ablation results are shown in Table 18, but Table 18 displays the activation-function ablation; the polynomial-type results appear in Table 19. The cross-reference is wrong.
[Appendix E] There are repeated typos in the proof, including 'hgih-pass' instead of 'high-pass' and 'grah filter' instead of 'graph filter'; these should be corrected.
[Figure 3] The manuscript does not describe how the filter response in Figure 3(a) is computed from the DeiT and DeiT+AGF models; a precise definition (which matrix or operator is analyzed, and over which inputs) is needed for the over-smoothing claim to be verifiable.
[Reproducibility] The reproducibility statement in Appendix A promises dataset descriptions and best hyperparameters, but no code release or seed information is mentioned; providing code or at least detailed training configurations would strengthen the empirical contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: AGF is an independently defined architecture evaluated on external benchmarks; theoretical motivation has correctness gaps but is not circular.

full rationale

The paper's central contribution is the AGF layer defined by Eqs. (7)-(10), which learns matrices U(X), Sigma(X), and V(X) from neural networks with softmax and sigmoid nonlinearities. These definitions do not presuppose the empirical results, and the empirical claims are tested on external held-out benchmarks (UEA, LRA, ImageNet) against standard baselines. The theoretical motivation is that softmax attention acts as a low-pass filter (Theorem 1) and that sign-changing polynomial coefficients can produce high-pass filtering (Theorem 2). Theorem 1 is proved via Perron-Frobenius/Jordan form reasoning and is not assumed as an input; Theorem 2 is explicitly adapted from an external reference (Chien et al. 2021) and proved in the appendix. The self-citations to Choi et al. 2024 and Kim et al. 2024 are used only for contextual claims about interpreting attention as a normalized adjacency matrix or polynomial self-attention; they are not load-bearing in the derivation of AGF's equations or in the empirical evaluation. The paper contains mathematical correctness concerns, such as applying undirected spectral facts (lambda_1 = 1 and |lambda_i| < 1) to singular values of directed row-stochastic matrices and the dimensional mismatch in Eq. (11) where VV^T is n x n while compared to I_d. However, these are soundness issues, not circularity: they do not make the predictions equivalent to the inputs by construction, nor do they rename a fitted parameter as a prediction. Therefore, no circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the graph-filter view of self-attention, the validity of SVD-based GSP for directed graphs, the chosen polynomial basis, and architectural hyperparameters. No new physical entities are introduced. The SVD interpretation is weakened by inconsistent orthogonality terms and by proof gaps in the low/high-pass theorems.

free parameters (3)

Polynomial coefficients theta_k = learned
Coefficients of the Jacobi expansion in Eq. (9) that determine the passband; learned during training and central to the claim that AGF can pass high frequencies.
Singular value/vector projections W_U, W_Sigma, W_V = learned
Linear maps in Eqs. (8)-(10) that generate U, Sigma, V from input X; they encode the attention-like behavior and are learned from data.
Jacobi parameters a, b, polynomial order K, orthogonality weight gamma = per dataset (Tables 10-12)
Chosen by grid search on each dataset, which makes the reported comparisons favorable but is not itself a fitted value of the model.

assumptions (4)

domain assumption Self-attention with softmax is a row-stochastic matrix and can be treated as a normalized adjacency matrix (graph shift operator) on a directed graph.
Section 3.1 and Fig. 1; this identification is the basis for applying GSP tools to attention.
domain assumption For a symmetrically normalized adjacency of a directed graph, SVD provides a valid spectral domain with non-negative singular values within the unit circle.
Section 3.2, relying on Maskey et al. 2023; the paper does not verify these conditions for its learned U, Sigma, V.
standard math Jacobi polynomials form an orthogonal basis on [-1,1] and stabilize the graph filter expansion.
Section 3.2 and Appendix F; standard result, but the singular values are in [0,1] and the polynomial arguments need not cover [-1,1].
ad hoc to paper The low-pass property of softmax attention under repeated application (Theorem 1) justifies the claim that vanilla attention suppresses high-frequency information.
Appendix D; the proof is for M^t as t goes to infinity, not for the single application used in transformers, so this premise is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain." pith.science (2026). https://pith.science/paper/X5DHSEHX

@misc{pith2026250508516,
  author       = {Pith},
  title        = {Pith review of: Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X5DHSEHX}},
  note         = {Machine review of arXiv:2505.08516}
}

abstract

Transformers have demonstrated remarkable performance across diverse domains. The key component of Transformers is self-attention, which learns the relationship between any two tokens in the input sequence. Recent studies have revealed that the self-attention can be understood as a normalized adjacency matrix of a graph. Notably, from the perspective of graph signal processing (GSP), the self-attention can be equivalently defined as a simple graph filter, applying GSP using the value vector as the signal. However, the self-attention is a graph filter defined with only the first order of the polynomial matrix, and acts as a low-pass filter preventing the effective leverage of various frequency information. Consequently, existing self-attention mechanisms are designed in a rather simplified manner. Therefore, we propose a novel method, called \underline{\textbf{A}}ttentive \underline{\textbf{G}}raph \underline{\textbf{F}}ilter (AGF), interpreting the self-attention as learning the graph filter in the singular value domain from the perspective of graph signal processing for directed graphs with the linear complexity w.r.t. the input length $n$, i.e., $\mathcal{O}(nd^2)$. In our experiments, we demonstrate that AGF achieves state-of-the-art performance on various tasks, including Long Range Arena benchmark and time series classification.

Figures

Figures reproduced from arXiv: 2505.08516 by the authors.

**Figure 2.** The proposed AGF performs the directed GSP in the singular value domain by learning [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Filter response and cosine similarity on ImageNet-1k for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 37 canonical work pages

[1]

[Askey and Wilson, 1985] Richard Askey and James Arthur Wilson.Some basic hypergeometric orthogonal polyno- mials that generalize Jacobi polynomials, volume

work page 1985
[9]

Convolutional neural net- works on graphs with fast localized spectral filtering

[Defferrardet al., 2016 ] Micha¨el Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural net- works on graphs with fast localized spectral filtering. In NeurIPS,

work page 2016
[12]

Mamba: Linear- time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

[Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

arXiv 2023
[13]

Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

[Guet al., 2021 ] Albert Gu, Karan Goel, and Christopher R´e. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

arXiv 2021
[14]

Transformers are rnns: Fast autoregressive transformers with linear attention

[Katharopouloset al., 2020 ] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InICML. PMLR,

work page 2020
[15]

Polynomial-based self-attention for table representation learning

[Kimet al., 2024 ] Jayoung Kim, Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. Polynomial-based self-attention for table representation learning. InInter- national Conference on Machine Learning, pages 24509– 24526. PMLR,

work page 2024
[16]

Kipf and Max Welling

[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. InICLR,

work page 2017
[18]

Learning multiple layers of features from tiny im- ages

[Krizhevskyet al., 2009 ] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages

2009

Show all 55 references

[19]

En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting.NeurIPS,

[Liet al., 2019 ] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. En- hancing the locality and breaking the memory bottleneck of transformer on time series forecasting.NeurIPS,

2019
[20]

Learn- ing long-range spatial dependencies with horizontal gated recurrent units.NeurIPS, 31,

[Linsleyet al., 2018 ] Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learn- ing long-range spatial dependencies with horizontal gated recurrent units.NeurIPS, 31,

2018
[21]

Swin transformer: Hierarchical vision transformer using shifted windows

[Liuet al., 2021 ] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InICCV,

2021
[22]

Soft: Softmax-free transformer with linear complexity.NeurIPS, 34,

[Luet al., 2021 ] Jiachen Lu, Jinghan Yao, Junge Zhang, Xi- atian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xi- ang, and Li Zhang. Soft: Softmax-free transformer with linear complexity.NeurIPS, 34,

2021
[23]

Learning word vectors for sentiment analysis

[Maaset al., 2011 ] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InProceed- ings of the annual meeting of the association for computa- tional linguistics: Human language technologies,

2011
[25]

A fractional graph laplacian approach to oversmoothing

[Maskeyet al., 2023 ] Sohir Maskey, Raffaele Paolino, Aras Bacho, and Gitta Kutyniok. A fractional graph laplacian approach to oversmoothing. InNeurIPS,

2023
[26]

Generalized legendre polynomials.Journal of mathematical analysis and applications, 177(2),

[McCarthyet al., 1993 ] PC McCarthy, JE Sayre, and BLR Shawyer. Generalized legendre polynomials.Journal of mathematical analysis and applications, 177(2),

1993
[28]

Graph neural networks exponentially lose expressive power for node classification

[Oono and Suzuki, 2020] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. InICLR,

2020
[29]

Graph signal processing: Overview, challenges, and ap- plications.IEEE, 106(5),

[Ortegaet al., 2018 ] Antonio Ortega, Pascal Frossard, Jelena Kovaˇcevi´c, Jos ´e MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and ap- plications.IEEE, 106(5),

2018
[30]

Spectformer: Frequency and attention is what you need in a vision transformer.arXiv preprint arXiv:2304.06446,

[Patroet al., 2023 ] Badri N Patro, Vinay P Namboodiri, and Vijay Srinivas Agneeswaran. Spectformer: Frequency and attention is what you need in a vision transformer.arXiv preprint arXiv:2304.06446,

2023 arXiv
[31]

cosformer: Rethinking soft- max in attention.arXiv preprint arXiv:2202.08791,

[Qinet al., 2022 ] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Ling- peng Kong, and Yiran Zhong. cosformer: Rethinking soft- max in attention.arXiv preprint arXiv:2202.08791,

2022 arXiv
[34]

Etc: Encoding long and structured inputs in transformers

[Ravulaet al., 2020 ] Anirudh Ravula, Chris Alberti, Joshua Ainslie, Li Yang, Philip Minh Pham, Qifan Wang, Santi- ago Ontanon, Sumit Kumar Sanghai, Vaclav Cvicek, and Zach Fisher. Etc: Encoding long and structured inputs in transformers. InEMNLP,

2020
[35]

Konstantin Rusch, Michael M

[Ruschet al., 2023 ] T. Konstantin Rusch, Michael M. Bron- stein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks.arXiv preprint arXiv: Arxiv- 2303.10993,

2023 arXiv
[36]

Imagenet large scale visual recognition challenge.IJCV, 115,

[Russakovskyet al., 2015 ] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.IJCV, 115,

2015
[38]

Revisiting over-smoothing in bert from the perspective of graph

[Shiet al., 2022 ] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. InICLR,

2022
[39]

Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006,

[Tayet al., 2020 ] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006,

2020 arXiv
[40]

Attention is all you need

[Vaswaniet al., 2017 ] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, volume 30,

2017
[41]

Graph Attention Networks

[Veliˇckovi´cet al., 2018 ] Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li `o, and Yoshua Bengio. Graph Attention Networks. InICLR,

2018
[42]

A tutorial on spectral clustering.Statistics and computing, 17,

[V on Luxburg, 2007] Ulrike V on Luxburg. A tutorial on spectral clustering.Statistics and computing, 17,

2007
[44]

Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice

[Wanget al., 2022 ] Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. InICLR,

2022
[45]

Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting

[Wuet al., 2021 ] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. NeurIPS, 34,

2021
[46]

Flowformer: Lineariz- ing transformers with conservation flows.arXiv preprint arXiv:2202.06258,

[Wuet al., 2022 ] Haixu Wu, Jialong Wu, Jiehui Xu, Jian- min Wang, and Mingsheng Long. Flowformer: Lineariz- ing transformers with conservation flows.arXiv preprint arXiv:2202.06258,

2022 arXiv
[47]

Singularformer: Learning to decompose self- attention to linearize the complexity of transformer

[Wuet al., 2023 ] Yifan Wu, Shichao Kan, Min Zeng, and Min Li. Singularformer: Learning to decompose self- attention to linearize the complexity of transformer. In IJCAI, pages 4433–4441,

2023
[48]

Nystr ¨omformer: A nystr ¨om-based algorithm for approximating self-attention

[Xionget al., 2021 ] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr ¨omformer: A nystr ¨om-based algorithm for approximating self-attention. InAAAI, volume 35, pages 14138–14148,

2021
[49]

Learning flexible body collision dynamics with hierarchical contact mesh transformer

[Yuet al., 2024 ] Youn-Yeol Yu, Jeongwhan Choi, Woojin Cho, Kookjin Lee, Nayong Kim, Kiseok Chang, ChangSe- ung Woo, Ilho Kim, SeokWoo Lee, Joon Young Yang, Sooyoung Yoon, and Noseong Park. Learning flexible body collision dynamics with hierarchical contact mesh transformer. InICLR,

2024
[50]

Big bird: Transformers for longer se- quences.NeurIPS, 33:17283–17297,

[Zaheeret al., 2020 ] Manzil Zaheer, Guru Guruganesh, Ku- mar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer se- quences.NeurIPS, 33:17283–17297,

2020
[51]

You only sample (almost) once: Linear cost self-attention via bernoulli sampling

[Zenget al., 2021 ] Zhanpeng Zeng, Yunyang Xiong, Sathya Ravi, Shailesh Acharya, Glenn M Fung, and Vikas Singh. You only sample (almost) once: Linear cost self-attention via bernoulli sampling. InICML. PMLR,

2021
[52]

A transformer-based framework for multivariate time series representation learning

[Zerveaset al., 2021 ] George Zerveas, Srideepika Jayara- man, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. InSIGKDD,

2021
[53]

Informer: Beyond efficient transformer for long se- quence time-series forecasting

[Zhouet al., 2021 ] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long se- quence time-series forecasting. InAAAI, volume 35,

2021
[54]

Importantly, this is in- dependent of the specific configurations of the input matrices Z

This behavior indicates a characteristic of a low-pass filter, reaffirming the low-pass nature ofM. Importantly, this is in- dependent of the specific configurations of the input matrices Z. E Proof of Theorem 2 Theorem 2.Assume the graphGis connected. Letλ 1 ě λ2 ě...ěλ n be ...

2007
[55]

• EthanolConcentration: A dataset of raw spectra of water-and-ethanol solu- tions in 44 distinct, real whisky bottles. • FaceDetection: A dataset consisting of MEG recordings and class labels (Face/Scramble) • HandWriting: A motion dataset taken from a smart- watch while the s...

2020
[1985]

The uea multivari- ate time series classification archive, 2018.arXiv preprint arXiv:1811.00075,

[Bagnallet al., 2018 ] Anthony Bagnall, Hoang Anh Dau, Ja- son Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivari- ate time series classification archive, 2018.arXiv preprint arXiv:1811.00075,

2018 arXiv
[1993]

Listops: A diagnostic dataset for latent tree learning.arXiv preprint arXiv:1804.06028,

[Nangia and Bowman, 2018] Nikita Nangia and Samuel R Bowman. Listops: A diagnostic dataset for latent tree learning.arXiv preprint arXiv:1804.06028,

2018 arXiv
[2007]

Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768,

[Wanget al., 2020 ] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768,

2020 arXiv
[2009]

BERT: Pre-training of deep bidirectional transformers for language understand- ing

[Devlinet al., 2019 ] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational...

2019
[2011]

Signal processing on directed graphs: The role of edge directionality when processing and learning from network data.IEEE Signal Processing Magazine, 37(6),

[Marqueset al., 2020 ] Antonio G Marques, Santiago Segarra, and Gonzalo Mateos. Signal processing on directed graphs: The role of edge directionality when processing and learning from network data.IEEE Signal Processing Magazine, 37(6),

2020
[2013]

Language models are unsupervised multitask learners

[Radfordet al., 2019 ] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9,

2019
[2015]

Efficient attention: Attention with linear complexities

[Shenet al., 2021 ] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. InICCV,

2021
[2016]

Imagenet: A large-scale hierarchical image database

[Denget al., 2009 ] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition,

2009
[2017]

Reformer: The efficient transformer

[Kitaevet al., 2020 ] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

2020 arXiv
[2018]

Longformer: The long-document trans- former.arXiv preprint arXiv:2004.05150,

[Beltagyet al., 2020 ] Iz Beltagy, Matthew E Peters, and Ar- man Cohan. Longformer: The long-document trans- former.arXiv preprint arXiv:2004.05150,

2020 arXiv
[2019]

Graph convolutions enrich the self- attention in transformers!Advances in Neural Information Processing Systems, 37:52891–52936,

[Choiet al., 2024 ] Jeongwhan Choi, Hyowon Wi, Jayoung Kim, Yehjin Shin, Kookjin Lee, Nathaniel Trask, and Noseong Park. Graph convolutions enrich the self- attention in transformers!Advances in Neural Information Processing Systems, 37:52891–52936,

2024
[2020]

Primal-attention: Self- attention through asymmetric kernel svd in primal repre- sentation.arXiv preprint arXiv:2305.19798,

[Chenet al., 2023 ] Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan AK Suykens. Primal-attention: Self- attention through asymmetric kernel svd in primal repre- sentation.arXiv preprint arXiv:2305.19798,

2023 arXiv
[2021]

Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

[Childet al., 2019 ] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

2019 arXiv
[2022]

The acl anthology network corpus.Language Resources and Eval- uation, 47,

[Radevet al., 2013 ] Dragomir R Radev, Pradeep Muthukr- ishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus.Language Resources and Eval- uation, 47,

2013
[2023]

Adaptive universal generalized PageR- ank graph neural network

[Chienet al., 2021 ] Eli Chien, Jianhao Peng, Pan Li, and Ol- gica Milenkovic. Adaptive universal generalized PageR- ank graph neural network. InICLR,

2021
[2024]

Rethinking attention with performers.arXiv preprint arXiv:2009.14794,

[Choromanskiet al., 2020 ] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794,

2020 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

[Askey and Wilson, 1985] Richard Askey and James Arthur Wilson.Some basic hypergeometric orthogonal polyno- mials that generalize Jacobi polynomials, volume

work page 1985

[2] [9]

Convolutional neural net- works on graphs with fast localized spectral filtering

[Defferrardet al., 2016 ] Micha¨el Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural net- works on graphs with fast localized spectral filtering. In NeurIPS,

work page 2016

[3] [12]

Mamba: Linear- time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

[Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

arXiv 2023

[4] [13]

Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

[Guet al., 2021 ] Albert Gu, Karan Goel, and Christopher R´e. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

arXiv 2021

[5] [14]

Transformers are rnns: Fast autoregressive transformers with linear attention

[Katharopouloset al., 2020 ] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InICML. PMLR,

work page 2020

[6] [15]

Polynomial-based self-attention for table representation learning

[Kimet al., 2024 ] Jayoung Kim, Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. Polynomial-based self-attention for table representation learning. InInter- national Conference on Machine Learning, pages 24509– 24526. PMLR,

work page 2024

[7] [16]

Kipf and Max Welling

[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. InICLR,

work page 2017

[8] [18]

Learning multiple layers of features from tiny im- ages

[Krizhevskyet al., 2009 ] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages

2009