Pith. sign in

REVIEW 4 major objections 4 minor 56 references

SchoenbAt: Rethinking Attention with Polynomial basis

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Polynomial-basis random Maclaurin features make dot-product kernelized attention unbiased and linear-time.

desk verdict SchoenbAt is a plausible efficient-attention idea with solid experiments, but its central unbiasedness theorem is not proven — the proof swaps expectation and division for dependent random variables. read the letter →

arxiv 2505.12252 v1 pith:MJWPXQUX submitted 2025-05-18 cs.LG

Yuhan Guo , Lizhong Ding , Yuwan Yang , Xuewei Guo This is my paper · ORCID
classification cs.LG
keywords kernelizedattentionrandomMaclaurinfeaturesSchoenberg'stheorempolynomialbasisefficienttransformersdot-productkernelsunbiasedapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SchoenbAt re-expresses dot-product kernelized attention, the family that includes softmax attention when the kernel is exponential, as a polynomial basis expansion guaranteed by Schoenberg's theorem. Instead of forming the n by n attention matrix, it estimates each polynomial term with random Maclaurin features, cutting the computation from O($n^{2}$ d) to O(ndD), where D is the number of random features. A pre-post scaling batch normalization keeps the input in the bounded domain the theorem requires and rescales the output so that the estimator can remain unbiased. The paper proves unbiasedness in expectation and a concentration bound that shrinks as D grows, and shows on the Long Range Arena tasks that the replacement speeds up training while keeping accuracy competitive.

What carries the argument

The load-bearing object is the Random Maclaurin Feature map Phi_K(x) = $\sqrt$(1/D)[phi_1(x), ..., phi_D(x)], with each phi_i(x) = $\sqrt$(a_N $p^{{N+1}}$) prod_{j=1}^N <omega_j, x>, where N is drawn from a geometric law, a_N is the N-th Maclaurin coefficient of the dot-product kernel, and the omega_j are Rademacher vectors. Its inner product reproduces, in expectation, the polynomial expansion that Schoenberg's theorem says such kernels possess. The two-stage ppSBN then normalizes Q and K into the $ell^{2}$(0,1) ball required by the theorem and applies trainable parameters to reverse the scaling on the output, so the estimator targets the original attention rather than a rescaled version.

What would settle it

Fix a Q, K, V and a dot-product kernel with non-negative Maclaurin coefficients such as exp(·), draw many independent random feature sets with a fixed D, and compute the empirical average of SchoenbAt's output alongside the exact kernelized attention; if the average differs from the exact value by more than Monte Carlo error even as D grows, the claimed unbiasedness is not empirically supported. A more targeted check compares E[numerator]/E[denominator] with E[numerator/denominator] on the same feature draws to isolate the ratio-of-expectations step.

Watch

Extended reading notes

Core claim

The central claim is that kernelized attention with any dot-product kernel whose Maclaurin coefficients are non-negative, including the exp, inverse, logarithmic, trigonometric-hyperbolic, and square-root kernels, can be approximated by a random Maclaurin feature attention (RMFA) that is unbiased in expectation and whose error concentrates like 2D exp(-D $epsilon^{2}$ / (2 $S^{2}$ $d^{2}$)). The proof route is that Schoenberg's theorem gives a polynomial expansion, random Maclaurin features estimate that expansion, and the ppSBN mechanism keeps inputs within the required bounded domain while trainable parameters restore the original output scale. When the scaling parameters are properly trained, SchoenbAt becomes a drop-in replacement for dot-product kernelized attention with cost linear in the sequence length, and the theory says accuracy improves as the random feature dimension D increases.

Load-bearing premise

The unbiasedness proof assumes that the expected numerator divided by the expected denominator equals the expected ratio, even though the numerator and denominator are dependent because they share the same random features; that equality is false in general.

Editorial extensions

If this is right

  • Sequence-length cost drops from O(n^2 d) to O(ndD), so the speed benefit grows as the context gets longer.
  • Because the approximation is unbiased in expectation, SchoenbAt can be plugged into existing Transformer layers without changing the layer's input or output signature.
  • The concentration bound gives a quantitative control knob: raise D when accuracy is critical, lower D when latency or memory dominates.
  • The exponential-kernel case behaves like softmax attention, making SchoenbAt a drop-in accelerated softmax for typical Transformers.
  • Among the five tested kernels, no single one strictly dominates, and the paper recommends the exponential kernel by default while leaving the kernel choice open per task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same Schoenberg basis could be applied to any other dot-product kernel with non-negative Maclaurin coefficients, turning SchoenbAt into a general library of linear-time attention variants beyond the five kernels tabulated here.
  • Truncating the random polynomial degree N instead of drawing it from a geometric law would produce a deterministic bias-variance trade-off that the current bounds do not yet describe.
  • An empirical check of the estimator's finite-D expectation over many independent feature draws would show how quickly E[SchoenbAt] approaches the exact attention, a quantity the theorem guarantees only asymptotically.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SchoenbAt, an approximation of dot-product kernelized attention using random Maclaurin features based on Schoenberg's theorem, with a two-stage normalization (ppSBN) intended to keep inputs in a bounded domain and restore output scale. The authors claim an unbiasedness result (Theorem 3) and a concentration bound (Theorem 4), and they report experiments on synthetic approximation error, speedup, and the LRA benchmark, together with an ablation study. The central theoretical contribution is the characterization of SchoenbAt as an unbiased estimator of kernelized attention.

Significance. If the theoretical guarantees were valid, the paper would offer a useful alternative to Fourier-based random feature attention by directly expanding dot-product kernels in a polynomial basis, with a complexity of O(ndD) and empirical speedups on long sequences. The empirical evaluation is broad: five kernels, synthetic convergence checks, and LRA comparisons with several efficient attention baselines. The anonymous code link and reproducible experimental protocol are positive features. However, the load-bearing theoretical results are not established: the unbiasedness proof contains an invalid expectation-of-ratio step, the concentration proof mischaracterizes the estimator's randomness, and the key domain assumption is unjustified. As a result, the paper's main claims of guaranteed approximation quality are unsupported, and the contribution reduces to an empirically tested heuristic.

major comments (4)
  1. [Appendix A.3, Theorem 3] The proof of Theorem 3 replaces E[RMFA_K(Q,K,V)], which is the expectation of a ratio, with a ratio of expectations. In the displayed equations, the numerator and denominator are built from the same random Maclaurin features phi(Q/d^{1/4}) and phi(K_i/d^{1/4}), so they are dependent random variables. For positive denominators, Jensen's inequality shows E[N/D] != E[N]/E[D] in general unless D is constant. Therefore the claimed unbiasedness E[SchoenbAt_K(Q,K,V)] = attn_K(Q,K,V) does not follow from the supplied argument.
  2. [Appendix A.4, Theorem 4] The proof of Theorem 4 rewrites the deviation event as the existence of a single feature phi_t whose ratio deviates from attn_K, and then unions over t. This does not match the actual estimator: both the numerator and denominator of RMFA are averages over all D random features, and the deviation of the averaged ratio cannot be decomposed into independent single-feature events. Moreover, Hoeffding's inequality is applied to the uncentered ratio terms without verifying their range or independence. Consequently the bound P(|SchoenbAt_K(D)-attn_K(D)|>epsilon) <= 2D exp(-D epsilon^2/(2 S^2 d^2)) is not established.
  3. [Appendix A.3 and Section 3.3] The claim that 'with d>=1, we can ensure that Q/d^{1/4}, K/d^{1/4} in ell_2(0,1)' is false in general: arbitrary attention inputs can have arbitrary norms, and dividing by d^{1/4} does not place them in the unit ball. The actual algorithm feeds the ppSBN-normalized inputs QSBN and KSBN to RMFA, but the proof of Theorem 3 uses the unnormalized Q and K. This gap affects both Theorem 3 and Theorem 4, since the input-domain assumption is needed for Schoenberg's theorem to apply.
  4. [Theorem 3 statement and Algorithm 1] The condition that 'the parameters gamma and beta are properly trained to restore the attention output' is not a mathematical assumption that can be verified or guaranteed. The proof of Theorem 3 relies on Equation (5), which already presupposes the exact restoration property expressed by Theorem 2 and the fitted gamma, beta. The toy experiment in Figure 3 only shows that training with ppSBN reaches similar loss, perplexity, and BLEU scores; it does not establish equality of the output with attn_K. Thus the unbiasedness result is conditional on an unproved learned-correction assumption.
minor comments (4)
  1. [Section 2.1] The notation X/Y for element-wise division is nonstandard and could be confused with matrix division; it should be defined more prominently or replaced with a symbol such as ⊘.
  2. [Appendix A.2] There is a typo: 'Combinign' should be 'Combining'. The proof of Theorem 2 is also difficult to follow because the definitions of r, s, and t are introduced only after they are used; reorganizing the derivation would improve readability.
  3. [Table 3 caption] The caption says 'reporting normalized training time and prediction accuracy under different settings', but the row labels 'exp(.) inv(.) logi(.) trigh(.) sqrt(.)' are ambiguous; they should be clearly identified as the kernel choices for the RMFA and SchoenbAt configurations.
  4. [Section 4.1] The statement that the approximation error 'confirms our Theorem 4' is too strong: a numerical decrease in error with increasing D is consistent with many convergence mechanisms and does not by itself validate the specific exponential bound in Theorem 4.

Circularity Check

1 steps flagged · score 8.0 of 10

Theorem 3's unbiasedness is assumed, not derived: the trainable rescaling γ,β is required to be 'properly trained to restore the attention output', and the proof then uses exactly that condition to drop the rescaling, so the central theoretical guarantee reduces to the fitted correction.

  1. self definitional [Theorem 3, Section 3.3; proof in Appendix A.3 using Eq. (5); same assumption repeated in Theorem 4]
    "Assuming that the parameters γ and β are properly trained to restore the attention output, ΦK(·) :R d→R D defines a random Maclaurin feature map for a dot-product kernelK(·), then for any attention inputsQ,K,V⊂R n×d, we haveE[SchoenbAtK(Q,K,V)] = attn K(Q,K,V)."

    The theorem's conclusion is contained in its assumption: 'properly trained to restore the attention output' already says that the trainable post-scaling undoes the RMFA distortion. The Appendix A.3 proof invokes Eq. (5) and the 'properly trained' condition to replace E[γ RMFA^β] by E[RMFA], then identifies the latter with attn via an expectation-of-ratio step. No training dynamics or parameter-free bound shows that γ,β achieve restoration; the unbiasedness and concentration guarantees are therefore conditional on the very fitted values they are supposed to justify. Section 4.1 confirms the empirical validation itself uses 'ideally trained' γ,β.

full rationale

The central circularity is the 'properly trained to restore the attention output' assumption in Theorems 3 and 4. That assumption is not an external, verified condition; it is the target property restated, so the claimed unbiasedness and concentration guarantees reduce by construction to saying that if the trainable rescaling fixes the output, then the output is fixed. The remaining steps—RMF unbiasedness for the dot-product kernel, Theorem 1's algebraic factorization, the complexity analysis, and the LRA experiments—are substantive and not circular, giving the method independent algorithmic and empirical content. There are also genuine mathematical errors in Appendix A.3 and A.4, such as replacing E[num/den] by E[num]/E[den] with dependent numerator and denominator and using a union-bound event decomposition that does not match the averaged estimator, but these are correctness flaws rather than circularity. No self-citation chain is load-bearing: Schoenberg's theorem, RMF [40], and batch normalization [41] are cited external results. Because the paper's headline theoretical guarantee is conditioned on fitted parameters whose restoration property is exactly the conclusion, the derivation is substantially circular, although not a full 10 since the architecture and empirical results stand independently.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a classical theorem (Schoenberg), an unproven bound on arbitrary inputs, and trainable scale parameters whose correct training is assumed rather than derived. The input-bound assumption is handled in practice by ppSBN, but the theory does not use the ppSBN inputs, and the output restoration is fitted.

free parameters (2)
  • gamma (post-SBN scale) = learned via backpropagation
    Trainable parameter in Algorithm 1 that rescales the RMFA output; Theorems 3 and 4 assume it is properly trained to restore the attention output.
  • beta (post-SBN exponent) = learned via backpropagation
    Trainable exponent in Algorithm 1; likewise assumed properly trained for the unbiasedness theorem.
assumptions (5)
  • standard math Schoenberg's theorem: a dot-product kernel on the unit sphere is positive definite iff its Maclaurin coefficients are non-negative
    Used in Section 3.1 to justify the polynomial basis expansion of dot-product kernels in attention.
  • standard math Mercer's theorem for kernel expansions
    Invoked in Section 3.1 as the general framework for viewing kernels as weighted sums of basis functions.
  • ad hoc to paper Attention inputs satisfy ||Q/d^{1/4}|| and ||K/d^{1/4}|| ≤ 1 (membership in ℓ2(0,1))
    Assumed in the proof of Theorem 3 (Appendix A.3) to apply Schoenberg's theorem; false for arbitrary inputs and not enforced by the stated algorithm without ppSBN, which the proof does not use.
  • domain assumption The kernels inv, logi, and sqrt have convergent Maclaurin series at all inputs reached after normalization
    Table 1 restricts the domains of these kernels to inputs <1, but normalized queries and keys can have inner products equal to 1; the paper does not describe how divergence is handled.
  • ad hoc to paper γ and β can be trained to restore the original attention output exactly
    Theorems 2, 3, and 4 assume this 'proper training' without proof; empirical results show trained parameters improve accuracy but do not demonstrate exact restoration of the softmax output.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SchoenbAt: Rethinking Attention with Polynomial basis." pith.science (2026). https://pith.science/paper/MJWPXQUX

@misc{pith2026250512252,
  author       = {Pith},
  title        = {Pith review of: SchoenbAt: Rethinking Attention with Polynomial basis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJWPXQUX}},
  note         = {Machine review of arXiv:2505.12252}
}
read the original abstract

Kernelized attention extends the attention mechanism by modeling sequence correlations through kernel functions, making significant progresses in optimizing attention. Under the guarantee of harmonic analysis theory, kernel functions can be expanded with basis functions, inspiring random feature-based approaches to enhance the efficiency of kernelized attention while maintaining predictive performance. However, current random feature-based works are limited to the Fourier basis expansions under Bochner's theorem. We propose Schoenberg's theorem-based attention (SchoenbAt), which approximates dot-product kernelized attention with the polynomial basis under Schoenberg's theorem via random Maclaurin features and applies a two-stage regularization to constrain the input space and restore the output scale, acting as a drop-in replacement of dot-product kernelized attention. Our theoretical proof of the unbiasedness and concentration error bound of SchoenbAt supports its efficiency and accuracy as a kernelized attention approximation, which is also empirically validated under various random feature dimensions. Evaluations on real-world datasets demonstrate that SchoenbAt significantly enhances computational speed while preserving competitive performance in terms of precision, outperforming several efficient attention methods.

Figures

Figures reproduced from arXiv: 2505.12252 by the authors.

Figure 1
Figure 1. The module design structure of SchoenbAt. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Computation graphs for kernelized attention and RMFA. In each figure, the data on the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The loss, perplexity (lower is better), and Bleu scores (higher is better) of the traditional [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The average absolute differences between SchoenbAt and kernelized attention on five [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The speedup ratio of SchoenbAt relative to kernelized attention on five dot-product kernels. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 44 canonical work pages

  1. [1]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. InICLR, 2015

    2015

  2. [2]

    Cross-lingual language model pretraining

    Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. InNeurIPS 32, 2019

    work page 2019

  3. [3]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  4. [4]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. InNeurIPS 35, 2022

    work page 2022

  5. [5]

    Dynamic perceiver for efficient visual recognition

    Yizeng Han, Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng, Junlan Feng, Shiji Song, and Gao Huang. Dynamic perceiver for efficient visual recognition. InICCV, 2023

    work page 2023

  6. [6]

    Learning to weight samples for dynamic early-exiting networks

    Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song, Junfeng Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic early-exiting networks. InECCV, 2022

    work page 2022

  7. [7]

    Contrastive language-image pre- training with knowledge graphs.NeurIPS 35, 2022

    Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, and Gao Huang. Contrastive language-image pre- training with knowledge graphs.NeurIPS 35, 2022

    work page 2022

  8. [8]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. InCVPR, 2024

    2024

Show all 56 references
  1. [9]

    ResaPred: A deep residual network with self-attention to predict protein flexibility

    Wei Wang, Shitong Wan, Hu Jin, Dong Liu, Hongjun Zhang, Yun Zhou, and Xianfang Wang. ResaPred: A deep residual network with self-attention to predict protein flexibility. InTCBB, 2025

    2025

  2. [10]

    MCHAN: Prediction of human microbe- drug associations based on multiview contrastive hypergraph attention network.Current Bioinformatics, 20(1):70–86, 2025

    Guanghui Li, Ziyan Cao, Cheng Liang, Qiu Xiao, and Jiawei Luo. MCHAN: Prediction of human microbe- drug associations based on multiview contrastive hypergraph attention network.Current Bioinformatics, 20(1):70–86, 2025

    2025

  3. [11]

    Addressing scalability and managing sparsity and dropout events in single-cell representation identification with ZIGACL.Briefings in Bioinformatics, 26(1):bbae703, 2025

    Mingguang Shi and Xuefeng Li. Addressing scalability and managing sparsity and dropout events in single-cell representation identification with ZIGACL.Briefings in Bioinformatics, 26(1):bbae703, 2025

    2025

  4. [12]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS 30, 2017

    2017

  5. [13]

    A brief overview of ChatGPT: The history, status quo and potential future development.IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136, 2023

    Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. A brief overview of ChatGPT: The history, status quo and potential future development.IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136, 2023

    2023

  6. [15]

    Deepseek-v3 technical report.arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv:2412.19437, 2024

    2024 arXiv

  7. [16]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv:2412.15115, 2024

    2024 arXiv

  8. [17]

    Transformers are Rnns: fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are Rnns: fast autoregressive transformers with linear attention. InICML, 2020

    2020

  9. [18]

    Transformer dissection: An unified understanding for transformer‘s attention via the lens of kernel

    Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer‘s attention via the lens of kernel. In EMNLP-IJCNLP, 2019

    2019

  10. [19]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv:2006.04768, 2020

    2006 arXiv

  11. [20]

    Nyströmformer: A nyström-based algorithm for approximating self-attention

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. InAAAI, 2021

    2021

  12. [21]

    Skyformer: Remodel self-attention with gaussian kernel and Nyström method

    Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and Nyström method. InNeurIPS 34, 2021

    2021

  13. [22]

    Primal-attention: Self-attention through asymmetric kernel svd in primal representation

    Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan Suykens. Primal-attention: Self-attention through asymmetric kernel svd in primal representation. InNeurIPS 36, 2024. 10

    2024

  14. [23]

    Large-kernel attention for 3D medical image segmentation.Cognitive Computation, 16(4):2063–2077, 2024

    Hao Li, Yang Nan, Javier Del Ser, and Guang Yang. Large-kernel attention for 3D medical image segmentation.Cognitive Computation, 16(4):2063–2077, 2024

    2024

  15. [24]

    Large kernel spectral and spatial attention networks for hyperspectral image classification.IEEE Transactions on Geoscience and Remote Sensing, 61:1–15, 2023

    Genyun Sun, Zhaojie Pan, Aizhu Zhang, Xiuping Jia, Jinchang Ren, Hang Fu, and Kai Yan. Large kernel spectral and spatial attention networks for hyperspectral image classification.IEEE Transactions on Geoscience and Remote Sensing, 61:1–15, 2023

    2023

  16. [25]

    Interscience Publishers, 1962

    Walter Rudin.Fourier Analysis on Groups. Interscience Publishers, 1962

    1962

  17. [26]

    Random features for large-scale kernel machines.NeurIPS 20, 2007

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines.NeurIPS 20, 2007

    2007

  18. [27]

    Learning with SGD and random features

    Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with SGD and random features. In NeurIPS 31, 2018

    2018

  19. [28]

    Local random feature approximations of the Gaussian kernel

    Jonas Wacker and Maurizio Filippone. Local random feature approximations of the Gaussian kernel. Procedia Computer Science, 207:987–996, 2022

    2022

  20. [29]

    New random projections for isotropic kernels using stable spectral distributions.arXiv:2411.02770, 2024

    Nicolas Langrené, Xavier Warin, and Pierre Gruet. New random projections for isotropic kernels using stable spectral distributions.arXiv:2411.02770, 2024

    2024

  21. [30]

    Random feature maps for the itemset kernel

    Kyohei Atarashi, Subhransu Maji, and Satoshi Oyama. Random feature maps for the itemset kernel. In AAAI, 2019

    2019

  22. [31]

    Explicit approximations of the Gaussian kernel

    Andrew Cotter, Joseph Keshet, and Nathan Srebro. Explicit approximations of the Gaussian kernel. arXiv:1109.4603, 2011

    2011 arXiv

  23. [32]

    Random feature attention

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. InICLR, 2021

    2021

  24. [33]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. InICLR, 2021

    2021

  25. [34]

    Random feature models with learnable activation functions

    Zailin Ma, Jiansheng Yang, and Yaodong Yang. Random feature models with learnable activation functions. arXiv:2411.19468, 2024

    2024 arXiv

  26. [35]

    Random maclaurin feature-based fuzzy clustering

    Yingxu Wang, Long Chen, Tianjun Li, and Yifei Tian. Random maclaurin feature-based fuzzy clustering. InICCSS, 2020

    2020

  27. [36]

    Learning dot-product polynomials for multiclass problems

    Ivano Lauriola, Michele Donini, and Fabio Aiolli. Learning dot-product polynomials for multiclass problems. InESANN, 2017

    2017

  28. [37]

    Duke Nguyen, Aditya Joshi, and Flora D. Salim. Spectraformer: A unified random feature framework for transformer.arXiv:2405.15310, 2024

    2024

  29. [38]

    Positive definite functions on spheres.Duke Mathematical Journal, 9(1):96–108, 1942

    Isaac Jacob Schoenberg. Positive definite functions on spheres.Duke Mathematical Journal, 9(1):96–108, 1942

    1942

  30. [39]

    Functions of positive and negative type, and their connection with the theory of integral equations.Philosophical Transactions of the Royal Society A, 209:415–446, 1909

    James Mercer. Functions of positive and negative type, and their connection with the theory of integral equations.Philosophical Transactions of the Royal Society A, 209:415–446, 1909

    1909

  31. [40]

    Random feature maps for dot product kernels

    Purushottam Kar and Harish Karnick. Random feature maps for dot product kernels. InAISTATS, 2012

    2012

  32. [41]

    Batch normalization: accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. InICML, 2015

    2015

  33. [42]

    Long range arena : A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Don Metzler. Long range arena : A benchmark for efficient transformers. InICLR, 2021

    2021

  34. [43]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InACL, 2011

    2011

  35. [44]

    ListOps: A diagnostic dataset for latent tree learning

    Nikita Nangia and Samuel Bowman. ListOps: A diagnostic dataset for latent tree learning. InNAACL, 2018

    2018

  36. [45]

    Radev, Pradeep Muthukrishnan, and Vahed Qazvinian

    Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. The ACL Anthology network corpus. InNLPIR4DL, 2009. 11

    2009

  37. [46]

    Houtkamp and P

    R. Houtkamp and P. R. Roelfsema. Parallel and serial grouping of image elements in visual perception. Journal of Experimental Psychology: Human Perception and Performance, 36(6):1443–1459, 2010

    2010

  38. [47]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  39. [48]

    Softmax-free linear transformers.International Journal of Computer Vision, 132(8):3355–3374, 2024

    Jiachen Lu, Junge Zhang, Xiatian Zhu, Jianfeng Feng, Tao Xiang, and Li Zhang. Softmax-free linear transformers.International Journal of Computer Vision, 132(8):3355–3374, 2024

    2024

  40. [49]

    Informer: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. InAAAI, 2020

    2020

  41. [50]

    CMAC neural network as an SVM with B-spline kernel functions

    Gabor Horvath. CMAC neural network as an SVM with B-spline kernel functions. InITC, 2003

    2003

  42. [51]

    Kernel methods for deep learning

    Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. InNeurIPS 22, 2009

    2009

  43. [52]

    Wavelet support vector machine.IEEE Transactions on Systems, Man, and Cybernetics, 34(1):34–39, 2004

    Li Zhang, Weida Zhou, and Licheng Jiao. Wavelet support vector machine.IEEE Transactions on Systems, Man, and Cybernetics, 34(1):34–39, 2004

    2004

  44. [53]

    An explicit description of the reproducing kernel Hilbert spaces of Gaussian RBF kernels.IEEE Transactions on Information Theory, 52(10):4635–4643, 2006

    Ingo Steinwart, Don Hush, and Clint Scovel. An explicit description of the reproducing kernel Hilbert spaces of Gaussian RBF kernels.IEEE Transactions on Information Theory, 52(10):4635–4643, 2006

    2006

  45. [54]

    Multi30K: Multilingual English-German image descriptions

    Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. Multi30K: Multilingual English-German image descriptions. InProceedings of the 5th Workshop on Vision and Language, 2016

    2016

  46. [55]

    Reformer: The efficient transformer

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. InICLR, 2020

    2020

  47. [56]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS 33, 2020

    2020

  48. [57]

    cosformer: Rethinking softmax in attention.arXiv preprint arXiv:2202.08791, 2022

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention.arXiv preprint arXiv:2202.08791, 2022. 12 Appendix A Proofs A.1 Proof of Theorem 1 Proof.We begin with attnK(Q,K,V) = nX ...

    2022 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.