Pith. sign in

REVIEW 3 major objections 5 minor 36 references

A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Across ten pretrained transformers, the claim that self-attention performs kernel PCA fails to reproduce.

desk verdict A broad reproduction study with a plausible negative result, but the eigenvalue refutation hinges on an undocumented normalization choice and the conclusion overreaches. read the letter →

arxiv 2505.07908 v1 pith:N4L27N4L submitted 2025-05-12 cs.LG cs.AIcs.CLcs.CV

Karahan Sar{i}taş , Çağatay Y{i}ld{i}z This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.CV
keywords self-attentionkernelPCAreproductionstudytransformersinterpretabilityGrammatrixeigenvaluesprojectionlossCenteredAlignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper re-tests the recent claim that a transformer's self-attention head is secretly doing kernel principal component analysis (KPCA): that learned value vectors encode the eigenvectors of the key Gram matrix and attention outputs are projections of queries onto principal-component axes in a feature space. Across ten pretrained vision transformers and nine encoder-only language models, the authors find almost no correspondence between learned value vectors and the KPCA-predicted value matrix under four similarity measures, with best cosine similarity reaching only 0.32 on vision models. They also argue that the decreasing projection loss reported as evidence mostly reflects a mismatch in scale: the two squared norms involved differ by orders of magnitude, so the apparent minimization tracks shrinking output norms rather than genuine alignment. Finally, they show that the Gram-matrix eigenvalue statistics that anchored the original argument cannot be reproduced without undocumented implementation-specific adjustments. If correct, the paper removes an appealing bridge between attention and classical kernel methods, while leaving open weaker descriptive kernel views.

What carries the argument

The load-bearing objects are the centered Gram matrix $\tilde{K}_\varphi(i,j)=\tilde{\varphi}(k_i)^\top\tilde{\varphi}(k_j)$, built from key vectors through the feature map $\varphi(k_j)=\phi(k_j)/g(k_j)$ with per-key scaling $g$, and the resulting KPCA value matrix $\dot{V}_{\text{KPCA}}=GA-G\mathbf{1}_N A$, where $G=\operatorname{diag}(1/g(k_j))$, $\mathbf{1}_N$ is the all-$1/N$ matrix, and $A$ holds the top $d_v$ eigenvectors of $\tilde{K}_\varphi$. The hypothesis under test is $V\approx\dot{V}_{\text{KPCA}}$, and the paper measures it with four scale-normalized similarity metrics: direct cosine similarity, optimally matched cosine similarity, linear CKA, and kernel CKA. For the projection-loss claim, the key algebraic step is expanding $J_{\text{proj}}$ into the squared norms $\|\varphi(q_i)\|^2$ and $\|h_i\|^2$ together with a cross-term that depends on eigenvector assignment; this decomposition shows the loss is dominated by the output norm. For the spectral claim, the paper examines the actual eigenvalues of $\tilde{K}_\varphi$ and demonstrates that small absolute differences in the ratio vector $\gamma$ do not certify the eigenvector condition when the eigenvalues themselves are tiny.

What would settle it

Run an unmodified implementation of the original spectral computation on one pretrained vision-transformer head and record the maximum absolute eigenvalue of the centered Gram matrix: if it is near 648 the original claim survives this test, and if it is below $10^{-5}$ the reproduction stands; repeat the value-matrix comparison with and without the $g(k_j)$ scaling to check whether the reported mismatch is robust.

Watch

Extended reading notes

Core claim

The paper's central claim is that the KPCA interpretation of self-attention collapses under independent empirical scrutiny. Using pretrained vision transformers and encoder-only language models, the authors test the three empirical pillars of that interpretation. First, the learned value matrix $V$ does not approximate the KPCA value matrix $\dot{V}_{\text{KPCA}}$: even after optimally matching columns one-to-one, the best cosine similarity reaches only 0.32 on vision models (0.57 on one small language encoder), while linear CKA stays at or below 0.13. Second, the reported decrease in the projection loss $J_{\text{proj}}$ is an artifact of scale: the squared feature-map norm $\|\varphi(q_i)\|^2$ is roughly $10^3$ times smaller than the output norm $\|h_i\|^2$, so the plotted decrease tracks shrinking output magnitudes rather than convergence of query features to their reconstruction. Third, the Gram-matrix eigenvalue statistics reported as spectral confirmation are not reproducible; computing the same quantities across ten models gives eigenvalues on the order of $10^{-6}$ rather than the reported values around hundreds, and the ratio-difference plots used in the original argument can look convincing even for random directions. The paper concludes that this specific derivation of self-attention as kernel PCA lacks empirical and theoretical support, while explicitly not rejecting robust PCA as an algorithm and not ruling out weaker, descriptive kernel views of attention.

Load-bearing premise

The refutation assumes the centered Gram matrix was built with the same per-key scaling $g(k_j)$ and feature map as the original derivation; if that normalization was omitted or misapplied, the tiny eigenvalues and low similarity scores could be artifacts rather than genuine refutations.

Editorial extensions

If this is right

  • The proposed equivalence between learned value vectors and the KPCA value matrix should be treated as empirically unsupported; four scale-invariant similarity measures all leave the correspondence below anything that would indicate convergence.
  • The decreasing projection-loss curves should not be read as evidence of KPCA alignment; the squared norms of the two reconstructed sides differ by orders of magnitude, so the loss mostly measures output-norm shrinkage.
  • The $\gamma$-difference plots are not a safe diagnostic: when Gram-matrix eigenvalues are around $10^{-6}$, nearly zero absolute differences can arise for directions that are not eigenvectors at all.
  • The original spectral claim is implementation-sensitive and needs explicit, reproducible eigenvalue computation before it can support the KPCA story; the reported magnitudes do not reappear in an independent run.
  • The negative result is limited to this derivation: descriptive kernel and projection views of attention, and algorithms that explicitly regularize toward singular-vector alignment, remain consistent with the evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test would be to train a small transformer with an added penalty $\|V-\dot{V}_{\text{KPCA}}\|$; if the KPCA geometry were actually favorable for attention, this penalty should be easy to satisfy and should preserve task accuracy, which would separate 'not spontaneously happening' from 'not realizable'.
  • The eigenvalue gap (reported values around hundreds versus reproduced values near $10^{-6}$) suggests the original spectral figures may have come from a different, undocumented preprocessing of the keys; the reproduction's own standardization experiments shift the values substantially but still do not reach the reported scale, leaving the original numbers unexplained.
  • The paper's negative result shifts the burden of proof for future interpretability claims: a kernel interpretation of a learned mechanism should come with exact, reproducible spectral statistics and a scale-invariant comparison to the actual learned weights, not ratio plots or log-scale curves.
  • The discussion of an explicit variance and orthonormality regularizer hints at a constructive way forward: kernel interpretations of attention may be better treated as training objectives that enforce low-rank structure than as properties that emerge spontaneously.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript reports a reproduction study of the claim by Teo and Nguyen (2024) that self-attention implements kernel PCA. The authors evaluate three pieces of evidence from the original paper: whether learned value matrices V align with the KPCA value matrix V_dot_KPCA (Eq. 6), whether the reported decrease in the projection loss J_proj indicates convergence, and whether the eigenvalues of the centered Gram matrix match the original statistics. Across 10 vision transformers and 9 encoder-only language models, the authors report consistently low similarity scores, argue that the observed J_proj decrease is driven by the shrinking squared norms of attention outputs rather than by genuine reconstruction, and find Gram eigenvalues on the order of 10^-6, far from the original paper's reported values. The manuscript concludes that the KPCA interpretation of self-attention lacks empirical support.

Significance. If the negative result is correct, the paper is a useful contribution: it challenges an influential interpretability claim with a broad model sweep, multiple similarity metrics, and released code. The authors are explicit about limitations, including the use of a proxy loss and numerical preprocessing. However, the force of the central claim depends on two technical points that are currently under-specified or internally inconsistent: the exact construction of the Gram matrix used to form V_dot_KPCA, and the role of Z-score standardization in the eigenvalue statistics. These issues are checkable against the released code, so the paper can be repaired, but as written the refutation is conditional on undocumented implementation details.

major comments (3)
  1. [Section 3 (first paragraph) and Eq. (6)] The construction of the quantity being compared is under-specified. Section 2 defines the feature map with per-key scaling g(k_j) and Eq. (6) defines V_dot_KPCA = G A - G 1_N A, where A must be the eigenvectors of the centered normalized Gram matrix with entries k(k_i, k_j)/(g(k_i) g(k_j)). Section 3 states only that "we first calculate the Gram matrix K_phi, center it, and then extract its eigenvectors" without saying whether K_phi is the raw kernel matrix or the g-normalized kernel matrix. If the raw kernel is centered and diagonalized, then A is not the coefficient matrix from Teo and Nguyen's KPCA derivation, and the similarity values in Tables 1 and 4, as well as the eigenvalue scale in Table 2, are not measuring the claimed object. Please state explicitly which Gram matrix is used and confirm from the released code that the g-normalization is applied before centering and eigendecomposition.
  2. [Appendix B.1 and Table 3] The claim that Z-score standardization has a negligible impact on the eigenvalues is contradicted by Table 3. For example, ViT-Large shows a +181% change in the maximum eigenvalue and a -67% change in the minimum, and DeiT-Small-D shows a +415% change in the maximum. Since Table 2 is computed after Z-score standardization, the reported 10^-6-scale statistics are not robust evidence against the original paper's numbers unless the authors either compute eigenvalues without standardization where numerically possible or explain why these large percentage changes are immaterial to the comparison. As written, the "irreproducible without undocumented implementation-specific adjustments" conclusion is itself dependent on an undocumented adjustment.
  3. [Section 3 (Figure 1) and Appendix A] The projection-loss argument uses a proxy loss J_proj = (1/N) sum | ||phi(q_i)||^2 - ||h_i||^2 | rather than the reconstruction loss defined in the text. The appendix correctly notes that the true projection loss contains an eigenvector-assignment-sensitive cross term when the eigenvectors are not orthonormal, so the decomposition into ||phi(q_i)||^2 and ||h_i||^2 is not the original J_proj. The conclusion that the observed decrease is due to collapsing ||h_i||^2 magnitudes should be stated as conditional on the proxy loss, or demonstrated for the actual loss under a fixed eigenvector assignment.
minor comments (5)
  1. [Abstract and Conclusion] The phrase "lacks empirical and theoretical robustness" overstates the evidence: the manuscript tests empirical claims only, so the word "theoretical" should be removed or supported by a theoretical analysis.
  2. [Table 2 caption] The caption reads "(x10^-6)" while the entries are given as values like 147 +/- 11; please state explicitly whether the entries are already multiplied by 10^6, since the current format invites misreading.
  3. [Appendix C] The text says "we used 100 randomly sampled images from WikiText-103 dataset"; WikiText-103 is a text corpus, so this should read "text sequences" or similar.
  4. [Figure 4] The label "x10^-6" on the lower panel is ambiguous: please specify whether the plotted relative error itself is multiplied by 10^-6 or whether the axis scale is intended.
  5. [Tables 1 and 4] Some language-model rows show notably higher values than the vision rows (e.g., MiniLM MOC 0.57, Longformer KCKA 0.45); reporting a permutation or random-baseline similarity value would help the reader calibrate whether the vision-model thresholds also apply to the NLP models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reproduction's negative claims are empirical comparisons between independently learned value matrices and KPCA quantities computed from the same models' keys, with no fitted parameter, self-citation chain, or definitional equivalence doing the work.

full rationale

The paper's central claims are empirical comparisons, not derivations that presuppose their conclusions. Learned value matrices V are parameters of pretrained transformers, while \dot{V}_{KPCA} is constructed from key Gram-matrix eigenvectors via Equations 5 and 6; nothing in that construction is fitted to V or defined in terms of V. The algebraic steps in Equations 4-7 are standard KPCA manipulations, and the paper does not use them to define the learned V. Citations to Teo and Nguyen (2024) are the target under test, not load-bearing self-citations, and the only external code cited is the original authors' repository used to reproduce and then criticize the original projection-loss computation. The acknowledged ambiguity about whether the g-normalized Gram matrix from Equation 6 is the one centered and diagonalized is a fidelity and reproducibility concern about the refutation's objects, not circularity: computing A from the wrong Gram matrix would change the comparison, but the conclusion would still not be an input to the computation. The paper also invokes no uniqueness theorem from its own authors, imports no ansatz through self-citation, and renames no known result. Its admitted limitations (Z-score preprocessing, proxy loss, eigenvalue instability) affect evidentiary strength, not circular structure. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted free parameters and no new entities. Its central claim rests on implementation fidelity assumptions about the kernel feature map, the per-key normalization, the use of pretrained checkpoints, and numerical stability of eigendecomposition. These are domain assumptions inherited from the original work and from standard practice in reproduction studies.

assumptions (4)
  • domain assumption The kernel k(x,y)=exp(x^T y / sqrt(d_q)) and the associated feature map used in the reproduction match the objects defined in Teo and Nguyen (2024).
    The reproduction's comparison depends on the KPCA quantities being built from the same kernel and feature map as the original; the paper assumes this equivalence in Sections 2 and 3 without proof.
  • domain assumption The centered Gram matrix is computed with the same normalization, including g(k_j), that Eq. 6 implies.
    Section 3 describes centering K_phi but does not explicitly state whether the per-key normalization G is applied before eigendecomposition; the eigenvalue-scale result and the V similarity comparison both depend on this choice.
  • domain assumption Pretrained transformer checkpoints faithfully represent models trained with standard objectives and no KPCA-specific regularizer.
    The test assumes that if the KPCA interpretation were true, it would emerge in standard trained transformers; this is the intended claim of Teo and Nguyen but is not independently proven.
  • domain assumption Numerical eigendecomposition of the centered Gram matrix is stable enough for the reported comparisons after Z-score standardization.
    Appendix B.1 acknowledges numerical instability and applies standardization, which changes eigenvalues by up to 415% in Table 3; the paper still interprets the result as a refutation of the original scale.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny." pith.science (2026). https://pith.science/paper/N4L27N4L

@misc{pith2026250507908,
  author       = {Pith},
  title        = {Pith review of: A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4L27N4L}},
  note         = {Machine review of arXiv:2505.07908}
}
abstract

In this reproduction study, we revisit recent claims that self-attention implements kernel principal component analysis (KPCA) (Teo et al., 2024), positing that (i) value vectors $V$ capture the eigenvectors of the Gram matrix of the keys, and (ii) that self-attention projects queries onto the principal component axes of the key matrix $K$ in a feature space. Our analysis reveals three critical inconsistencies: (1) No alignment exists between learned self-attention value vectors and what is proposed in the KPCA perspective, with average similarity metrics (optimal cosine similarity $\leq 0.32$, linear CKA (Centered Kernel Alignment) $\leq 0.11$, kernel CKA $\leq 0.32$) indicating negligible correspondence; (2) Reported decreases in reconstruction loss $J_\text{proj}$, arguably justifying the claim that the self-attention minimizes the projection error of KPCA, are misinterpreted, as the quantities involved differ by orders of magnitude ($\sim\!10^3$); (3) Gram matrix eigenvalue statistics, introduced to justify that $V$ captures the eigenvector of the gram matrix, are irreproducible without undocumented implementation-specific adjustments. Across 10 transformer architectures, we conclude that the KPCA interpretation of self-attention lacks empirical support.

Figures

Figures reproduced from arXiv: 2505.07908 by the authors.

Figure 1
Figure 1. Reconstruction loss (Jproj) over training epochs for ViT-Tiny and DeiT-Tiny models, along with the values of the individual squared norms, shown with markers. Circle markers indicate average of squared output norms (∥hi∥ 2 ) and triangle markers (extremely low values around 10−3 ) show the average of squared feature map norms (∥φ(qi)∥ 2 ). At first, decreasing projection loss Jproj may seem to indicate a meaningful … view at source ↗
Figure 2
Figure 2. Comparison of squared norms across transformer layers. The plots show medians (solid lines) and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Relative absolute reconstruction train/test er [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: (ViT Tiny) Top row: Mean and standard deviation of the absolute differences of entries in the [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]
Figure 5
Figure 5. Figure 5: Distribution of |φ(qi)| 2 (blue) and |hi | 2 (red) across layers of nine pre-trained encoder-only language models (log scale) (ordered by the parameter count). Each plot shows the median (solid line) and 95th percentile (shaded region) of the squared norm values across…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 5 canonical work pages

  1. [1]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150

    arXiv 2020

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://arxiv.org/abs/2005.14165 Lan...

    arXiv 2020

  3. [3]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. https://arxiv.org/abs/2104.14294 Emerging properties in self-supervised vision transformers . Preprint, arXiv:2104.14294

    arXiv 2021

  4. [4]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. https://arxiv.org/abs/2106.01345 Decision transformer: Reinforcement learning via sequence modeling . Preprint, arXiv:2106.01345

    arXiv 2021

  5. [5]

    Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan A. K. Suykens. 2023. https://arxiv.org/abs/2305.19798 Primal-attention: Self-attention through asymmetric kernel svd in primal representation . Preprint, arXiv:2305.19798

    arXiv 2023

  6. [6]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022. https://arxiv.org/abs/2009.14794 Rethinking attention with performers . Preprint, arXiv:2009.14794

    arXiv 2022

  7. [7]

    Sankalan Pal Chowdhury, Adamos Solomou, Avinava Dubey, and Mrinmaya Sachan. 2022. https://arxiv.org/abs/2110.08323 On learning the transformer kernel . Preprint, arXiv:2110.08323

    work page Pith review arXiv 2022

  8. [8]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://arxiv.org/abs/2003.10555 Electra: Pre-training text encoders as discriminators rather than generators . Preprint, arXiv:2003.10555

    arXiv 2020

Show all 36 references
  1. [9]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . C...

    2019 arXiv

  2. [10]

    David F. Crouse. 2016. https://doi.org/10.1109/TAES.2016.140952 On implementing 2d rectangular assignment algorithms . IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679--1696

    2016

  3. [11]

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2019. https://arxiv.org/abs/1807.03819 Universal transformers . Preprint, arXiv:1807.03819

    2019 arXiv

  4. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding . Preprint, arXiv:1810.04805

    2019 arXiv

  5. [14]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://arxiv.org/abs/2010.11929 An image is worth 16x16 words...

    2021 arXiv

  6. [15]

    Patrick Esser, Robin Rombach, and Björn Ommer. 2021. https://arxiv.org/abs/2012.09841 Taming transformers for high-resolution image synthesis . Preprint, arXiv:2012.09841

    2021 arXiv

  7. [16]

    Dai, Matthew D

    Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2018. https://arxiv.org/abs/1809.04281 Music transformer . Preprint, arXiv:1809.04281

    2018 arXiv

  8. [17]

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. https://arxiv.org/abs/1905.00414 Similarity of neural network representations revisited . Preprint, arXiv:1905.00414

    2019 arXiv

  9. [18]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized BERT pretraining approach . CoRR, abs/1907.11692

    2019 arXiv

  10. [19]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. https://arxiv.org/abs/2103.14030 Swin transformer: Hierarchical vision transformer using shifted windows . Preprint, arXiv:2103.14030

    2021 arXiv

  11. [20]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su \'a rez, Yoann Dupont, Laurent Romary, \'E ric Villemonte de la Clergerie, Djam \'e Seddah, and Beno \^ t Sagot. 2020. Camembert: a tasty french language model. In Proceedings of the 58th Annual Meeting of the Association fo...

    2020

  12. [21]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . Preprint, arXiv:1609.07843

    2016 arXiv

  13. [22]

    Nguyen, Tam Nguyen, Nhat Ho, Andrea L

    Tan M. Nguyen, Tam Nguyen, Nhat Ho, Andrea L. Bertozzi, Richard G. Baraniuk, and Stanley J. Osher. 2024. https://arxiv.org/abs/2406.13781 A primal-dual framework for transformers and neural networks . Preprint, arXiv:2406.13781

    2024 arXiv

  14. [23]

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. https://arxiv.org/abs/1802.05751 Image transformer . Preprint, arXiv:1802.05751

    2018 arXiv

  15. [24]

    Pedro Ponte and Roger G. Melko. 2017. https://doi.org/10.1103/physrevb.96.205146 Kernel methods for interpretable machine learning of order parameters . Physical Review B, 96(20)

    2017 doi

  16. [25]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683

    2023 arXiv

  17. [26]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  18. [27]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. https://arxiv.org/abs/1409.0575 Imagenet large scale visual recognition challenge . Prepri...

    2015 arXiv

  19. [28]

    Hunter, Costas Bekas, and Alpha A

    Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A. Hunter, Costas Bekas, and Alpha A. Lee. 2019. https://doi.org/10.1021/acscentsci.9b00576 Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction . ACS Central Scien...

    2019 doi

  20. [29]

    Johan A.K. Suykens. 2016. https://doi.org/10.1016/j.acha.2015.09.004 Svd revisited: A new variational principle, compatible feature maps and nonlinear extensions . Applied and Computational Harmonic Analysis, 40(3):600--609

    2016 doi

  21. [30]

    Rachel S. Y. Teo and Tan M. Nguyen. 2024. https://arxiv.org/abs/2406.13762 Unveiling the hidden structure of self-attention via kernel principal component analysis . Preprint, arXiv:2406.13762

    2024 arXiv

  22. [31]

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. https://arxiv.org/abs/2012.12877 Training data-efficient image transformers & distillation through attention . Preprint, arXiv:2012.12877

    2021 arXiv

  23. [32]

    Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. https://arxiv.org/abs/1908.11775 Transformer dissection: A unified understanding of transformer's attention via the lens of kernel . Preprint, arXiv:1908.11775

    2019 arXiv

  24. [33]

    Leena Chennuru Vankadara and Debarghya Ghoshdastidar. 2019. https://arxiv.org/abs/1912.00458 On the optimality of kernels for high-dimensional clustering . Preprint, arXiv:1912.00458

    2019 arXiv

  25. [34]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    2023 arXiv

  26. [35]

    Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep contextualized entity representations with entity-aware self-attention. In EMNLP

    2020

  27. [36]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.