{"id":"1d3ff14d-8d76-4547-a6af-db9b8388af93","arxiv_id":"2505.07908","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper reports that learned value matrices do not align with KPCA quantities, the projection-loss decrease is dominated by output-norm collapse, and the reported Gram eigenvalue statistics cannot be reproduced without undocumented preprocessing.","lead":"This reproduction study tests a recent claim that transformer self-attention performs kernel principal component analysis (KPCA). Across ten vision and nine language models, the authors find that the claimed empirical evidence for this interpretation does not hold up.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reproduction never states whether the g-normalized Gram matrix of Eq. 6 is centered and diagonalized; if only the raw kernel is used, both the similarity tables and the 10^-6 eigenvalue statistics could be artifacts.","rationale":"The reader's weakest_assumption identifies the g-normalization ambiguity, and I agree that this is the most load-bearing point. The reproduction's contribution is a negative empirical verdict, and its two strongest pieces of evidence, the V-versus-\\dot{V} similarity scores and the eigenvalue scale, both require the Gram matrix from the original theory. Without a clear statement or code-level confirmation that K_\\varphi is the g-normalized matrix before centering and eigendecomposition, the numbers in Tables 1, 2, and 4 could be measuring a different object. This is not just a documentation nit: the original's reported eigenvalues are orders of magnitude larger than the reproduction's, and the most natural explanation is that the two pipelines diagonalize different matrices. A second concern, noted in Section 5 and Table 3, is that the reproduction's own preprocessing changes eigenvalue statistics by up to 415% while being described as negligible, which weakens the eigenvalue pillar independently. Also, no null baseline is reported for the similarity metrics, so 'negligible' is not calibrated against chance. However, none of these points establishes that the paper's conclusion is false; they establish that it is conditional on implementation choices that the manuscript should make explicit. Since the paper provides code, the condition is checkable, and the reader's CONDITIONAL verdict is the right status.","tokens_in":14096,"tokens_out":10991,"duration_ms":103542,"concrete_test":"Inspect the released anonymized code at anonymous.4open.science/r/Reproduction-Study-KPCA-B01F and locate the Gram-matrix construction. Then run one patched variant: build K_\\varphi(i,j) = k(k_i, k_j) / (g(k_i) g(k_j)) with g(k_i) = \\sum_j k(k_i, k_j), center it, eigendecompose it, form \\dot{V}_{KPCA} from Eq. 6 with the same G, and recompute Table 1 for ViT-Tiny and Table 4 for BERT-Base. If the maximum optimal cosine similarity rises above a random-column-matching baseline or the eigenvalue scale changes from 10^-6 to 10^2, the paper's negative conclusion is not supported for those architectures; if both remain essentially unchanged, the g-normalization concern is settled and the reproduction stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative claim is carried by two comparisons: V versus \\dot{V}_{KPCA} (Tables 1 and 4) and the eigenvalue scale of the centered Gram matrix (Table 2). Both depend on constructing the exact Gram matrix that Teo and Nguyen define. In Section 2, the feature map is \\varphi(k_j) = \\phi(k_j)/g(k_j) with g(k_j) = \\sum_{j'} k(k_j, k_{j'}), so the Gram matrix entering Eqs. 5 and 6 has entries k(k_i, k_j) / (g(k_i) g(k_j)) before centering, not the raw kernel values. Section 3 only says 'we first calculate the Gram matrix K_\\varphi, center it, and then extract its eigenvectors' and never states whether this g-normalization is applied before centering and eigendecomposition. If the implementation centers the raw kernel and only applies G = diag(1/g) in Eq. 6, the eigenvectors A are eigenvectors of the wrong matrix and \\dot{V}_{KPCA} is not the quantity claimed by Teo and Nguyen. Since all similarity scores are computed after vector normalization, the mismatch would propagate directly into Tables 1 and 4. The same ambiguity affects the eigenvalue statistics: the 10^-6 scale in Table 2 is what one expects from a g-normalized RBF Gram matrix, while the original's 10^2 scale suggests the raw matrix; if the reproduction diagonalized the raw-centered matrix, it is comparing different objects. The released code could resolve this, but the manuscript does not state the choice, so the refutation is conditional on an undocumented implementation detail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a reproduction study of the claim by Teo and Nguyen (2024) that self-attention implements kernel PCA. The authors evaluate three pieces of evidence from the original paper: whether learned value matrices V align with the KPCA value matrix V_dot_KPCA (Eq. 6), whether the reported decrease in the projection loss J_proj indicates convergence, and whether the eigenvalues of the centered Gram matrix match the original statistics. Across 10 vision transformers and 9 encoder-only language models, the authors report consistently low similarity scores, argue that the observed J_proj decrease is driven by the shrinking squared norms of attention outputs rather than by genuine reconstruction, and find Gram eigenvalues on the order of 10^-6, far from the original paper's reported values. The manuscript concludes that the KPCA interpretation of self-attention lacks empirical support.","tokens_in":14323,"tokens_out":5074,"duration_ms":49063,"significance":"If the negative result is correct, the paper is a useful contribution: it challenges an influential interpretability claim with a broad model sweep, multiple similarity metrics, and released code. The authors are explicit about limitations, including the use of a proxy loss and numerical preprocessing. However, the force of the central claim depends on two technical points that are currently under-specified or internally inconsistent: the exact construction of the Gram matrix used to form V_dot_KPCA, and the role of Z-score standardization in the eigenvalue statistics. These issues are checkable against the released code, so the paper can be repaired, but as written the refutation is conditional on undocumented implementation details.","major_comments":[{"comment":"The construction of the quantity being compared is under-specified. Section 2 defines the feature map with per-key scaling g(k_j) and Eq. (6) defines V_dot_KPCA = G A - G 1_N A, where A must be the eigenvectors of the centered normalized Gram matrix with entries k(k_i, k_j)/(g(k_i) g(k_j)). Section 3 states only that \"we first calculate the Gram matrix K_phi, center it, and then extract its eigenvectors\" without saying whether K_phi is the raw kernel matrix or the g-normalized kernel matrix. If the raw kernel is centered and diagonalized, then A is not the coefficient matrix from Teo and Nguyen's KPCA derivation, and the similarity values in Tables 1 and 4, as well as the eigenvalue scale in Table 2, are not measuring the claimed object. Please state explicitly which Gram matrix is used and confirm from the released code that the g-normalization is applied before centering and eigendecomposition.","section":"Section 3 (first paragraph) and Eq. (6)"},{"comment":"The claim that Z-score standardization has a negligible impact on the eigenvalues is contradicted by Table 3. For example, ViT-Large shows a +181% change in the maximum eigenvalue and a -67% change in the minimum, and DeiT-Small-D shows a +415% change in the maximum. Since Table 2 is computed after Z-score standardization, the reported 10^-6-scale statistics are not robust evidence against the original paper's numbers unless the authors either compute eigenvalues without standardization where numerically possible or explain why these large percentage changes are immaterial to the comparison. As written, the \"irreproducible without undocumented implementation-specific adjustments\" conclusion is itself dependent on an undocumented adjustment.","section":"Appendix B.1 and Table 3"},{"comment":"The projection-loss argument uses a proxy loss J_proj = (1/N) sum | ||phi(q_i)||^2 - ||h_i||^2 | rather than the reconstruction loss defined in the text. The appendix correctly notes that the true projection loss contains an eigenvector-assignment-sensitive cross term when the eigenvectors are not orthonormal, so the decomposition into ||phi(q_i)||^2 and ||h_i||^2 is not the original J_proj. The conclusion that the observed decrease is due to collapsing ||h_i||^2 magnitudes should be stated as conditional on the proxy loss, or demonstrated for the actual loss under a fixed eigenvector assignment.","section":"Section 3 (Figure 1) and Appendix A"}],"minor_comments":[{"comment":"The phrase \"lacks empirical and theoretical robustness\" overstates the evidence: the manuscript tests empirical claims only, so the word \"theoretical\" should be removed or supported by a theoretical analysis.","section":"Abstract and Conclusion"},{"comment":"The caption reads \"(x10^-6)\" while the entries are given as values like 147 +/- 11; please state explicitly whether the entries are already multiplied by 10^6, since the current format invites misreading.","section":"Table 2 caption"},{"comment":"The text says \"we used 100 randomly sampled images from WikiText-103 dataset\"; WikiText-103 is a text corpus, so this should read \"text sequences\" or similar.","section":"Appendix C"},{"comment":"The label \"x10^-6\" on the lower panel is ambiguous: please specify whether the plotted relative error itself is multiplied by 10^-6 or whether the axis scale is intended.","section":"Figure 4"},{"comment":"Some language-model rows show notably higher values than the vision rows (e.g., MiniLM MOC 0.57, Longformer KCKA 0.45); reporting a permutation or random-baseline similarity value would help the reader calibrate whether the vision-model thresholds also apply to the NLP models.","section":"Tables 1 and 4"}],"recommendation":"major_revision","confidential_remarks":"The key to this paper is the released code. The authors should be asked to state precisely, and verify in the code, whether the g-normalized Gram matrix of Section 2 is the one centered and diagonalized in Section 3; this single detail determines whether the similarity tables and eigenvalue statistics address Teo and Nguyen's actual construction. The internal contradiction about Z-score preprocessing is also concerning and should be resolved before the paper can be accepted. If the code verifies the construction, the paper could be a solid negative result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious, code-releasing reproduction study that gives the KPCA interpretation of self-attention a hard empirical knock. Across 10 vision and 9 language transformers, the learned value matrices never come close to the KPCA-constructed Vdot; that consistency is worth respecting. The paper also ships code, which is more than many reproduction studies do. But the strongest negative claim—that the original Gram eigenvalue statistics are irreproducible—rests on an under-documented construction of the Gram matrix, and the paper's own Table 3 contradicts its \"negligible\" preprocessing claim. Keep treating the negative conclusion as conditional, not closed.\n\nWhat is genuinely new: the systematic similarity evaluation, the norm-magnitude reinterpretation of Jproj (the decrease is real but driven by shrinking ||h_i||^2, not by alignment with φ(q_i)), and the identification of the log-scale issue in the original plots. The eigenvalue comparison is the weakest leg. The manuscript defines the feature map as φ(k_j)=φ(k_j)/g(k_j) in Section 2, and Eq. 6 includes G=diag(1/g), so readers may infer the normalized Gram was used. But Section 3 just says \"calculate the Gram matrix K_φ, center it, and extract eigenvectors\" without stating whether the g-normalization is applied inside K_φ before centering. If the released code centers the raw kernel instead, both the 10^-6 eigenvalue scale and the low similarities could be artifacts. That detail is load-bearing and must be stated.\n\nThe internal inconsistency: Section 5 says Z-score standardization \"minimally impacts overall trends and conclusions,\" but Table 3 shows swings like ViT-Large max +181%, min -67%, DeiT-Small-D max +415%. That is not negligible. Either Table 3 is wrong or the claim is wrong; right now the text and table disagree.\n\nAlso, no null baselines for the similarity metrics. MOC 0.32 at best is only meaningful if we know what random directions or shuffled columns produce. And the conclusion overreaches: \"lacks empirical and theoretical robustness\"—the paper provides empirical counterevidence, not a theoretical refutation.\n\nWho this is for: anyone working on kernel interpretations of attention, or building on Teo and Nguyen. It deserves referee time, but the referee should ask for a precise statement of the Gram construction, null baselines, and a toned-down conclusion. Worth engaging.","headline":"A broad reproduction study with a plausible negative result, but the eigenvalue refutation hinges on an undocumented normalization choice and the conclusion overreaches.","tokens_in":14915,"tokens_out":4611,"would_cite":true,"duration_ms":41088,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across ten pretrained transformers, the claim that self-attention performs kernel PCA fails to reproduce.","keywords":["self-attention","kernel PCA","reproduction study","transformers","interpretability","Gram matrix eigenvalues","projection loss","Centered Kernel Alignment"],"falsifier":"Run an unmodified implementation of the original spectral computation on one pretrained vision-transformer head and record the maximum absolute eigenvalue of the centered Gram matrix: if it is near 648 the original claim survives this test, and if it is below $10^{-5}$ the reproduction stands; repeat the value-matrix comparison with and without the $g(k_j)$ scaling to check whether the reported mismatch is robust.","tokens_in":13836,"feed_emoji":"🔍","tokens_out":8552,"duration_ms":74837,"temperature":0.7,"pith_summary":"This paper re-tests the recent claim that a transformer's self-attention head is secretly doing kernel principal component analysis (KPCA): that learned value vectors encode the eigenvectors of the key Gram matrix and attention outputs are projections of queries onto principal-component axes in a feature space. Across ten pretrained vision transformers and nine encoder-only language models, the authors find almost no correspondence between learned value vectors and the KPCA-predicted value matrix under four similarity measures, with best cosine similarity reaching only 0.32 on vision models. They also argue that the decreasing projection loss reported as evidence mostly reflects a mismatch in scale: the two squared norms involved differ by orders of magnitude, so the apparent minimization tracks shrinking output norms rather than genuine alignment. Finally, they show that the Gram-matrix eigenvalue statistics that anchored the original argument cannot be reproduced without undocumented implementation-specific adjustments. If correct, the paper removes an appealing bridge between attention and classical kernel methods, while leaving open weaker descriptive kernel views.","feed_headline":"Kernel PCA view of self-attention fails a reproduction test","feed_subtitle":"Ten vision and nine language transformers show low value-matrix alignment and irreproducible eigenvalue claims.","key_machinery":"The load-bearing objects are the centered Gram matrix $\\tilde{K}_\\varphi(i,j)=\\tilde{\\varphi}(k_i)^\\top\\tilde{\\varphi}(k_j)$, built from key vectors through the feature map $\\varphi(k_j)=\\phi(k_j)/g(k_j)$ with per-key scaling $g$, and the resulting KPCA value matrix $\\dot{V}_{\\text{KPCA}}=GA-G\\mathbf{1}_N A$, where $G=\\operatorname{diag}(1/g(k_j))$, $\\mathbf{1}_N$ is the all-$1/N$ matrix, and $A$ holds the top $d_v$ eigenvectors of $\\tilde{K}_\\varphi$. The hypothesis under test is $V\\approx\\dot{V}_{\\text{KPCA}}$, and the paper measures it with four scale-normalized similarity metrics: direct cosine similarity, optimally matched cosine similarity, linear CKA, and kernel CKA. For the projection-loss claim, the key algebraic step is expanding $J_{\\text{proj}}$ into the squared norms $\\|\\varphi(q_i)\\|^2$ and $\\|h_i\\|^2$ together with a cross-term that depends on eigenvector assignment; this decomposition shows the loss is dominated by the output norm. For the spectral claim, the paper examines the actual eigenvalues of $\\tilde{K}_\\varphi$ and demonstrates that small absolute differences in the ratio vector $\\gamma$ do not certify the eigenvector condition when the eigenvalues themselves are tiny.","core_discovery":"The paper's central claim is that the KPCA interpretation of self-attention collapses under independent empirical scrutiny. Using pretrained vision transformers and encoder-only language models, the authors test the three empirical pillars of that interpretation. First, the learned value matrix $V$ does not approximate the KPCA value matrix $\\dot{V}_{\\text{KPCA}}$: even after optimally matching columns one-to-one, the best cosine similarity reaches only 0.32 on vision models (0.57 on one small language encoder), while linear CKA stays at or below 0.13. Second, the reported decrease in the projection loss $J_{\\text{proj}}$ is an artifact of scale: the squared feature-map norm $\\|\\varphi(q_i)\\|^2$ is roughly $10^3$ times smaller than the output norm $\\|h_i\\|^2$, so the plotted decrease tracks shrinking output magnitudes rather than convergence of query features to their reconstruction. Third, the Gram-matrix eigenvalue statistics reported as spectral confirmation are not reproducible; computing the same quantities across ten models gives eigenvalues on the order of $10^{-6}$ rather than the reported values around hundreds, and the ratio-difference plots used in the original argument can look convincing even for random directions. The paper concludes that this specific derivation of self-attention as kernel PCA lacks empirical and theoretical support, while explicitly not rejecting robust PCA as an algorithm and not ruling out weaker, descriptive kernel views of attention.","pith_inferences":["A natural next test would be to train a small transformer with an added penalty $\\|V-\\dot{V}_{\\text{KPCA}}\\|$; if the KPCA geometry were actually favorable for attention, this penalty should be easy to satisfy and should preserve task accuracy, which would separate 'not spontaneously happening' from 'not realizable'.","The eigenvalue gap (reported values around hundreds versus reproduced values near $10^{-6}$) suggests the original spectral figures may have come from a different, undocumented preprocessing of the keys; the reproduction's own standardization experiments shift the values substantially but still do not reach the reported scale, leaving the original numbers unexplained.","The paper's negative result shifts the burden of proof for future interpretability claims: a kernel interpretation of a learned mechanism should come with exact, reproducible spectral statistics and a scale-invariant comparison to the actual learned weights, not ratio plots or log-scale curves.","The discussion of an explicit variance and orthonormality regularizer hints at a constructive way forward: kernel interpretations of attention may be better treated as training objectives that enforce low-rank structure than as properties that emerge spontaneously."],"forward_implications":["The proposed equivalence between learned value vectors and the KPCA value matrix should be treated as empirically unsupported; four scale-invariant similarity measures all leave the correspondence below anything that would indicate convergence.","The decreasing projection-loss curves should not be read as evidence of KPCA alignment; the squared norms of the two reconstructed sides differ by orders of magnitude, so the loss mostly measures output-norm shrinkage.","The $\\gamma$-difference plots are not a safe diagnostic: when Gram-matrix eigenvalues are around $10^{-6}$, nearly zero absolute differences can arise for directions that are not eigenvectors at all.","The original spectral claim is implementation-sensitive and needs explicit, reproducible eigenvalue computation before it can support the KPCA story; the reported magnitudes do not reappear in an independent run.","The negative result is limited to this derivation: descriptive kernel and projection views of attention, and algorithms that explicitly regularize toward singular-vector alignment, remain consistent with the evidence."],"supporting_citations":[{"why":"The original KPCA interpretation of self-attention under test; supplies the value-matrix construction, projection-loss argument, and eigenvalue statistics the paper re-examines.","marker":"(Teo and Nguyen, 2024)"},{"why":"Supplies the DeiT vision transformers used for the value-matrix similarity and eigenvalue experiments.","marker":"(Touvron et al., 2021)"},{"why":"Supplies the ViT vision transformers used as test architectures.","marker":"(Dosovitskiy et al., 2021)"},{"why":"Supplies the CKA similarity metrics used to compare learned value matrices with KPCA predictions.","marker":"(Kornblith et al., 2019)"},{"why":"Supplies the optimal one-to-one matching algorithm used to compute the best-case cosine similarity.","marker":"(Crouse, 2016)"},{"why":"Supplies ImageNet-1K, the dataset from which the test images are drawn.","marker":"(Russakovsky et al., 2015)"},{"why":"Contextualizes the negative result by describing an asymmetric kernel-SVD view in which alignment is enforced by an explicit regularizer rather than assumed to emerge.","marker":"(Chen et al., 2023)"}],"fun_headline_variants":["Self-attention as KPCA fails reproduction test","KPCA interpretation of attention unsupported by data","Reproduction study discredits KPCA self-attention claim","Self-attention doesn't match kernel PCA predictions","KPCA view of self-attention lacks empirical support"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The refutation assumes the centered Gram matrix was built with the same per-key scaling $g(k_j)$ and feature map as the original derivation; if that normalization was omitted or misapplied, the tiny eigenvalues and low similarity scores could be artifacts rather than genuine refutations.","fun_headline_variants_meta":{"raw":{"variants":["Self-attention as KPCA fails reproduction test","KPCA interpretation of attention unsupported by data","Reproduction study discredits KPCA self-attention claim","Self-attention doesn't match kernel PCA predictions","KPCA view of self-attention lacks empirical support"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1494,"prompt_tokens":1073,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":689,"tokens_out":421,"duration_ms":3910,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:15:51.784553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an unmodified implementation of the original spectral computation on one pretrained vision-transformer head and record the maximum absolute eigenvalue of the centered Gram matrix: if it is near 648 the original claim survives this test, and if it is below $10^{-5}$ the reproduction stands; repeat the value-matrix comparison with and without the $g(k_j)$ scaling to check whether the reported mismatch is robust.","supporting_citations":[],"review_version":1}