{"id":"fd6ef994-fd06-4554-8635-63dc110cd334","arxiv_id":"2507.21040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Transformers can be viewed as unrolled inference in a probabilistic Laplacian Eigenmaps model, and replacing the attention matrix by attention minus identity improves validation performance.","lead":"This paper interprets transformer attention as graph diffusion and proposes subtracting the identity matrix from the attention matrix as a principled modification. The change improves performance on a small language model and a vision transformer, and slightly speeds up GPT-2 training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The derivation's gradient is algebraically wrong: with a row-stochastic softmax attention matrix, L = I - A is non-symmetric, so d tr(L X X^T)/dX = (L + L^T)X, not 2 L X; the A-I update is not the ELBO gradient.","rationale":"The paper's central claim has two parts: an interpretation of transformer blocks as unrolled inference in probabilistic Laplacian eigenmaps, and an architectural A-I change that follows from that interpretation. The second part is only as strong as the first: the paper says 'our derivation shows' the graph Laplacian arises. The weakest point in that derivation is not just the explicitly acknowledged stop-grad approximation; it is that the matrix treated as a graph Laplacian is not symmetric. The Wishart distribution used for q(Gamma) requires a symmetric positive-definite scale matrix, and the trace derivative used to produce the update is only correct for symmetric L. This is an internal inconsistency that can be checked by direct differentiation, independent of any empirical results. If the test confirms the exact gradient is (L + L^T)X, then the A-I update is not a gradient step of the stated objective, and the probabilistic interpretation as written collapses. The paper's small-scale empirical gains may still justify A-I as a heuristic, but they do not rescue the central 'so' connecting the theory to the modification. I therefore recommend rejecting or substantially revising the paper's theoretical claim, while acknowledging the empirical observation may be worth a cleaner study.","tokens_in":6614,"tokens_out":16507,"duration_ms":188647,"concrete_test":"Take a small concrete instance (n=8, dq=4, kappa=1, beta=1), define A = softmax(X X^T) row-wise, L = I - A, and f = tr(L (X X^T + beta I)). Compute the exact gradient df/dX by automatic differentiation and compare it with the paper's formula 2 L X and with (L + L^T)X. Also record max |L - L^T| and the minimum eigenvalue of (L + L^T)/2. If the exact gradient equals (L + L^T)X to numerical precision and differs from 2 L X by more than a few percent, or if the symmetrized Laplacian has a negative eigenvalue, the derivation's update is invalid and the A-I modification is not the ELBO gradient.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3 defines the variational constraint using a graph Laplacian L = I - A, with A = softmax(kappa Z Z^T - M) applied row-wise. This object is not a graph Laplacian in the sense required by the ProbDR Wishart model: it is non-symmetric (softmax row normalization breaks symmetry) and generally not positive semidefinite, so Wishart(Gamma | L, d) is not a valid distribution. More importantly, the gradient step that produces A-I is computed as d(ELBO)/dX = 2 L X. For a constant but non-symmetric matrix L, the correct derivative of tr(L X X^T) is (L + L^T)X, not 2 L X. Therefore X <- X + 2 eta (A - I)X is not the gradient-descent step of the stated ELBO, even if one grants the paper's explicit stop-grad treatment of the variational constraint. The correct update would involve L + L^T and would not reduce to the implemented A-I modification. The empirical A-I results may still be true, but the paper's central interpretive claim — that the A-I form follows from unrolled inference — is not supported by the derivation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probabilistic interpretation of transformer blocks as unrolled inference steps in a probabilistic Laplacian Eigenmaps model from the ProbDR framework. The authors argue that, at initialization, a transformer block corresponds to gradient descent on a variational lower bound, and that this derivation yields a graph Laplacian term A-I in place of the standard attention matrix A. They test the A-I modification on a language model and a vision transformer, reporting improved validation performance and faster convergence. The manuscript also includes a qualitative experiment showing that a randomly initialized transformer clusters MNIST digits in its latent space.","tokens_in":6947,"tokens_out":5083,"duration_ms":60742,"significance":"If the derivation were correct, the paper would offer a novel probabilistic grounding for the transformer architecture and a simple, broadly applicable architectural modification with empirical support. The paper is transparent about two key approximations (the stop-grad treatment of the variational constraint and the replacement of the regularizer gradient by a linear term), which is commendable. The empirical finding that subtracting the identity from the attention matrix improves performance is interesting even independently of the theory, and the MNIST clustering experiment provides a concrete, falsifiable illustration. However, the central derivation contains a load-bearing algebraic error regarding the gradient of a non-symmetric quadratic form, and the experiments lack statistical rigor (no seeds, no error bars). The paper does not provide code or machine-checked proofs, so the empirical claims rest on the reported figures alone.","major_comments":[{"comment":"The equation dL/dX = 2 L X is incorrect for the non-symmetric matrix L = I - A. Because A = softmax(κ Z Z^T - M) is applied row-wise, A is generally not symmetric, so L is not symmetric. For a constant non-symmetric L, the derivative of tr(L X X^T) with respect to X is (L + L^T)X, not 2 L X. Consequently, the update X <- X + 2η(A - I)X is not the gradient-descent step of the stated ELBO, even under the paper's explicit stop-grad treatment. The correct update would involve (A + A^T)/2 - I, which is not what is implemented or tested in the experiments. This error undermines the central claim that the A-I form follows from unrolled inference.","section":"Section 3, derivative of Ldata"},{"comment":"The matrix L = I - A with row-stochastic softmax A is not a graph Laplacian in the sense required by the ProbDR model: it is non-symmetric and generally not positive semidefinite. A Wishart distribution W(Γ | L, d) is defined only for a positive definite scale matrix, so the variational distribution q(Γ, Z) = W(Γ | L(Z), d) is not a valid distribution for arbitrary Z. The paper should either symmetrize the Laplacian (e.g., L_sym = I - (A + A^T)/2) or justify the use of a non-PSD scale matrix. This is load-bearing because the entire ELBO simplification depends on the Wishart form.","section":"Section 3, variational constraint and Wishart validity"},{"comment":"The paper states, 'we treat the variational constraint as an observed random variable, and hence do not account for gradient updates to terms leading from the variational constraint.' This is a significant and nonstandard assumption. If the dependence of A on X were included in the gradient, the update would contain additional terms and the clean A-I form would not follow. The SimSiam analogy is not a sufficient justification, since stop-grad in SimSiam is an algorithmic design choice rather than a consequence of the probabilistic model. The assumption should be empirically tested, for example by comparing the proposed update with the full-gradient update on a small task.","section":"Section 3, stop-grad on the variational constraint"},{"comment":"The replacement dLreg/dX = 2(XX^T + βI)^{-1}X ≈ 2/(dq + β) X is stated without derivation or numerical verification. This approximation is used to argue that the feed-forward block corresponds to a gradient step, but the error in this approximation could be large, especially at initialization when XX^T may not be close to a scaled identity. The paper should either justify this approximation analytically (e.g., under the assumed initialization) or demonstrate empirically that it does not affect the qualitative conclusions.","section":"Section 3, approximation of the regularizer gradient"},{"comment":"The empirical claims rest on single runs with no reported seeds, error bars, or statistical significance. Section 4.2 says the model was 'run multiple times' but does not report the number of runs or the variance; Figure 3 appears to show a single comparison between two runs. Given the small performance differences, the claim that the A-I modification 'consistently' improves performance is not supported. The authors should report multiple seeds, mean and variance, and ideally a significance test for the language and vision experiments.","section":"Section 4, experimental methodology"}],"minor_comments":[{"comment":"The claim that 'transformer blocks cluster points in the latent space' is supported only by a qualitative plot of the first two latent dimensions. The paper should provide a quantitative clustering metric (e.g., silhouette score or purity) to make the dimensionality-reduction claim more precise.","section":"Section 4.1, Figure 1"},{"comment":"The sentence 'The query and key matrices were set up such that the attention matrix, pre-normalisation, has a diagonal equal to κ' is ambiguous: the softmax output is non-negative, but the pre-softmax scores have diagonal κ. Please clarify whether κ refers to the pre-softmax or post-softmax diagonal.","section":"Section 4.1, initialization details"},{"comment":"The notation is inconsistent: sometimes the graph Laplacian is written as L(Y) based on data Y, and later as L(Z) based on latent Z. Please use distinct symbols or clarify the dependence to avoid confusion.","section":"Section 3, notation"},{"comment":"The manuscript refers to 'the element highlighted in red' in equations, but the color is not visible in a black-and-white printout. Please use a symbol or footnote instead.","section":"General, color-dependent text"}],"recommendation":"major_revision","confidential_remarks":"The paper's central theoretical derivation is flawed in a way that directly affects the claimed A-I update: the gradient of tr(L X X^T) is not 2 L X when L is non-symmetric, and L = I - A is not a valid Wishart scale matrix. That said, the empirical observation that A-I improves performance is independent of the derivation and could still be valuable. I recommend major revision rather than rejection because the authors could potentially re-derive the update with a symmetrized Laplacian or reposition the paper as an empirical study with a heuristic motivation. However, if the derivation cannot be repaired, the theoretical claims should be withdrawn or substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: the empirical A-I trick looks worth testing, but the derivation that motivates it has a real algebraic error. With row-softmax attention, L = I - A is not symmetric, so d tr(L X X^T)/dX is (L+L^T)X, not 2LX. The paper's update X <- X + 2 eta (A-I)X is therefore not the gradient step of the stated ELBO, even if you grant the stop-grad treatment.\n\nThat said, the paper does something useful. It gives a fresh interpretation of transformer blocks as unrolled inference in a probabilistic Laplacian Eigenmaps model, distinct from Yu et al.'s Gaussian mixture view. The proposed change—subtracting the identity from the attention matrix—is simple, novel, and shows consistent gains across two data modalities (Shakespeare and downsampled ImageNet) plus faster GPT-2 convergence. Those are real empirical observations, and the authors are transparent about their approximations.\n\nThe soft spots are not minor. The non-symmetric L also breaks the Wishart setup: a Wishart scale must be symmetric and PSD, and a row-stochastic softmax adjacency is neither. So the probabilistic model isn't well-defined. The regularizer gradient approximation ((XX^T+beta I)^{-1}X ~= cX) is crude but stated; that's a lesser concern. The experiments also lack error bars and multiple seeds, as the reader noted, so the effect size is uncertain.\n\nWho is this for? People thinking about transformer interpretability via probabilistic dimensionality reduction, and anyone considering the A-I attention change. The interpretation as written does not hold, but the empirical puzzle is interesting enough to deserve a careful follow-up. I wouldn't cite the interpretative claim, but I might cite the empirical observation if it survives better-controlled tests.\n\nRecommendation: send to peer review with a serious referee. The paper has a load-bearing flaw, but the question and the empirical result warrant expert scrutiny and a path to revision. My own reading-group interest is a maybe—good as a case study in how derivations can go sideways.\n\nBest","headline":"The A-I attention tweak is worth a serious test, but the paper's derivation has a load-bearing flaw: with row-softmax, I-A is non-symmetric, so the A-I update is not the ELBO gradient.","tokens_in":7403,"tokens_out":4516,"would_cite":false,"duration_ms":48639,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transformer blocks can be read as unrolled inference in a probabilistic Laplacian Eigenmaps model, so the attention matrix should really be a graph Laplacian $A-I$, and swapping it in improves validation performance and training speed.","keywords":["transformers","probabilistic Laplacian Eigenmaps","unrolled inference","graph diffusion","attention mechanism","variational inference","dimensionality reduction","Laplacian smoothing"],"falsifier":"Compute or numerically approximate the full gradient of the variational objective including the derivative of $\\sigma(\\kappa XX^T-M)$ with respect to $X$ (without stop-gradient) and check whether the resulting update is still proportional to $(A-I)X$ plus only negligible corrections. If the extra terms are not negligible, the paper's central interpretive claim fails at its stated assumption.","tokens_in":6419,"feed_emoji":"🧠","tokens_out":6027,"duration_ms":57177,"temperature":0.7,"pith_summary":"This paper tries to show that a transformer block is not just analogous to but actually an unrolled optimisation step for a probabilistic Laplacian Eigenmaps model, a latent-variable dimensionality-reduction model. On that reading, the softmax attention matrix should be treated as a graph adjacency matrix, and the correct update inside attention is a graph diffusion step, i.e. $A-I$ rather than $A$. The authors derive this form from a variational lower bound and then test the architectural consequence: subtracting the identity from the attention matrix improves validation performance on a small language model and on a vision transformer, and speeds up GPT-2 training. The reason to care is that the derivation turns a heuristic attention design into a concrete inference prediction with a simple, testable modification.","feed_headline":"Graph diffusion beats plain attention in transformer tests","feed_subtitle":"A new reading of transformers as unrolled Laplacian Eigenmaps inference says attention should smooth, not just attend; tests confirm it.","key_machinery":"The load-bearing object is the variational Laplacian Eigenmaps objective $KL(q(\\Gamma,Z)\\|p(\\Gamma,Z)) \\propto \\mathrm{tr}(\\tilde L(XX^T+\\beta I)) - \\log\\det(XX^T+\\beta I)$, with the variational constraint $\\tilde L = I - \\sigma(\\kappa ZZ^T - M)$ treated as an observed random variable (stop-gradient). Gradient descent on the first term produces the update $X \\leftarrow X + 2\\eta(\\sigma(\\kappa XX^T - \\beta I - M)-I)X$, whose parenthesised factor is the graph Laplacian $A-I$; the second term plus a uniform-matrix prior yields the linear feed-forward update and LayerNorm projection. This $A-I$ identity is what carries the argument: it turns the attention matrix into a diffusion operator and is precisely the modification tested in the experiments.","core_discovery":"At initialisation, with weights set to diagonal matrices, a transformer block performs gradient descent on the KL divergence between a variational Wishart distribution whose constraint is a soft nearest-neighbour graph Laplacian $\\tilde L = I - \\sigma(\\kappa ZZ^T - M)$ and a model Wishart distribution over the same precision matrix. Because the derivative of the data term is $d\\,\\mathrm{tr}(\\tilde L(XX^T+\\beta I))/dX = 2\\tilde L X = 2(A-I)X$, the attention operation should be the negative graph Laplacian $A-I$, i.e. a Laplacian-smoothing or graph-diffusion step, not the raw attention matrix. The paper then shows empirically that replacing the attention matrix by $A-I$ improves validation performance on Shakespeare-character and downsampled ImageNet transformers and makes GPT-2 training converge faster.","pith_inferences":["If the $A-I$ result is robust, it suggests that part of attention's value is implicit Laplacian smoothing, so other graph-regularisation terms (degree normalisation, higher-order diffusion) may yield further gains; this is not tested in the paper.","The stop-gradient assumption is the likely breaking point: relaxing it would couple attention to the update and may change the architecture, so a natural extension is to train with the full gradient and compare.","The derivation may transfer to other attention-based architectures (e.g. cross-attention, multi-head with product-of-experts) even though the paper only analyses single-head self-attention at initialisation.","The MNIST clustering result suggests a possible diagnostic: check whether the $A-I$ modification tightens class-cluster structure in deeper layers; the paper does not report this."],"forward_implications":["If transformer blocks are unrolled Laplacian Eigenmaps inference, then at initialisation a randomly initialised transformer performs linear dimensionality reduction and clustering of token or patch representations, as shown on MNIST.","The attention matrix should be read as an adjacency matrix, and graph diffusion ($A-I$) is a principled replacement for raw attention; the paper reports consistent validation gains on language and vision.","Training a GPT-2-scale model with the $A-I$ modification converges slightly faster than the vanilla attention run.","The weight matrices in a transformer can be interpreted as rotation and learning-rate (step-size) parameters, giving a concrete 'learn to optimise' story for the architecture.","The same derivation can be extended to kernelised dimensionality-reduction objectives, which the authors suggest as future work; if that holds, further non-linear graph-diffusion updates may improve low-dimensional models."],"supporting_citations":[{"why":"Supplies the probabilistic Laplacian Eigenmaps model and its variational inference objective that the paper reuses and extends.","marker":"Ravuri et al. (2023)"},{"why":"Provides the unrolled-optimisation reading of transformer blocks and the derivation of the LayerNorm and feed-forward steps.","marker":"Yu et al. (2023)"},{"why":"Defines the transformer and attention baseline that the paper reinterprets and modifies.","marker":"Vaswani et al. (2017)"},{"why":"Motivates graph-Laplacian updates from spectral graph convolution; the paper's update parallels $\\theta_0 x + \\theta_1 Lx$.","marker":"Kipf & Welling (2017)"},{"why":"Prior evidence that graph convolutions enrich transformer attention, which the paper's graph-diffusion modification extends.","marker":"Choi et al. (2024)"},{"why":"Provides the nanoGPT code base used for the language-model and GPT-2 training experiments.","marker":"Karpathy (2022)"},{"why":"Provides the downsampled ImageNet benchmark and baseline accuracy used in the vision experiment.","marker":"Chrabaszcz et al. (2017)"}],"fun_headline_variants":["Attention should be diffusion, not just attention","Graph Laplacian interpretation of transformers improves them","Attention minus identity beats plain attention in tests","New probabilistic view: attention is graph diffusion","Transformers as unrolled Laplacian inference, tests profit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation treats the softmax attention matrix as a fixed, observed object when computing gradients, so the dependence of the attention matrix on the latent vectors $X$ is ignored; if that dependence were included, the update would acquire extra terms and the clean $A-I$ form would no longer follow.","fun_headline_variants_meta":{"raw":{"variants":["Attention should be diffusion, not just attention","Graph Laplacian interpretation of transformers improves them","Attention minus identity beats plain attention in tests","New probabilistic view: attention is graph diffusion","Transformers as unrolled Laplacian inference, tests profit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2699,"prompt_tokens":814,"completion_tokens":1885,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1814}},"tokens_in":430,"tokens_out":1885,"duration_ms":13864,"temperature":1.0,"reasoning_tokens":1814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:00:01.684461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute or numerically approximate the full gradient of the variational objective including the derivative of $\\sigma(\\kappa XX^T-M)$ with respect to $X$ (without stop-gradient) and check whether the resulting update is still proportional to $(A-I)X$ plus only negligible corrections. If the extra terms are not negligible, the paper's central interpretive claim fails at its stated assumption.","supporting_citations":[],"review_version":1}