Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Attention's forward pass and Frank-Wolfe

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The zero-temperature limit of self-attention is a Frank–Wolfe method, and its geometry (Voronoi cells, stationary vertices, exponential metastability) explains how token clusters form before the eventual collapse to a single point.

desk verdict The hardmax/Frank–Wolfe analysis is clean and new, but the abstract's headline claim about approximating finite-β attention for exponential times is proven for a stochastic chain, not for the deterministic softmax update. read the letter →

arxiv 2508.09628 v1 pith:3H5NSZBV submitted 2025-08-13 math.OC

Albert Alcalde , Borjan Geshkovski , Domènec Ruiz-Balet This is my paper
classification math.OC MSC 68T0790C2560J10
keywords self-attentiondynamicsFrank–WolfemethodhardmaxlimitVoronoidiagrammetastabilityconvexhullverticesMarkovchaininversetemperature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In the zero-temperature ($\beta \to +\infty$) limit, a single-head softmax self-attention update becomes a Frank–Wolfe step: each token moves a fraction $\gamma$ of the way toward the token that maximizes its attention score. The paper proves that when the key–query matrix is negative semidefinite, all tokens shrink to a single cluster at the origin, and when it is positive semidefinite, the convex hull of the tokens decomposes into Voronoi-like cells, each particle travels in a straight line to its cell’s vertex with super-exponential convergence, and the vertices stay fixed. It then argues that this hardmax picture describes what actually happens at finite temperature: particles first cluster near the vertices in a constant number of steps and remain there for exponentially long times (in the inverse temperature $\beta$) before the soft dynamics pull everything to one point.

What carries the argument

The central object is the hardmax Frank–Wolfe update (SA$\infty$) with step size $\gamma_t = h_t/(1+h_t)$, together with the cell decomposition $\mathcal{C}_i(v)$ induced by a positive-definite key–query matrix. The cells are convex, have pairwise disjoint interiors, and cover the token polytope; when the vertices lie on a common level set of the quadratic, they coincide with the $B$-norm Voronoi cells. The Markov chain (SAP) — where a token is sampled with probability proportional to $e^{\beta\langle x_i, x_j\rangle}$ — is the bridge that carries the hardmax geometry into finite-$\beta$ metastability, producing two time scales: fast clustering to vertices, then an exponentially long residen

What would settle it

Simulate (SA$\beta$) with $B=I$, small $\gamma$ and moderate $\beta$ for a fixed random initial configuration of $n$ points in $\mathbb{R}^d$; record the first exit time of all particles from an $\varepsilon$-neighborhood of their assigned convex-hull vertices. If this time does not scale like $e^{c\beta}$ — or if it scales but with a markedly smaller constant than the Markov-chain bound — the paper's headline approximation statement is falsified.

Watch

Extended reading notes

Core claim

Under the renormalized update (1.3) with $V_t = h_t I$, the $\beta \to +\infty$ limit is $x_i^{t+1} = x_i^t + \gamma_t (\arg\max_{y\in\mathcal{K}_t} \langle B_t x_i^t, y\rangle - x_i^t)$, where $\mathcal{K}_t$ is the convex hull of all tokens and $\gamma_t = h_t/(1+h_t)$. If $B_t$ is symmetric, this is exactly a Frank–Wolfe iteration on a quadratic. When $-B_t$ is positive semidefinite, standard Frank–Wolfe theory gives a linear contraction to the origin. When $B_t \equiv B \succ 0$ and the initial vertices satisfy a genericity condition, the cells $\mathcal{C}_i(v) = \{x\in\mathcal{K} : \langle Bx, v_i\rangle \ge \langle Bx, y\rangle \ \forall y\in\mathcal{K}\}$ form a partition of the conv

Load-bearing premise

The exponential-time approximation is proved for the Markov chain (SAP), where each particle randomly samples one token with probability equal to its attention weight, and the paper applies the conclusion to the deterministic softmax update (SA$\beta$) without proving a quantitative coupling between the two processes.

Editorial extensions

If this is right

  • For positive semidefinite key–query matrices, hardmax attention is exactly a projection-free walk to the convex hull's vertices; the token cloud's internal points are spectators while the vertices govern the dynamics.
  • At finite $\beta$ the same geometry predicts two time scales: $O(1)$ clustering to near-vertex states, then an exponentially long (in $\beta$) metastable plateau before the eventual full collapse to the origin.
  • The ODE limit of hardmax attention, which is discontinuous and outside classical Cauchy–Lipschitz theory, has unique solutions under the same vertex genericity conditions.
  • Because only the convex hull vertices matter, the cost of a forward pass could in principle drop from $O(n^2)$ to a hull-based count when the number of vertices is much smaller than the number of tokens.
  • When the key–query matrix is negative semidefinite, the update is a plain Frank–Wolfe method whose known convergence theory transfers directly, with the origin as the global attractor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Closing the gap between the Markov chain (SAP) and the deterministic softmax update (SA$\beta$) — via a coupling, concentration, or averaging argument — is the main technical step needed to make the abstract's approximation claim fully rigorous.
  • The paper's own remarks flag condition (5.1) as one it does not know how to remove; relaxing or replacing this acute-angle condition would extend the clustering theorem to polytopes with obtuse vertex geometry.
  • The Voronoi-cell picture yields a testable prediction for trained models: attention scores should concentrate on a small set of extreme tokens, so the effective number of attended tokens should track the number of convex-hull vertices rather than the context length.
  • The two-scale behavior (fast approach to vertices, exponentially slow escape) matches the slow-motion pattern of metastable gradient systems; transferring it to the mean-field limit $n \to +\infty$ would require a separate argument.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper analyzes the hardmax (β→∞) limit of single-head self-attention without MLP, after rescaling V_t=h_t I_d. It identifies the limit update (SA∞) with a Frank-Wolfe step for a quadratic objective with step-size γ_t. For negative semidefinite key-query it proves a Frank-Wolfe-type convergence bound; for positive definite it introduces a cell decomposition of the convex hull of tokens, shows that, under a vertex non-degeneracy assumption, tokens move along straight segments to their cell vertex, and uses this to prove well-posedness of a discontinuous ODE. For finite β, the paper defines a stochastic Markov chain (SAP) whose transition probabilities are the softmax attention weights, and proves two metastability theorems: a first constant-time clustering event and a second exponential-in-β confinement time. The abstract concludes that hardmax accurately approximates the finite-β process over exponentially long horizons.

Significance. If the claims were fully established, this would be a valuable contribution to the mathematical theory of transformers: the Voronoi-cell interpretation is elegant, the rates are explicit, the ODE well-posedness result addresses an open question from [GLPR25], and the Markov-chain metastability is a new quantitative result in discrete time. The code for the numerical illustrations is available, and no data fitting or parameter tuning is involved; the hardmax update is the exact singular limit. However, the advertised bridge to the deterministic finite-β dynamics is not proven: the metastability theorems are proved for the auxiliary stochastic process (SAP), not for the deterministic softmax iteration (SAβ).

major comments (3)
  1. [Abstract; §5.1–5.2, (SAP), (SAβ), Theorems 5.2–5.4] Theorems 5.2 and 5.4 are proved for the Markov chain (SAP) defined in §5.1, in which token i updates to (1−γ)x_i^t + γ x_j^t with probability equal to the softmax weight. The deterministic finite-β iteration (SAβ) is only the conditional expectation of one SAP step. The paper supplies no coupling, concentration, or ergodic argument transferring the metastability bounds from single sample paths of (SAP) to the mean path (SAβ). This is not cosmetic: a standard martingale argument gives |X_t − y_t| = O(γ√T) between a SAP path and the SAβ path over T steps, which for T=O(e^{cβ}) swamps the O(1)/O(β^{-1/4}) radii in Theorems 5.2–5.4 unless γ is exponentially small. The abstract's claim that 'hardmax dynamics accurately approximate the finite-β process over exponentially long time horizons' and the sentence at the end of §1.2 ('Thus (SA∞) can be seen as a valid approximation of (1.1) up to O(e
  2. [Appendix A.3, Claim 3] In Claim 3 (Appendix A.3), the displayed chain p^t_{i→i} ≥ 1/(1+(n−1)e^{βτ/4}) ≥ 1−n e^{−βτ/4} is algebraically invalid: the middle term is much smaller than the right-hand term, and its exponent has the wrong sign. From the preceding bound (A.11), ⟨y,y⟩−⟨y,z⟩ ≥ τ²/2 − 3d(K)β^{-1/4} − β^{-1/2}, one can only conclude p ≥ 1 − n exp(−β(τ²/2 − o(1))), not the stated e^{−βτ/4} rate. Since Claim 3 feeds directly into the binomial estimate (A.13) and hence into the probability bound of Theorem 5.2, the proof of Theorem 5.2 needs to be corrected and the affected estimates rechecked.
  3. [Theorem 3.1 and Theorem 4.2, §§3–4] The rate statements in Sections 3 and 4 are not supported by the displayed formulas. In Theorem 3.1, with γ_t=2/(t+2), the bound J_t(x^t_i) ≤ 2λ_max d(K)^2/(t+1) is O(1/t), so the phrase 'linear convergence' after the theorem is incorrect. In Theorem 4.2, the conclusion 'at least exponentially fast' is false for arbitrary γ_t∈(0,1): for γ_t=1/(t+1), the product ∏_{τ=0}^{t-1}(1−γ_τ) equals 1/(t+1), giving polynomial decay. The section title 'Super-exponential convergence to vertices' is therefore only valid under additional assumptions on γ_t (e.g., a uniform lower bound γ_t≥γ_* > 0). Please correct the rate claims or add the missing assumptions.
minor comments (5)
  1. [§3] The theorem uses γ_t=2/(t+2) but the paragraph after it says 'We fix γ_t=2/(t+1)'; please align the step-size notation.
  2. [Theorem 5.2 and Step 4.4 of Appendix A.3] The constant C in Theorem 5.2 is described as universal, but the proof in Step 4.4 defines C as a ratio of initial distances. Define C in the statement or remove 'universal'.
  3. [Theorem 5.4] The relabeling of points into balls B(v_i,Cτ) assumes these balls are disjoint; if not, a point may belong to several balls. Add a condition such as Cτ < (1/3)min_{i≠j}‖v_i−v_j‖.
  4. [Proposition 5.1] Proposition 5.1 is stated without proof; 'The proof follows mutatis mutandis from [GRRB24]' is not enough for a result used in the narrative. At minimum, specify the modifications.
  5. [Appendix A.3] There are typos ('β¯ 1/4' instead of β^{-1/4}) and a mismatch between the Bin(t,p) in (A.13) and Bin(t+1,p) in Claim 4; please clean these up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: hardmax limit is an algebraic identity and metastability proofs are self-contained; the SAP/SAβ gap is a proof gap, not a circular reduction.

full rationale

Walking the derivation chain, I find no circular step. (SA∞) is derived from (1.3) by the formal limit β→∞ and the choice V_t=h_t I_d; with γ_t=h_t/(1+h_t), it is an algebraic identity, not a fitted equivalence. The Frank–Wolfe interpretation in §2 is a rewriting of the linear oracle, and the negative-definite theorem is proved in A.1 by a standard descent estimate. The positive-definite theorem (4.2), the Voronoi cell analysis, and the well-posedness theorem (4.8) are proved directly from the geometry of the cells, without importing their conclusions from cited work. Section 5 proves metastability for the auxiliary Markov chain (SAP); the one-step conditional expectation of (SAP) is exactly (SAβ), but no coupling transfers the pathwise bounds to the deterministic process. This is a genuine proof gap, not circularity—the claimed approximation is not equivalent by construction to any fitted parameter. The many self-citations (e.g., [GLPR23], [GLPR25], [GKPR24], [GRRB24]) are used for background, motivation, or a non-central proposition; none is the mechanism forcing the headline theorems. I also flag two non-circular concerns: the sentence in §1.2 'Thus (SA∞) can be seen as a valid approximation of (1.1) up to O(e^β) steps' overstates what Theorems 5.2/5.4 establish, and Claim 3's displayed inequality 'p ≥ 1/(1+(n−1)e^{βτ/4}) ≥ 1−ne^{−βτ/4}' is algebraically inconsistent as written. These affect correctness, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper relies on a specific rescaled self-attention model, a scalar value matrix, symmetric key-query matrices, and the Markov-chain substitution for the finite-beta regime. No free parameters are fitted to data; the exponents used in the metastability statements are chosen for proof convenience and are not tuned.

assumptions (5)
  • domain assumption The layer-normalization proxy R_t = I_d + V_t is used instead of true layer normalization; the iteration (1.3) is taken as the model.
    Section 1.1. This rescaling is a proxy for layer normalization and changes the dynamics; all subsequent results apply to this rescaled model, not to the original softmax attention with standard layer norm.
  • domain assumption V_t = h_t I_d (scalar value matrix) for the main analysis, and later V_t = gamma I_d.
    Section 2.3 states the restriction because it is not clear how to extend the methods to general value matrices.
  • domain assumption B_t is symmetric and of fixed sign throughout; in Section 4 B_t = B positive definite, and in Section 5 B_t = I_d.
    The Frank-Wolfe interpretation requires symmetric B_t; the Voronoi cell theory uses positive definiteness.
  • ad hoc to paper The Markov chain (SAP) with softmax transition probabilities is taken as the finite-beta model; no quantitative relation to the deterministic update (SA-beta) is established.
    Section 5.1. This is the load-bearing modeling assumption for the metastability theorems.
  • domain assumption The hardmax argmax is generically a singleton (Lemma 2.1), relying on B_t invertible and initialization off a finite union of hyperplanes.
    Section 2.1. This allows the update to be written with a unique argmax.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attention's forward pass and Frank-Wolfe." pith.science (2026). https://pith.science/paper/3H5NSZBV

@misc{pith2026250809628,
  author       = {Pith},
  title        = {Pith review of: Attention's forward pass and Frank-Wolfe},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3H5NSZBV}},
  note         = {Machine review of arXiv:2508.09628}
}
abstract

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature ($\beta\to+\infty$) regime, and relate it to the finite-$\beta$ setting. In this limit, the update rule can be viewed as a Frank-Wolfe step for a quadratic objective over the convex hull of the current token embeddings. When the key-query matrix is negative semidefinite, the method linearly contracts all tokens to a single cluster at the origin. When it is positive semidefinite, extending the hardmax rule to the entire convex hull induces a Voronoi diagram: vertices are stationary, interior points remain in their initial cells, and each token moves along a straight line toward its cell's vertex, yielding (super-)exponential convergence. As a byproduct, we also establish well-posedness of the associated ODE limit in this regime. Returning to the finite-$\beta$ regime, we model self-attention dynamics as a Markov chain and prove dynamic metastability: with high probability, interior tokens reach near-vertex configurations in a constant number of steps and remain within a small neighborhood for times that grow exponentially in the inverse temperature $\beta$, before ultimately collapsing to the origin. Thus, the hardmax dynamics accurately approximate the finite-$\beta$ process over exponentially long time horizons.

Figures

Figures reproduced from arXiv: 2508.09628 by the authors.

Figure 1
Figure 1. Each panel shows the cells C𝑖(𝑣) (colored). Left to right, top to bottom: 𝜅 = 5, 7, 9, and 10. Axes are suppressed for visual clarity; all plots are rendered on the same met￾ric scale. Code available at https://github.com/borjanG/2025-transformers-frank-wolfe. Taking a convex combination, ⟨𝐵(𝜆𝑥1 + (1 − 𝜆)𝑥2), 𝑣𝑖⟩ = 𝜆⟨𝐵𝑥1, 𝑣𝑖⟩ + (1 − 𝜆)⟨𝐵𝑥2, 𝑣𝑖⟩ and similarly for ⟨𝐵(𝜆𝑥1 + (1 − 𝜆)𝑥2), 𝑦⟩. Thus, ⟨𝐵(𝜆𝑥1 + (1 − 𝜆)𝑥2), 𝑣𝑖… view at source ↗
Figure 2
Figure 2. Here every vertex lies on S 1 , so the cells coincide with the classical Voronoi partition in R 2 intersected with the polygon 𝒦. Because points of equal length compete only by direction, the cells are radially symmetric wedges truncated by the boundary of 𝒦. Panels correspond to the same values of 𝜅 as in [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Plot of the quadratic 𝑥 ↦→ 1 2 ⟨𝐵𝑥, 𝑥⟩ with 𝐵 = diag(1, 2), over the convex hull of 5 (left) and 7 (right) vertices. Remark 4.6 (Gaussian vertices in high dimension). Let 𝑣1, . . . , 𝑣𝜅 i.i.d. ∼ 𝒩 (0, 𝐼𝑑) and 𝐵 ≻ 0 with condition number bounded independently of 𝑑. Assume that 𝜅 is fixed while 𝑑 → +∞. Then, with probability tending to 1 as 𝑑 → +∞, for all 𝑖 ∈ J1, 𝜅K, ⟨𝐵𝑣𝑖 , 𝑣𝑖⟩ > ⟨𝐵𝑣𝑖 , 𝑣𝑗 ⟩ for all 𝑗 ̸= 𝑖, that is, … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The cone I𝑖(η) (left) and the eroded cone I𝑖(η) ⊖ 𝛿𝐵1 (right), for η = 0.05 and 𝛿 = 0.02. Given a convex polytope 𝒦 ⊂ R 𝑑 with vertices 𝑣 = {𝑣𝑖}𝑖∈J1,𝜅K , recall the definition of the cells C𝑖(𝑣) := {𝑥 ∈ 𝒦 : ⟨𝑣𝑖 − 𝑣𝑗 , 𝑥⟩ ⩾ 0 for all 𝑗 ∈ J1, 𝜅K} . Assuming that 𝑣𝑖 ∈ C𝑖(…

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Linear self-attention in 2D reduces to a second-harmonic Kuramoto model whose order parameter obeys one ODE; explicit matrix conditions yield clustering, rotating clusters, Hamiltonian oscillations, and bifurcations.

  2. Perceptrons and localization of attention's mean-field landscape

    cs.LG 2026-01 unverdicted novelty 7.0 of 10

    In the mean-field limit of attention with perceptron blocks, critical points of the energy landscape are generically atomic and localized on subsets of the unit sphere.

  3. Propagation of Chaos in Contextual Flow Maps

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Derives forward and backward propagation-of-chaos bounds for finite vs. infinite-context transformers modeled as contextual flow maps, achieving Wasserstein rate n^{-1/d} generally and n^{-1/2} for transformer-like cases.

  4. Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

    math.AP 2026-05 unverdicted novelty 6.0 of 10

    In the low-temperature regime, the token distribution in mean-field transformers concentrates onto the push-forward under a key-query-value projection with Wasserstein distance scaling as √(log(β+1)/β) exp(Ct) + exp(-ct).

  5. Physics- and geometry-aware spatio-spectral graph neural operator for time-independent and time-dependent PDEs

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    A submission whose abstract describes a new graph neural operator for PDEs but whose full text is a different paper, leaving the claimed method and results unverifiable.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    Cluster- ing in pure-attention hardmax transformers and its role in sentiment analysis

    [AFZ24] Albert Alcalde, Giovanni Fantuzzi, and Enrique Zuazua. Cluster- ing in pure-attention hardmax transformers and its role in sentiment analysis. arXiv preprint arXiv:2407.01602,

    arXiv

  2. [3]

    Why do LLMs attend to the first token? arXiv preprint arXiv:2504.02732,

    [BAG+25] Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do LLMs attend to the first token? arXiv preprint arXiv:2504.02732,

    arXiv

  3. [8]

    What does bert look at? an analysis of bert’s attention

    [CKLM19] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341,

    arXiv 1906

  4. [10]

    Quantitative clustering in mean-field transformer models

    47 [CLPR25] Shi Chen, Zhengjiang Lin, Yury Polyanskiy, and Philippe Rigollet. Quantitative clustering in mean-field transformer models. arXiv preprint arXiv:2504.14697,

    arXiv

  5. [11]

    Geometric dynamics of signal propagation predict trainability of transformers

    [CNQG24] Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi, and Surya Ganguli. Geometric dynamics of signal propagation predict trainability of transformers. arXiv preprint arXiv:2403.02579,

    arXiv

  6. [12]

    Synchronization on circles and spheres with non- linear interactions

    [CRMB24] Christopher Criscitiello, Quentin Rebjock, Andrew D McRae, and Nicolas Boumal. Synchronization on circles and spheres with non- linear interactions. arXiv preprint arXiv:2405.18273,

    arXiv

  7. [13]

    Setting the record straight on transformer oversmoothing

    [DBK24] Gbètondji JS Dovonon, Michael M Bronstein, and Matt J Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.04301,

    arXiv

  8. [15]

    Hashattention: Semantic spar- sity for faster inference

    [DYC+24] Aditya Desai, Shuo Yang, Alejandro Cuadron, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Hashattention: Semantic spar- sity for faster inference. arXiv preprint arXiv:2412.14468,

    arXiv

Show all 37 references
  1. [16]

    Large-time asymptotics in deep learning

    [EGPZ20] Carlos Esteve, Borjan Geshkovski, Dario Pighin, and Enrique Zuazua. Large-time asymptotics in deep learning. arXiv preprint arXiv:2008.02491,

    2008 arXiv

  2. [18]

    Dynamic metastability in the self-attention model

    [GKPR24] Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attention model. arXiv preprint arXiv:2410.06833,

    arXiv

  3. [19]

    When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781,

    [GPD+24] Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781,

    arXiv

  4. [20]

    Measure-to-measure interpolation using Transformers

    [GRRB24] Borjan Geshkovski, Philippe Rigollet, and Domènec Ruiz-Balet. Measure-to-measure interpolation using Transformers. arXiv preprint arXiv:2411.04551,

    arXiv

  5. [21]

    On the num- ber of modes of Gaussian kernel density estimators

    [GRS24] Borjan Geshkovski, Philippe Rigollet, and Yihang Sun. On the num- ber of modes of Gaussian kernel density estimators. arXiv preprint arXiv:2412.09080,

  6. [23]

    Ot-transformer: a continuous-time transformer architecture with optimal transport regularization

    [KLO25] Kelvin Kan, Xingjian Li, and Stanley Osher. Ot-transformer: a continuous-time transformer architecture with optimal transport regularization. arXiv preprint arXiv:2501.18793,

    arXiv

  7. [24]

    Convergence rate of Frank-Wolfe for non- convex objectives

    [LJ16] Simon Lacoste-Julien. Convergence rate of Frank-Wolfe for non- convex objectives. arXiv preprint arXiv:1607.00345,

    arXiv

  8. [26]

    The sparse frontier: Sparse atten- tion trade-offs in transformer llms

    [NLH+25] Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M Ponti. The sparse frontier: Sparse atten- tion trade-offs in transformer llms. arXiv preprint arXiv:2504.17768 ,

    arXiv

  9. [29]

    Revisiting over-smoothing in bert from the perspective of graph

    [SGX+22] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08625 ,

    arXiv

  10. [30]

    Exploiting sparsity for long context inference: Million token contexts on commodity gpus

    [SHK+25] Ryan Synk, Monte Hoover, John Kirchenbauer, Neel Jain, Alex Stein, Manli Shu, Josue Melendez Sanchez, Ramani Duraiswami, and Tom Goldstein. Exploiting sparsity for long context inference: Million token contexts on commodity gpus. arXiv preprint arXiv:2502.06766,

    arXiv

  11. [31]

    Prefixing attention sinks can mitigate activation outliers for large language model quantization

    [SPH+24] Seungwoo Son, Wonpyo Park, Woohyun Han, Kyuyeun Kim, and Jaeho Lee. Prefixing attention sinks can mitigate activation outliers for large language model quantization. arXiv preprint arXiv:2406.12016,

    arXiv

  12. [32]

    Solutions of stationary McKean-Vlasov equation on a high-dimensional sphere and other Riemannian manifolds

    [SS24] Anna Shalova and André Schlichting. Solutions of stationary McKean-Vlasov equation on a high-dimensional sphere and other Riemannian manifolds. arXiv preprint arXiv:2412.14813,

    arXiv

  13. [33]

    Residual connections and normalization can provably pre- vent oversmoothing in gnns

    51 [SWJS24] Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normalization can provably pre- vent oversmoothing in gnns. arXiv preprint arXiv:2406.02997,

  14. [34]

    Recurrent self-attention dy- namics: An energy-agnostic perspective from Jacobians

    [TK25] Akiyoshi Tomihari and Ryo Karakida. Recurrent self-attention dy- namics: An energy-agnostic perspective from Jacobians. arXiv preprint arXiv:2505.19458,

  15. [36]

    Transformer-based causal language models perform clustering

    [WV24] Xinbo Wu and Lav R Varshney. Transformer-based causal language models perform clustering. arXiv preprint arXiv:2402.12151,

    arXiv

  16. [37]

    Learning Gaussian Mixture Models via Transformer Mea- sure Flows

    52 [ZKPR25] Aleksandr Zimin, Anastasiia Kutakh, Yury Polyanskiy, and Philippe Rigollet. Learning Gaussian Mixture Models via Transformer Mea- sure Flows. In ICML 2025 Workshop on Methods and Opportunities at Small Scale,

    2025

  17. [1973]

    The impact of lora on the emergence of clusters in transformers

    [KBH24] Hugo Koubbi, Matthieu Boussard, and Louis Hernandez. The impact of lora on the emergence of clusters in transformers. arXiv preprint arXiv:2402.15415,

    arXiv

  18. [1996]

    Finding clustering algorithms in the transformer ar- chitecture

    [CHI+25] Kenneth L Clarkson, Lior Horesh, Takuya Ito, Charlotte Park, and Parikshit Ram. Finding clustering algorithms in the transformer ar- chitecture. arXiv preprint arXiv:2506.19125,

    arXiv

  19. [2004]

    Two failure modes of deep transformers and how to avoid them: a unified theory of signal prop- agation at initialisation

    [GG25] Alessio Giorlandino and Sebastian Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal prop- agation at initialisation. arXiv preprint arXiv:2505.24333,

    arXiv

  20. [2005]

    Self-attention net- works localize when qk-eigenspectrum concentrates

    [BHK24] Han Bao, Ryuichiro Hataya, and Ryo Karakida. Self-attention net- works localize when qk-eigenspectrum concentrates. arXiv preprint arXiv:2402.02098,

    arXiv

  21. [2007]

    Synchro- nization of mean-field models on the circle

    [PRY25] Yury Polyanskiy, Philippe Rigollet, and Andrew Yao. Synchro- nization of mean-field models on the circle. arXiv preprint arXiv:2507.22857,

    arXiv

  22. [2012]

    Towards understanding how attention mechanism works in deep learning

    [RZ24] Tianyu Ruan and Shihua Zhang. Towards understanding how attention mechanism works in deep learning. arXiv preprint arXiv:2412.18288,

    arXiv

  23. [2015]

    The geometry of tokens in in- ternal representations of large language models

    [VGP+25] Karthik Viswanathan, Yuri Gardinazzi, Giada Panerai, Alberto Caz- zaniga, and Matteo Biagetti. The geometry of tokens in in- ternal representations of large language models. arXiv preprint arXiv:2501.10573,

    arXiv

  24. [2016]

    Understanding and improving transformer from a multi-particle dynamic system point of view

    [LLH+19] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762,

    1906 arXiv

  25. [2019]

    Continuum attention for neural operators

    [CKLS24] Edoardo Calvello, Nikola B Kovachki, Matthew E Levine, and An- drew M Stuart. Continuum attention for neural operators. arXiv preprint arXiv:2406.06486,

  26. [2021]

    Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models

    [DDZ+24] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.06066,

    arXiv

  27. [2022]

    A unified perspective on the dynamics of deep transformers

    [CACP25] Valérie Castin, Pierre Ablin, José Antonio Carrillo, and Gabriel Peyré. A unified perspective on the dynamics of deep transformers. arXiv preprint arXiv:2501.18322,

    arXiv

  28. [2024]

    Only large weights (and not skip con- nections) can prevent the perils of rank collapse

    [AS25] Josh Alman and Zhao Song. Only large weights (and not skip con- nections) can prevent the perils of rank collapse. arXiv preprint arXiv:2505.16284,

    arXiv

  29. [2025]

    Breaking BERT: evaluating and optimizing sparsified attention

    [BZM22] Siddhartha Brahma, Polina Zablotskaia, and David Mimno. Breaking BERT: evaluating and optimizing sparsified attention. arXiv preprint arXiv:2210.03841,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.