Pith. sign in

REVIEW 4 major objections 5 minor 112 references

In two dimensions, every linear self-attention transformer becomes a Kuramoto-type oscillator system, and the query/key/value matrices alone determine whether tokens cluster, rotate, or oscillate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:59 UTC pith:ONN5XUX4

load-bearing objection Genuine new reduction and case analysis for d=2 linear self-attention; oscillation/bifurcation claims remain OA-ansatz-bound, so the abstract overstates the stability reach. the 4 major comments →

arxiv 2607.18584 v1 pith:ONN5XUX4 submitted 2026-07-20 cs.LG math.DS

On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

Sixu Li , Thomas Jacob Maranzatto , Jan Peszek , Trevor Teolis , Semih Akkoc , Konstantin Riedl , Sennur Ulukus , Nicol\'as Garc\'ia Trillos This is my paper
classification cs.LG math.DS MSC 34C1534D0635Q8382C2237C20
keywords linear self-attentiontransformer inference dynamicsKuramoto modelOtt–Antonsen ansatzWatanabe–Strogatz transformationtoken clusteringoscillations and bifurcationsmean-field limit
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the inference-time dynamics of a deep linear encoder-only transformer in embedding dimension two are, for arbitrary real key/query/value matrices, exactly a generalized Kuramoto model with pure second-harmonic coupling. As a result, the entire n-token system is low-dimensional: a Watanabe–Strogatz transformation and the Ott–Antonsen ansatz reduce it to a single complex ODE for the second order parameter. On the OA manifold, the authors prove that different parameter matrices produce qualitatively different long-time behavior: fixed antipodal clustering, antipodal clustering whose cluster centers keep rotating, sustained periodic oscillations, and a saddle-node bifurcation. The paper also proves structural stability of the converging cases for initializations near the OA manifold, and reports numerical evidence that similar behaviors persist in higher dimensions and for softmax attention. A sympathetic reader should care because it offers matrix-level conditions that predict whether repeatedly applying self-attention layers stabilizes, drifts, or never settles.

Core claim

The paper's central claim is that the two-dimensional linear self-attention dynamics are governed by the second order parameter R₂, through θ̇_k = 2Re(B(R₂)e^{2iθ_k}) + C(R₂), with B and C fixed by the matrices A=QᵀK and V. On the Ott–Antonsen manifold—where the doubled-angle distribution is wrapped Cauchy—R₂ itself obeys the closed ODE Ṙ₂ = 2i(B(R₂)R₂² + C(R₂)R₂ + conj(B(R₂))). Using this reduced dynamics, the paper proves explicit matrix conditions: V=I with tr(A)>0 or det(A)≤0 forces convergence to antipodal points; A=I with symmetric V is decided by the sign of the largest eigenvalue; certain non-commuting A and V give ρ→1 while the phase never converges; and A=I with V of the form [[v11

What carries the argument

The load-bearing object is the order parameter R₂, the mean of e^{2iθ}, which converts pairwise interactions into mean-field coupling. The Watanabe–Strogatz Möbius transformation expresses each doubled angle as a Möbius image of a fixed constant of motion, reducing the n-particle system to two variables (α, η). The Ott–Antonsen ansatz is the special case where the constant of motion is uniformly distributed, i.e., the doubled-angle distribution is a wrapped Cauchy (Poisson-kernel) family; under this ansatz R₂ equals α and the dynamics close on a single ODE. A hidden Hamiltonian structure in Case 4, with Hamiltonian H(ρ,φ) after a time rescaling, is what turns periodic orbits and the saddle-n

Load-bearing premise

The oscillation and bifurcation theorems are proved only for initializations lying exactly on the Ott–Antonsen manifold (doubled angles uniformly distributed, hence wrapped Cauchy forever), and the paper itself states that any finite-particle system cannot satisfy that ansatz exactly.

What would settle it

In dimension two, simulate the finite-particle linear self-attention dynamics with A=I and V=[[v11, v12], [−v12, −v11]] with v11<v12, starting from tokens drawn uniformly and independently on the circle rather than from the wrapped Cauchy OA family. If |R₂(t)| decays to a fixed point or approaches 1 instead of cycling indefinitely, the sustained-oscillation result is an artifact of the OA ansatz rather than a property of the linear self-attention dynamics.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For V=I, conditions tr(A)>0 or det(A)≤0 imply almost every OA-initialized mean-field trajectory converges to a fixed pair of antipodal points; if instead (A+Aᵀ)/2 is negative definite, the order parameter settles at an interior point and token representations need not converge.
  • For A=I and symmetric V, λ_max(V)≥0 forces convergence to antipodal clusters, while λ_max(V)<0 leads to convergence without cluster formation.
  • When A and V are non-commuting in the stated family satisfying condition (3.13), tokens asymptotically form two antipodal clusters whose centers keep rotating; the phase never converges.
  • For A=I and V=[[v11, v12], [−v12, −v11]], the dynamics has a saddle-node bifurcation: v11<v12 gives periodic non-converging orbits for almost every initialization, while v11≥v12 gives convergence to a boundary equilibrium.
  • Theorems 4.7 and 4.8 extend the converging cases beyond the OA ansatz: for initial distributions with L^p density sufficiently close to uniform in bounded-Lipschitz distance, the same convergence to antipodal clusters holds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • For the oscillatory regimes, OA-ansatz initializations are a measure-zero condition that finite token systems cannot meet exactly; a natural test is whether periodic orbits persist under generic finite-n initializations or decay into slow convergence.
  • Because the behavior changes discontinuously at v11=v12, the theory implies that tiny perturbations of trained value matrices—for instance from quantization—could qualitatively alter layer-by-layer token evolution in parameter regimes near the bifurcation.
  • The same reduction may transfer to recurrent or looped transformer inference, where outputs cycle instead of converging; that connection is suggested by the paper but not proved.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the inference-time dynamics of deep linear encoder-only transformers through an interacting-particle-system perspective, focusing on the linear self-attention (LSA) model in embedding dimension d=2. The main mathematical claim is Theorem 2.1: for arbitrary real matrices A and V, the angle dynamics reduce to a Kuramoto-type model with pure second-harmonic coupling, ˙θ_k = 2Re(B(R_2)e^{i2θ_k}) + C(R_2). Using the Watanabe–Strogatz transformation and the Ott–Antonsen ansatz, the authors derive a single complex ODE for the mean-field order parameter R_2 and then analyze four parameter regimes: V=I with arbitrary A (Theorem 3.1), A=I with symmetric V (Theorem 3.3), a non-commuting A,V family with clustering without convergence (Theorem 3.6), and a hidden Hamiltonian regime with periodic orbits and a saddle-node bifurcation (Theorem 3.9). Section 4 presents structural stability results (Theorems 4.7 and 4.8) showing, under L^p and near-uniformity assumptions on the distribution of constants of motion, that convergence to antipodal clustering persists for the first two cases. Numerical experiments in dimension d=100 and, qualitatively, for softmax self-attention are reported in Section 5.

Significance. If the full transfer from the OA-reduced mean-field dynamics to finite-token transformers were established, this would be a substantial contribution: the exact second-harmonic Kuramoto reduction for arbitrary A,V, the WS and OA reductions, and the explicit matrix conditions for clustering, rotating clusters, periodic oscillations, and bifurcations are all derived without fitting any parameter. The paper ships detailed appendix proofs and makes the computer-algebra-assisted derivation reproducible via linked code. The spot-check A=V=I recovers the expected dynamics, and the threshold conditions in Theorems 3.1, 3.3, and 3.9 are explicit and falsifiable. However, the most novel behaviors — oscillations and bifurcations — are proved only inside the OA ansatz, and the paper itself acknowledges the finite-particle system cannot satisfy that ansatz exactly. The claimed token-level conclusions therefore rest on a robustness step that has not been proved for those regimes. This is a significant gap, but it is fixable by either extending the stability analysis, proving a finite-n approximation, or carefully re-scoping the claims.

major comments (4)
  1. [§2.2.2, Lemma 2.6, §3.3, §3.4] Theorems 3.6 and 3.9 are proved only for the OA system (2.31), which by Lemma 2.6 is valid only when the doubled-angle distribution is exactly wrapped Cauchy, equivalently ν=Unif. The paper explicitly states in §2.2.2 that the finite-particle system (2.9) cannot satisfy the OA ansatz exactly. The robustness results Theorems 4.7 and 4.8 cover only the converging Cases 1 and 2, require Assumption 4.4 and d_BL(ν,Unif)<ε with ε existential, and are mean-field statements. Consequently, the abstract's assertion that 'dynamics initialized near the OA manifold exhibit the same long-time behavior' is unsupported for the paper's most novel behaviors: clustering without convergence, sustained oscillations, and bifurcations. The paper should either provide a stability/attractiveness analysis for Cases 3 and 4 or explicitly present those token-level phenomena as conjectures supported by simulations.
  2. [§2.1, Eqs. (2.6)–(2.8)] The mean-field limit behind the PDE is introduced formally ('by letting n→∞, we formally consider...') without a propagation-of-chaos or finite-n quantitative error estimate. This matters because the OA ansatz is exact only at the mean-field level, while any finite token set has atomic ν. As written, Section 3 describes the mean-field OA dynamics, not the finite-particle LSA model (LSA) itself. A precise statement of the approximation error between the finite-n order parameter and the OA closure, or an explicit restriction of the theorems to the mean-field setting, is needed to support the token-level interpretation.
  3. [§4.2, Theorems 4.7 and 4.8] The title and abstract claim that the dynamics 'initialized near the OA manifold' exhibit the same long-time behavior, but the rigorous stability results are substantially weaker. Theorem 4.7 requires the stronger assumption (A+A⊤)/2≻0 rather than the OA condition tr(A)>0 or det(A)≤0, and excludes initial data in a neighborhood of an interior equilibrium via (4.18). Theorem 4.8 similarly requires λ_max(V)>0 and excludes a set via (4.22). Both require ν∈L^p and d_BL(ν,Unif)<ε with ε existential, and no finite-particle statement is given. These limitations should be stated in the abstract and in Section 1.1, because they materially narrow the scope of contribution (iii).
  4. [§3.3, Eq. (3.12) and surrounding text] The text states 'the functions H, K and Q depend on the matrices A and V' and then lists sufficient conditions H(φ)>0, |K(φ)|>0, |Q(ρ,φ)|<∞, but H has not been introduced at that point in the main text. It is defined only later in Eq. (C.34). This makes the statement of Theorem 3.6 hard to verify from the main text alone. Please rearrange the presentation or add a pointer to the definition.
minor comments (5)
  1. [Abstract and §1.1] The abstract says 'dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold.' As detailed above, this is not established for the oscillatory and bifurcating regimes. A qualifier such as 'for the converging regimes studied in Section 4' would avoid overclaiming.
  2. [§3.4] The text refers to 'the following proposition' before Theorem 3.9; it should say 'theorem'.
  3. [§5, Eq. (5.2)] The formula for \hat R_2(t) uses 1/\binom{n}{2}, which requires n≥2; the case n=1 is trivial but should be mentioned, and the equality |R_2^n|^2=\hat R_2 + (n-2)/n(\hat R_2 -1) should be stated for n≥2.
  4. [§1.3] The outline says 'reduced and trackable dynamics'; 'trackable' should be 'tractable'.
  5. [§4.2, Remark 4.6] The remark correctly notes that Assumption 4.4 excludes atomic distributions. To avoid confusion, this should be mentioned immediately after Theorem 4.7/4.8 as well, since finite-token initializations produce atomic ν.

Circularity Check

0 steps flagged

No significant circularity: the reduction from LSA to second-harmonic Kuramoto dynamics and the OA case analyses are self-contained derivations; the stated limitations are robustness gaps, not circular inputs.

full rationale

The paper's derivation chain is self-contained. Theorem 2.1 is obtained by explicit trigonometric expansion (Appendix B.2) of the linear self-attention ODE; the coupling functions B and C are written explicitly in terms of A and V in (A.4), with no fitted or data-dependent parameters. The Watanabe–Strogatz reduction (Proposition 2.4) is derived from the Riccati structure of the double-angle equation (B.4), and the proof is included; the citation to [71] is corroborated rather than load-bearing. The Ott–Antonsen closure R2 = α in (2.30)–(2.31) is an exact identity following from Lemma 2.6 (proved by residue computation) when ν = Unif; it is a mathematical reduction within an explicitly stated ansatz, not a self-consistent fit, and it is not equivalent to the Section 3 conclusions. The thresholds in Theorems 3.1, 3.3, 3.6, and 3.9 (e.g., tr(A)>0 or det(A)≤0, λmax(V)≥0, condition (3.13), v11<v12) come from phase-plane and Hamiltonian analyses of the reduced ODEs, not from fitting outputs to inputs. There is no renaming of an existing result: the pure second-harmonic Kuramoto formulation is a new exact representation with explicit coefficient formulas. No load-bearing self-citation appears; the authors' prior works cited are related-work context only. The paper explicitly flags its own limitations — 'the finite-particle system (2.9) cannot satisfy the OA ansatz exactly' (Sec. 2.2.2), and the mean-field limit is introduced as 'by letting n → ∞, we formally consider' (Sec. 2.1) — and the structural-stability theorems cover Cases 1–2 only. These are gaps in genericity and rigor, not circular reductions of the central claims.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No fitted parameters: every threshold (tr(A)>0, det(A)≤0, λ_max(V)≥0, v11<v12, (3.13)) is derived from the phase-plane analysis of the reduced ODE. The cost of the framework is paid instead in restrictions on initializations and on the model: (i) the mean-field limit (Eqs. (2.6)–(2.8)) is formal; (ii) the single-ODE reduction (2.31) and all Section 3 theorems require the doubled-angle law to be exactly wrapped Cauchy (ν = Unif); (iii) the stability theorems require ν∈L^p with bounded norm and d_BL(ν,Unif)<ε with ε existential, and exclude an initial equilibrium set; (iv) the whole analysis is for the continuous-time linear-attention model with tokens on the sphere, β absorbed into A/V. No new entities are postulated; the doubled angle is a change of variables and the OA manifold is imported from oscillator theory. A residual verification cost is the CAS-assisted coefficient algebra (b1..b5, w0..w8 in (A.2)–(A.4)), which is stated rather than symbolically derived in the text.

axioms (6)
  • domain assumption The continuous-time model (LSA) with linear kernel h(y)=y, Z_k=n, β absorbed into A/V, tokens constrained to S^{d-1}, is the right idealization of deep encoder-only transformer inference.
    Introduced at Eq. (LSA), Section 1; the entire framework studies this model rather than discrete layers or softmax attention.
  • domain assumption Mean-field limit: as n→∞ the empirical measure is replaced by a law f satisfying the continuity equation (2.8); no propagation-of-chaos proof is given.
    Section 2.1, Eqs. (2.6)–(2.8): 'we formally consider a process Θ'.
  • ad hoc to paper Ott–Antonsen ansatz: the doubled-angle distribution g is exactly wrapped Cauchy, equivalently ν = Unif, for the single-ODE reduction (2.31) and all Section 3 theorems.
    Lemma 2.6 and Section 2.2.2; finite-particle systems cannot satisfy this (stated in Section 2.2.2); Section 4 is the partial de-risking for Cases 1–2.
  • ad hoc to paper For stability theorems: ν has L^p density with ∥ν∥_{L^p} ≤ M and d_BL(ν,Unif) < ε(ι,p,M,A/V) with ε existential; initials must avoid interior equilibria (4.18)/(4.22).
    Assumption 4.4, (4.20), (4.24); the smallness is not quantified, so the regime of validity is qualitative.
  • standard math Watanabe–Strogatz / OA / Möbius group machinery as in [71,79,80,105], including the residue-theorem computation of Fourier coefficients of the wrapped Cauchy.
    Proposition 2.4, Lemma 2.6; standard results applied to the derived second-harmonic field.
  • standard math LaSalle invariance, Barbălat lemma, Poincaré–Bendixson, Hartman–Grobman used for long-time conclusions.
    Appendix C proofs (Claims 1–3, Theorem 3.9).

pith-pipeline@v1.3.0-alltime-deepseek · 66613 in / 27201 out tokens · 295973 ms · 2026-08-01T14:59:37.444814+00:00 · methodology

0 comments
read the original abstract

We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter matrices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that the parameter matrices induce a diverse variety of long-time behaviors in linear transformers, including clustering, oscillations, and bifurcations. The oscillations and bifurcations are characterized by uncovering a hidden Hamiltonian structure in the dynamics. By establishing a structural stability result, we further show that dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold. Motivated by our theory in dimension two, we conduct numerical experiments for analogous parameter regimes in higher-dimensional transformers. Our numerical experiments suggest that the long-time behaviors characterized in our theoretical results persist in higher dimensions.

Figures

Figures reproduced from arXiv: 2607.18584 by Jan Peszek, Konstantin Riedl, Nicol\'as Garc\'ia Trillos, Semih Akkoc, Sennur Ulukus, Sixu Li, Thomas Jacob Maranzatto, Trevor Teolis.

Figure 1
Figure 1. Figure 1: Snapshots of two systems with 20 particles evolving according to ( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Diagram illustrating the M¨obius transformation. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The family of wrapped Cauchy distributions generated by fixing the argument Arg( [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cartesian and complex phase portraits of ( [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cartesian and complex phase portraits of ( [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Phase portrait of (ρ, ϕ)-dynamics (3.10) and the trajectories of particle evolving under the trans￾former dynamics (2.4) for A = I, V = [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Phase portrait of (ρ, ϕ)-dynamics (3.10) and the trajectories of particle evolving under the trans￾former dynamics (2.4) for A = I, V = −2 2 2 −3  , which satisfies λmax(V ) < 0. Theorem 3.3 states that, when A = I and the matrix V is symmetric, the long-term behavior of dynamics (3.10) is entirely determined by the sign of the largest eigenvalue of V . More precisely, if λmax(V ) ≥ 0, then the order para… view at source ↗
Figure 8
Figure 8. Figure 8: Phase portrait of (ρ, ϕ)-dynamics (3.12) and the trajectories of particle evolving under the trans￾former dynamics (2.4) for A = ( 1 0 0 2 ), V = [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Figures showing the phase portraits and bifurcation phenomena in the dynamics ( [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Case 1, parameter regime: tr(A) > 0 and V = I. For both the LSA and USA models, the mean-squared cosine similarity Rb2(t) converges to 1. However, the mean cosine similarity Rb1(t) converges to 1 for the USA model, shown by the red dotted line, but to 0 for the LSA model, shown by the blue dashed line. This indicates that the USA dynamics clusters at a single point, whereas the LSA dynamics converges to t… view at source ↗
Figure 11
Figure 11. Figure 11: Case 1, parameter regime: tr(A) ≤ 0, det(A) ≤ 0 and V = I. The same qualitative behavior as [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Case 1, parameter regime: (A + A⊤)/2 ≺ 0 and V = I. Both clustering diagnostics Rb1(t) and Rb2(t) converge to almost 0 for the LSA model and converge to 1 for the USA model. This implies that the LSA model does not synchronize whereas the USA model clusters at a single point. (a) Evolution of the clustering diagnostics Rb1(t) and Rb2(t) for the LSA (blue) and USA (red) models. (b) The Gram matrices G(t) f… view at source ↗
Figure 13
Figure 13. Figure 13: Case 2, parameter regime: A = I and λmax(V ) > 0. For both the LSA and USA models, the mean-squared cosine similarity Rb2(t) converges to 1. However, the mean cosine similarity Rb1(t) converges to 1 for the USA model, shown by the red dotted line, but to 0 for the LSA model, shown by the blue dashed line. This indicates that the USA dynamics clusters at a single point, whereas the LSA dynamics converges t… view at source ↗
Figure 14
Figure 14. Figure 14: Case 2, parameter regime: A = I and λmax(V ) = 0. For both the LSA and USA models, the mean-squared cosine similarity Rb2(t) converges to 1, while the mean cosine similarity Rb1(t) remains close to 0. This indicates that both dynamics converge to antipodal clustering configurations. Compared with the regime λmax(V ) > 0, the convergence in this boundary regime is significantly slower for both models. (a) … view at source ↗
Figure 15
Figure 15. Figure 15: Case 2, parameter regime: A = I and λmax(V ) < 0. For both the LSA and USA models, the clustering diagnostics remain at Rb1(t) = Rb2(t) = 0, while the per-dimensional means ml(t) converge across dimensions. These observations indicate that the dynamics converge, but without cluster formation. For the USA model, the dynamics exhibit a similar combination of clustering and non-convergence. The main differen… view at source ↗
Figure 16
Figure 16. Figure 16: Case 3, parameter regime: A = I and V = ( v11 v12 −v12 v11 ) with v11 > 0 and v12 ̸= 0. For both the LSA and USA models, the mean-squared cosine similarity Rb2(t) converges to 1. However, the mean cosine similarity Rb1(t) converges to 1 for the USA model, shown by the red dotted line, but to 0 for the LSA model, shown by the blue dashed line; see Figure 16a. Thus, the USA dynamics clusters at a single poi… view at source ↗
Figure 17
Figure 17. Figure 17: , convergence is relatively rapid for both models. However, at the bifurcation point v11 = v12, shown in [PITH_FULL_IMAGE:figures/full_fig_p035_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Case 4, parameter regime: v11 = v12. The qualitative behavior is the same as in the regime v11 > v12: for both the LSA and USA models, Rb2(t) converges to 1, while Rb1(t) converges to 1 for the USA model and to 0 for the LSA model. This indicates that the USA dynamics clusters at a single point, whereas the LSA dynamics converges to two antipodal clusters. However, convergence is notably slower in this re… view at source ↗
Figure 19
Figure 19. Figure 19: Case 4, parameter regime: v11 < v12. For both the LSA and USA models, the clustering diagnostics remain at Rb1(t) = Rb2(t) = 0, while the per-dimensional means ml(t) oscillate periodically. This indicates persistent oscillatory dynamics without cluster formation. 6 Conclusion In this paper, we develop an interacting-particle-system-based approach to analyze the inference-time be￾havior of deep linear enco… view at source ↗
Figure 20
Figure 20. Figure 20: The visualization of the trajectory z(t) relative to the interior equilibrium zeq and the boundary equilibrium set ∂D. The solid dot is the initialization point, while the arrow indicates the moving direction of z(t). The second important quantity is the angular motion of the displacement z(t) − zeq, that is, ζ(t) := Arg(z(t) − zeq). (C.8) This quantity describes how the trajectory z(t) rotates around zeq… view at source ↗
Figure 21
Figure 21. Figure 21: Level sets of the Hamiltonian H(ρ, ϕ) for different parameter regimes. The dashed orange curve indicates the special level set {H = −b/2}, while the colored solid curves represent other level sets. The blue dots denote the fixed points of the dynamics (3.18), introduced in (C.39) and (C.40). The arrows indicate the direction of the flow. curve in [PITH_FULL_IMAGE:figures/full_fig_p066_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

112 extracted references · 23 linked inside Pith

  1. [1]

    ´A. R. Abella, J. P. Silvestre, and P. Tabuada. The asymptotic behavior of attention in transformers. arXiv preprint arXiv:2412.02682, 2024

    arXiv 2024

  2. [2]

    ´A. R. Abella, J. P. Silvestre, and P. Tabuada. Consensus is all you get: The role of attention in transformers. InForty-second International Conference on Machine Learning, 2025

    2025

  3. [3]

    J. A. Acebr´ on, L. L. Bonilla, C. J. P´ erez Vicente, F. Ritort, and R. Spigler. The kuramoto model: A simple paradigm for synchronization phenomena.Reviews of modern physics, 77(1):137–185, 2005

    2005

  4. [4]

    Agazzi, G

    A. Agazzi, G. Bruno, E. M. Garc ´ ıa, S. Saviozzi, and M. Romito. Stochastic scaling limits and syn- chronization by noise in deep transformer models.arXiv preprint arXiv:2604.26898, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Alcalde, L

    A. Alcalde, L. Bungert, K. Riedl, and T. Roith. Quantifying concentration phenomena of mean-field transformers in the low-temperature regime.arXiv preprint arXiv:2605.10931, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Alcalde, G

    A. Alcalde, G. Fantuzzi, and E. Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis.SIAM Journal on Mathematics of Data Science, 7(3):1367–1393, 2025

    2025

  7. [7]

    Alcalde, B

    A. Alcalde, B. Geshkovski, and D. Ruiz-Balet. Attention’s forward pass and frank-wolfe.arXiv preprint arXiv:2508.09628, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Alcalde, Z

    A. Alcalde, Z. Ji, and E. Zuazua. Reachability and asymptotics of gaussian transformer dynamics. arXiv preprint arXiv:2606.07600, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Altafini

    C. Altafini. Multistability of self-attention dynamics in transformers.IEEE Transactions on Automatic Control, 2026

    2026

  10. [10]

    ´Alvarez-L´ opez, B

    A. ´Alvarez-L´ opez, B. Geshkovski, and D. Ruiz-Balet. Perceptrons and localization of attention’s mean- field landscape.arXiv preprint arXiv:2601.21366, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    Balasubramanian, S

    K. Balasubramanian, S. Banerjee, and P. Rigollet. On the structure of stationary solutions to mckean- vlasov equations with applications to noisy transformers.arXiv preprint arXiv:2510.20094, 2025

    arXiv 2025

  12. [12]

    Barb˘ alat

    I. Barb˘ alat. Syst` emes d’´ equations diff´ erentielles d’oscillations non lin´ eaires.Rev. Math. Pures Appl., 4:267–270, 1959. see p. 269

    1959

  13. [13]

    Blayney, ´A

    H. Blayney, ´A. Arroyo, J. Obando-Ceron, P. S. Castro, A. Courville, M. M. Bronstein, and X. Dong. A mechanistic analysis of looped reasoning language models.arXiv preprint arXiv:2604.11791, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    B¨ ohm, C

    C. B¨ ohm, C. Plant, J. Shao, and Q. Yang. Clustering by synchronization. InProceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 583–592, 2010

    2010

  15. [15]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Breakspear, S

    M. Breakspear, S. Heitmann, and A. Daffertshofer. Generative models of cortical oscillations: Neuro- biological implications of the kuramoto model.Frontiers in Human Neuroscience, Volume 4 - 2010, 2010

    2010

  17. [17]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amod...

    2020

  18. [18]

    Bruno, S

    G. Bruno, S. Chen, Z. Lin, Y. Polyanskiy, and P. Rigollet. Scaling limits of long-context transformers. arXiv preprint arXiv:2605.08505, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Bruno, F

    G. Bruno, F. Pasqualotto, and A. Agazzi. Emergence of meta-stable clustering in mean-field trans- former models.arXiv preprint arXiv:2410.23228, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Bruno, F

    G. Bruno, F. Pasqualotto, and A. Agazzi. A multiscale analysis of mean-field transformers in the moderate interaction regime.arXiv preprint arXiv:2509.25040, 2025

    arXiv 2025

  21. [21]

    Burger, S

    M. Burger, S. Kabri, Y. Korolev, T. Roith, and L. Weigand. Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization.Philos. Trans. Roy. Soc. A, 383(2298):Paper No. 20240233, 48, 2025

    2025

  22. [22]

    Caponigro, A

    M. Caponigro, A. C. Lai, and B. Piccoli. A nonlinear model of opinion formation on the sphere. Discrete and Continuous Dynamical Systems, 35(9):4241–4268, 2015

    2015

  23. [23]

    Castin, P

    V. Castin, P. Ablin, J. A. Carrillo, and G. Peyr´ e. A unified perspective on the dynamics of deep transformers.arXiv preprint arXiv:2501.18322, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Cestnik and A

    R. Cestnik and A. Pikovsky. Hierarchy of exact low-dimensional reductions for populations of coupled oscillators.Physical Review Letters, 128(5):054101, 2022

    2022

  25. [25]

    Chandra, M

    S. Chandra, M. Girvan, and E. Ott. Complexity reduction ansatz for systems of interacting orientable agents: Beyond the kuramoto model.Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(5), 2019

    2019

  26. [26]

    S. Chen, Z. Lin, Y. Polyanskiy, and P. Rigollet. Critical attention scaling in long-context transformers. arXiv preprint arXiv:2510.05554, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    S. Chen, Z. Lin, Y. Polyanskiy, and P. Rigollet. Quantitative clustering in mean-field transformer models.arXiv preprint arXiv:2504.14697, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    T. Chen, Z. Zhang, Y. Cheng, A. Awadallah, and Z. Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12020–12030, 2022

    2022

  29. [29]

    Z. Chen, Z. Lin, S. Chen, Y. Polyanskiy, and P. Rigollet. Residual connections provably mitigate oversmoothing in graph neural networks.arXiv preprint arXiv:2501.00762, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. InInternational Conference on Learning Representations, 2021

    2021

  31. [31]

    Cowsik, T

    A. Cowsik, T. Nebabu, X. Qi, and S. Ganguli. Geometric dynamics of signal propagation predict trainability of transformers.Physical Review E, 112(5):055301, 2025

    2025

  32. [32]

    Criscitiello, Q

    C. Criscitiello, Q. Rebjock, A. D. McRae, and N. Boumal. Synchronization on Circles and Spheres with Nonlinear Interactions.SIAM J. Appl. Dyn. Syst., 25(2):1207–1237, 2026

    2026

  33. [33]

    H. Daido. Order function and macroscopic mutual entrainment in uniformly coupled limit-cycle oscil- lators.Progress of theoretical physics, 88(6):1213–1218, 1992

    1992

  34. [34]

    H. Daido. Generic scaling at the onset of macroscopic mutual entrainment in limit-cycle oscillators with uniform all-to-all coupling.Phys. Rev. Lett., 73:760–763, Aug 1994

    1994

  35. [35]

    Degond, A

    P. Degond, A. Frouvelle, and J.-G. Liu. Macroscopic limits and phase transition in a system of self- propelled particles.Journal of nonlinear science, 23(3):427–456, 2013

    2013

  36. [36]

    Degond and S

    P. Degond and S. Motsch. Continuum limit of self-driven particles with orientation interaction.Math- ematical Models and Methods in Applied Sciences, 18(supp01):1193–1215, 2008. 39

    2008

  37. [37]

    Devlin, M

    J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, U...

    2019

  38. [38]

    Dong, J.-B

    Y. Dong, J.-B. Cordonnier, and A. Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. InInternational conference on machine learning, pages 2793–2803. PMLR, 2021

    2021

  39. [39]

    D¨ orfler and F

    F. D¨ orfler and F. Bullo. Synchronization in complex networks of phase oscillators: A survey.Auto- matica, 50(6):1539–1564, 2014

    2014

  40. [40]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations, 2021

    2021

  41. [41]

    Dutta, T

    S. Dutta, T. Gautam, S. Chakrabarti, and T. Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, Neur...

    2021

  42. [42]

    Engel and A

    M. Engel and A. Shalova. Random quadratic form on a sphere: Synchronization by common noise. arXiv preprint arXiv:2603.06187, 2026

    arXiv 2026

  43. [43]

    Fedorov, M

    L. Fedorov, M. E. Sander, R. Elie, P. Marion, and M. Lauri` ere. Clustering in deep stochastic trans- formers.arXiv preprint arXiv:2601.21942, 2026

    arXiv 2026

  44. [44]

    R. Feng, K. Zheng, Y. Huang, D. Zhao, M. Jordan, and Z.-J. Zha. Rank diminishing in deep neural networks. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  45. [45]

    Filatrella, A

    G. Filatrella, A. H. Nielsen, and N. F. Pedersen. Analysis of a power grid using a kuramoto-like model. The European Physical Journal B, 61(4):485–491, 2008

    2008

  46. [46]

    Geiping, S

    J. Geiping, S. M. McLeish, N. Jain, J. Kirchenbauer, S. Singh, B. R. Bartoldson, B. Kailkhura, A. Bhatele, and T. Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  47. [47]

    Geshkovski, H

    B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet. Dynamic metastability in the self-attention model, 2024

    2024

  48. [48]

    Geshkovski, C

    B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet. A mathematical perspective on transform- ers.Bull. Amer. Math. Soc. (N.S.), 62(3):427–479, 2025

    2025

  49. [49]

    Giacomin, K

    G. Giacomin, K. Pakdaman, X. Pellegrin, and C. Poquet. Transitions in active rotator systems: invariant hyperbolic manifold approach.SIAM Journal on Mathematical Analysis, 44(6):4165–4194, 2012

    2012

  50. [50]

    Giannou, S

    A. Giannou, S. Rajput, J.-y. Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos. Looped transformers as programmable computers. InInternational Conference on Machine Learning, pages 11398–11442. PMLR, 2023

    2023

  51. [51]

    Giorlandino and S

    A. Giorlandino and S. Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation. InThe Fourteenth International Conference on Learning Representations, 2026. 40

    2026

  52. [52]

    C. C. Gong and A. Pikovsky. Low-dimensional dynamics for higher-order harmonic, globally coupled phase-oscillator ensembles.Physical Review E, 100(6), Dec. 2019

    2019

  53. [53]

    X. Guo, Y. Wang, T. Du, and Y. Wang. Contranorm: A contrastive learning perspective on over- smoothing and beyond. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  54. [54]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memory.Neural computation, 9(8):1735–1780, 1997

    1997

  55. [55]

    Hooper, S

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    2024

  56. [56]

    P. Ji, T. K. Peron, F. A. Rodrigues, and J. Kurths. Low-dimensional behavior of kuramoto model with inertia in complex networks.Scientific reports, 4(1):4783, 2014

    2014

  57. [57]

    Karagodin, Y

    N. Karagodin, Y. Polyanskiy, and P. Rigollet. Clustering in causal attention masking, 2024

    2024

  58. [58]

    Karagodin, Y

    N. Karagodin, Y. Polyanskiy, and P. Rigollet. Clustering in causal attention masking.Advances in neural information processing systems, 37:115652–115681, 2024

    2024

  59. [59]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  60. [60]

    I. Z. Kiss, Y. Zhai, and J. L. Hudson. Emerging coherence in a population of chemical oscillators. Science, 296(5573):1676–1678, 2002

    2002

  61. [61]

    Koubbi, B

    H. Koubbi, B. Geshkovski, and P. Rigollet. Homogenized transformers.arXiv preprint arXiv:2604.01978, 2026

    arXiv 2026

  62. [62]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    2012

  63. [63]

    Kuramoto

    Y. Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In H. Araki, editor, International Symposium on Mathematical Problems in Theoretical Physics, pages 420–422, Berlin, Heidelberg, 1975. Springer Berlin Heidelberg

    1975

  64. [64]

    Kuramoto

    Y. Kuramoto. Chemical turbulence. InChemical oscillations, waves, and turbulence, pages 111–140. Springer, 1984

    1984

  65. [65]

    Lipton, R

    M. Lipton, R. Mirollo, and S. H. Strogatz. The kuramoto model on a sphere: Explaining its low- dimensional dynamics with group theory and hyperbolic geometry.Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(9):093113, 09 2021

    2021

  66. [66]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    M. A. Lohe. Higher-dimensional generalizations of the watanabe–strogatz transform for vector models of synchronization.Journal of Physics A: Mathematical and Theoretical, 51(22):225101, may 2018

    2018

  68. [68]

    M. A. Lohe. Exact reduction of synchronized systems in higher-dimensional spaces.Chaos: An Interdisciplinary Journal of Nonlinear Science, 35(2), 2025

    2025

  69. [69]

    Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T. Liu. Understanding and improving transformer from a multi-particle dynamic system point of view.CoRR, abs/1906.02762, 2019

    Pith/arXiv arXiv 1906

  70. [70]

    K. V. Mardia and P. E. Jupp.Directional statistics. John Wiley & Sons, 2009. 41

    2009

  71. [71]

    S. A. Marvel, R. E. Mirollo, and S. H. Strogatz. Identical phase oscillators with global sinusoidal coupling evolve by m¨ obius group action.Chaos: An Interdisciplinary Journal of Nonlinear Science, 19(4), 2009

    2009

  72. [72]

    Massucco, L

    A. Massucco, L. Del Grande, M. Carioni, C. Brune, and C.-B. Sch¨ onlieb. Multi-headed transformer architectures as time-dependent wasserstein gradient flows.arXiv preprint arXiv:2605.18870, 2026

    Pith/arXiv arXiv 2026

  73. [73]

    Miyato, S

    T. Miyato, S. L¨ owe, A. Geiger, and M. Welling. Artificial kuramoto oscillatory neurons. InInternational Conference on Learning Representations, volume 2025, pages 44278–44322, 2025

    2025

  74. [74]

    Mun and M

    K. Mun and M. Rosenzweig. Phase transitions and linear stability for the mean-field kuramoto-daido model.arXiv preprint arXiv:2602.14954, 2026

    arXiv 2026

  75. [75]

    Mun and M

    K. Mun and M. Rosenzweig. Phase transitions for the noisy transformer model in arbitrary dimension. arXiv preprint arXiv:2606.05140, 2026

    Pith/arXiv arXiv 2026

  76. [76]

    Mun and M

    K. Mun and M. Rosenzweig. Phase transitions in doi-onsager, noisy transformer, and other multimodal models.arXiv preprint arXiv:2604.16288, 2026

    Pith/arXiv arXiv 2026

  77. [77]

    Nguyen, H

    T. Nguyen, H. Honda, T. Sano, V. Nguyen, S. Nakamura, and T. M. Nguyen. From coupled oscilla- tors to graph neural networks: Reducing over-smoothing via a kuramoto model-based approach. In International Conference on Artificial Intelligence and Statistics, pages 2710–2718. PMLR, 2024

    2024

  78. [78]

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.Advances in Neural Information Processing Systems, 35:27198–27211, 2022

    2022

  79. [79]

    Ott and T

    E. Ott and T. M. Antonsen. Low dimensional behavior of large systems of globally coupled oscillators. Chaos: An Interdisciplinary Journal of Nonlinear Science, 18(3), 2008

    2008

  80. [80]

    Ott and T

    E. Ott and T. M. Antonsen. Long time evolution of phase oscillator systems.Chaos: An Interdisci- plinary Journal of Nonlinear Science, 19(2), May 2009

    2009

Showing first 80 references.