REVIEW 4 cited by
Emergence of meta-stable clustering in mean-field transformer models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Emergence of meta-stable clustering in mean-field transformer models
read the original abstract
We model the evolution of tokens within a deep stack of Transformer layers as a continuous-time flow on the unit sphere, governed by a mean-field interacting particle system, building on the framework introduced in (Geshkovski et al., 2023). Studying the corresponding mean-field Partial Differential Equation (PDE), which can be interpreted as a Wasserstein gradient flow, in this paper we provide a mathematical investigation of the long-term behavior of this system, with a particular focus on the emergence and persistence of meta-stable phases and clustering phenomena, key elements in applications like next-token prediction. More specifically, we perform a perturbative analysis of the mean-field PDE around the iid uniform initialization and prove that, in the limit of large number of tokens, the model remains close to a meta-stable manifold of solutions with a given structure (e.g., periodicity). Further, the structure characterizing the meta-stable manifold is explicitly identified, as a function of the inverse temperature parameter of the model, by the index maximizing a certain rescaling of Gegenbauer polynomials.
Forward citations
Cited by 4 Pith papers
-
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Linear self-attention in 2D reduces to a second-harmonic Kuramoto model whose order parameter obeys one ODE; explicit matrix conditions yield clustering, rotating clusters, Hamiltonian oscillations, and bifurcations.
-
Spectral Selection in Symmetric Self-Attention Dynamics
Symmetric self-attention dynamics select the dominant eigendirection of V, producing homogeneous alignment when one positive eigenvalue dominates or sign-split polarization when V is negative definite.
-
Formation of clusters and coarsening in weakly interacting diffusions
Global minimizers of the McKean–Vlasov free energy for short-range attractive potentials on the circle are uniform or single symmetric clusters, and coarsening is governed by exponentially slow mass exchange between clusters.
-
Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
Analytical theory of signal propagation in deep transformers at initialization yields quantitative prescriptions for weights and residuals to avoid rank and entropy collapse via Random Energy Model analogy.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.