Transformers converge pathwise to a stochastic particle system and SPDE in the scaling limit, exhibiting synchronization by noise and exponential energy dissipation when common noise is coercive relative to self-attention drift.
Formation of clusters and coarsening in weakly interacting diffusions
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
abstract
This paper studies the clustering behavior of weakly interacting diffusions under the influence of sufficiently localized attractive interaction potentials on the one-dimensional torus. We describe how this clustering behavior is closely related to the presence of discontinuous phase transitions in the mean-field PDE. For local attractive interactions, we employ a new variant of the strict Riesz rearrangement inequality to prove that all global minimizers of the free energy are either uniform or single-cluster states, in the sense that they are symmetrically decreasing. We analyze different timescales for the particle system and the mean-field (McKean-Vlasov) PDE, arguing that while the particle system can exhibit coarsening by both coalescence and diffusive mass exchange between clusters, the clusters in the mean-field PDE are unable to move and coarsening occurs via the mass exchange of clusters. By introducing a new model for this mass exchange, we argue that the PDE exhibits dynamical metastability. We conclude by presenting careful numerical experiments that demonstrate the validity of our model.
years
2026 6verdicts
UNVERDICTED 6representative citing papers
In the mean-field limit of attention with perceptron blocks, critical points of the energy landscape are generically atomic and localized on subsets of the unit sphere.
Models multi-head transformer data flow as time-dependent Wasserstein gradient flows of an attention-capturing interaction energy, with proofs on omega-limit stationary points and stability under weight and input perturbations.
Reduced SPDE models for co-evolving opinion dynamics capture clustering behavior efficiently with lower cost than full-state models.
A data-driven framework reduces particle-based transfer operators via concentration projection, geometric manifold, and finite-state discretization to reproduce clustering transitions and metastable states from simulation data.
Numerical experiments on McKean-Vlasov equations with attractive kernels reveal alternating aggregation-diffusion regimes in transient clustering, with density peak height only partially coupled to energetic mechanisms.
citing papers explorer
-
Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
Transformers converge pathwise to a stochastic particle system and SPDE in the scaling limit, exhibiting synchronization by noise and exponential energy dissipation when common noise is coercive relative to self-attention drift.
-
Perceptrons and localization of attention's mean-field landscape
In the mean-field limit of attention with perceptron blocks, critical points of the energy landscape are generically atomic and localized on subsets of the unit sphere.
-
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
Models multi-head transformer data flow as time-dependent Wasserstein gradient flows of an attention-capturing interaction energy, with proofs on omega-limit stationary points and stability under weight and input perturbations.
-
Clustering in co-evolving opinion dynamics: reduced SPDE models
Reduced SPDE models for co-evolving opinion dynamics capture clustering behavior efficiently with lower cost than full-state models.
-
Data-driven Reduction of Transfer Operators for Particle Clustering Dynamics
A data-driven framework reduces particle-based transfer operators via concentration projection, geometric manifold, and finite-state discretization to reproduce clustering transitions and metastable states from simulation data.
-
Energetic characterisation of transient clustering dynamics in aggregation-diffusion systems
Numerical experiments on McKean-Vlasov equations with attractive kernels reveal alternating aggregation-diffusion regimes in transient clustering, with density peak height only partially coupled to energetic mechanisms.