Positional schemes set the default spectral algebra of attention heads: previous-token heads are rotational under RoPE and content-like under absolute/ALiBi, as a post-function fingerprint rather than a hard constraint.
Gregor Bachmann and Vaishnavh Nagarajan
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their internal mechanisms in order to make them more reliable. These models appear to store vast amounts of knowledge from their training data, and to adapt quickly to new information provided in their context or prompt. We study how transformers balance these two types of knowledge by considering a synthetic setup where tokens are generated from either global or context-specific bigram distributions. By a careful empirical analysis of the training process on a simplified two-layer transformer, we illustrate the fast learning of global bigrams and the slower development of an "induction head" mechanism for the in-context bigrams. We highlight the role of weight matrices as associative memories, provide theoretical insights on how gradients enable their learning during training, and study the role of data-distributional properties.
citation-role summary
baseline 1
citation-polarity summary
years
2026 4roles
baseline 1polarities
baseline 1representative citing papers
Semantic geometry emerges transiently early in next-token prediction training before collapsing to Neural Collapse symmetry in synthetic settings with latent semantic factors.
Gradient descent reaches equal-weight superposition on Erdős-Rényi reachability via a Möbius attractor under S_n symmetry and Cascade Supervision, with a parameter-free decay law matching experimental cosines within 0.02.
citing papers explorer
-
Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
Positional schemes set the default spectral algebra of attention heads: previous-token heads are rotational under RoPE and content-like under absolute/ALiBi, as a post-function fingerprint rather than a hard constraint.
-
Structure Before Collapse: Transient semantic geometry in next-token prediction
Semantic geometry emerges transiently early in next-token prediction training before collapsing to Neural Collapse symmetry in synthetic settings with latent semantic factors.
-
Emergence of Frontier Superposition: M\"obius attractor and Cascade Supervision
Gradient descent reaches equal-weight superposition on Erdős-Rényi reachability via a Möbius attractor under S_n symmetry and Cascade Supervision, with a parameter-free decay law matching experimental cosines within 0.02.
- Sharp Capacity Thresholds in Linear Associative Memory: From Top-1 Retrieval to Tail-Average Learning