Conflict and hallucination in transformers are basin competition versus basin absence in hidden-state space; geometric margin detects them with zero false refusals while entropy cannot, and confident hallucinations scale as exp(-c/Δ̄).
Universal One-third Time Scaling in Learning Peaked Distributions
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.
years
2026 3representative citing papers
Derives α^{-1/3} scaling for generalization error in online softmax classification from boundary layers in a teacher-student model.
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
citing papers explorer
-
Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination
Conflict and hallucination in transformers are basin competition versus basin absence in hidden-state space; geometric margin detects them with zero false refusals while entropy cannot, and confident hallucinations scale as exp(-c/Δ̄).
-
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
Derives α^{-1/3} scaling for generalization error in online softmax classification from boundary layers in a teacher-student model.
-
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.