Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.
hub
arXiv preprint arXiv:2210.16859 , year=
19 Pith papers cite this work. Polarity classification is still indexing.
19
Pith papers citing it
hub tools
citation-role summary
background 3
dataset 1
citation-polarity summary
representative citing papers
Under a shared-head/disjoint-tail assumption, multi-domain loss decomposes into a capacity-competition term c_i x_i^*(h)^{-b_i} plus a per-domain noise term A_i(Dh_i)^{-a_i}, and the fitted law extrapolates optimal mixtures to unseen scales.
A solvable hierarchical model with power-law feature strengths yields explicit power-law scaling of prediction error through sequential recovery of latent directions by a layer-wise spectral algorithm.
Derives layer-wise recursions for finite-width tensors under orthogonal initialization that reproduce the observed large-depth stability of nonlinear networks.
For a sketched bilinear contrastive model trained by full-batch gradient descent, the expected risk decomposes into irreducible, approximation, bias, and variance terms, with variance set by a product effective dimension.
Critical percolation clusters embedded in high dimensions, combined with taxonomic latent variables, form an analytically tractable synthetic data model whose ground-truth hierarchy can be linearly decoded from network activations.
Mean-field equations for attention retrieval, teacher alignment, and logic overlap quantitatively match simulations and predict a sharp accuracy transition in a solvable transformer for permutation state tracking.
Larger neural networks keep learning from ever-weaker spectral directions of the empirical neural tangent kernel, a capacity the authors call spectral reach.
Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B parameters.
Derives α^{-1/3} scaling for generalization error in online softmax classification from boundary layers in a teacher-student model.
A new scaling law L(N, D, T) = E + (L0 - E) h/(1+h) with h = a/N^α + b/T^β + c N^γ/D^δ that decomposes loss into undercapacity, undertraining, and overfitting terms and saturates between E and L0.
Spectral analysis of activations and gradients provides new diagnostics that link batch size to representation geometry, early covariance tails to token efficiency, and spectral shifts to learning dynamics in decoder-only LLMs, backed by a mechanistic model.
Strong superposition causes neural loss to scale as the inverse of model dimension due to geometric feature overlaps, explaining scaling laws for broad frequency distributions.
Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.
A sparse-activation model predicts double-descent loss with distinct under- and over-parameterized scaling exponents set by sparsity, plus a compute-optimal frontier favoring dataset growth.
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
Formalizes emergent intelligence in foundation models as the limit of E(N,P,K) as N,P,K approach infinity, proves existence conditions via nonlinear Lipschitz operators, and derives scaling laws from covering numbers.
This review synthesizes representative advances in high-dimensional statistics, highlights common themes and open problems, and points to key entry works.
citing papers explorer
-
Smooth Scaling Laws Hide Stepwise Token Learning
Power-law LLM scaling laws are largely the aggregate of stepwise token learning events whose heavy-tailed learning-time spectrum reconstructs loss derivatives along step, data, and model axes.
-
Explaining Data Mixing Scaling Laws
Under a shared-head/disjoint-tail assumption, multi-domain loss decomposes into a capacity-competition term c_i x_i^*(h)^{-b_i} plus a per-domain noise term A_i(Dh_i)^{-a_i}, and the fitted law extrapolates optimal mixtures to unseen scales.
-
Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
A solvable hierarchical model with power-law feature strengths yields explicit power-law scaling of prediction error through sequential recovery of latent directions by a layer-wise spectral algorithm.
-
Criticality and Saturation in Orthogonal Neural Networks
Derives layer-wise recursions for finite-width tensors under orthogonal initialization that reproduce the observed large-depth stability of nonlinear networks.
-
Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling
For a sketched bilinear contrastive model trained by full-batch gradient descent, the expected risk decomposes into irreducible, approximation, bias, and variance terms, with variance set by a product effective dimension.
-
Critical Percolation as a Synthetic Data Model for Interpretability
Critical percolation clusters embedded in high dimensions, combined with taxonomic latent variables, form an analytically tractable synthetic data model whose ground-truth hierarchy can be linearly decoded from network activations.
-
Learning Dynamics of Chain-of-Thought State Tracking in a Solvable Transformer Model
Mean-field equations for attention retrieval, teacher alignment, and logic overlap quantitatively match simulations and predict a sharp accuracy transition in a solvable transformer for permutation state tracking.
-
Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail
Larger neural networks keep learning from ever-weaker spectral directions of the empirical neural tangent kernel, a capacity the authors call spectral reach.
-
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models succeed on rare and complex tasks by reducing gradient interference from common tasks, allowing rare-task features to accumulate, as shown via synthetic task mixtures and OLMo pretraining from 4M to 4B parameters.
-
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
Derives α^{-1/3} scaling for generalization error in online softmax classification from boundary layers in a teacher-student model.
-
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
A new scaling law L(N, D, T) = E + (L0 - E) h/(1+h) with h = a/N^α + b/T^β + c N^γ/D^δ that decomposes loss into undercapacity, undertraining, and overfitting terms and saturates between E and L0.
-
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
Spectral analysis of activations and gradients provides new diagnostics that link batch size to representation geometry, early covariance tails to token efficiency, and spectral shifts to learning dynamics in decoder-only LLMs, backed by a mechanistic model.
-
Superposition Yields Robust Neural Scaling
Strong superposition causes neural loss to scale as the inverse of model dimension due to geometric feature overlaps, explaining scaling laws for broad frequency distributions.
-
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.
-
Asymmetric Scaling Laws from Sparse Features
A sparse-activation model predicts double-descent loss with distinct under- and over-parameterized scaling exponents set by sparsity, plus a compute-optimal frontier favoring dataset growth.
-
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Position paper claims fixed exponents in scaling laws arise from generic mechanisms while coefficients vary with data and architecture, making the latter the focus for improvements.
-
A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws
Formalizes emergent intelligence in foundation models as the limit of E(N,P,K) as N,P,K approach infinity, proves existence conditions via nonlinear Lipschitz operators, and derives scaling laws from covering numbers.
-
High-Dimensional Statistics: Reflections on Progress and Open Problems
This review synthesizes representative advances in high-dimensional statistics, highlights common themes and open problems, and points to key entry works.
- Scaling Laws for Dynamic Mini-Batch SGD in Sketched Linear Regression