REVIEW 7 cited by
Tensor Programs II: Neural Tangent Kernel for Any Architecture
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity. We demonstrate how to calculate this limit. In prior literature, the heuristic study of neural network gradients often assumes every weight matrix used in forward propagation is independent from its transpose used in backpropagation (Schoenholz et al. 2017). This is known as the *gradient independence assumption (GIA)*. We identify a commonly satisfied condition, which we call *Simple GIA Check*, such that the NTK limit calculation based on GIA is correct. Conversely, when Simple GIA Check fails, we show GIA can result in wrong answers. Our material here presents the NTK results of Yang (2019a) in a friendly manner and showcases the *tensor programs* technique for understanding wide neural networks. We provide reference implementations of infinite-width NTKs of recurrent neural network, transformer, and batch normalization at https://github.com/thegregyang/NTK4A.
Forward citations
Cited by 7 Pith papers
-
Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
As ResNets grow deep and wide with fixed dropout rate, dropout training and random-gradient-masking training converge to the same limiting dynamics, and the common masking variants collapse to one limit.
-
Information-Theoretic Limits of Reliability and Scaling in Language Models
A theoretical framework derives a reliability ceiling and a max-form Chinchilla-type scaling law for LLMs from task entropy and dependency spectra.
-
Reactivation: Empirical NTK Dynamics Under Task Shifts
Task transitions in continual learning cause abrupt, width-persistent changes in the Neural Tangent Kernel of past data, a phenomenon the authors call reactivation, which is modulated by semantic novelty of the new classes.
-
Decoupled Relative Learning Rate Schedules
Per-component relative learning-rate schedules speed up Transformer pretraining by up to 23% for MoE models, and the schedules tuned on a 34M model transfer to models 27x larger.
-
Schoenberg characterization of continuous non-stationary isotropic positive definite kernels
Continuous isotropic positive definite kernels on R^d have the form K(x,y) = sum of alpha_n(||x||,||y||) times Gegenbauer polynomials in <x/||x||, y/||y||>, with alpha_n positive definite and zero at the origin.
-
Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens
Using NTK trace and effective rank, this paper shows that model and data scaling improve test loss at similar rates but drive internal dynamics in opposite directions, and estimates a feature-learning width limit well...
-
LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs
Under a shared retrieval-augmented memory, multiple LLMs' outputs converge to near-identical semantic answers, and the analogous Gaussian mixture system is proven to collapse.
Discussion (0). Continue with ORCID to comment.