REVIEW 25 cited by
Feature Learning in Infinite-Width Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTK parametrization). However, we show that the standard and NTK parametrizations of a neural network do not admit infinite-width limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT. We propose simple modifications to the standard parametrization to allow for feature learning in the limit. Using the *Tensor Programs* technique, we derive explicit formulas for such limits. On Word2Vec and few-shot learning on Omniglot via MAML, two canonical tasks that rely crucially on feature learning, we compute these limits exactly. We find that they outperform both NTK baselines and finite-width networks, with the latter approaching the infinite-width feature learning performance as width increases. More generally, we classify a natural space of neural network parametrizations that generalizes standard, NTK, and Mean Field parametrizations. We show 1) any parametrization in this space either admits feature learning or has an infinite-width training dynamics given by kernel gradient descent, but not both; 2) any such infinite-width limit can be computed using the Tensor Programs technique. Code for our experiments can be found at github.com/edwardjhu/TP4.
Forward citations
Cited by 25 Pith papers
-
Precise gradient descent training dynamics for finite-width multi-layer neural networks
Gradient descent iterates of finite-width multi-layer networks on single-index data obey a state evolution law, giving exact training/test error formulas and a data-driven test error estimator.
-
Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Gradient descent on any feedforward network with Lipschitz-smooth, linearly bounded activations drives the minimum squared gradient norm to zero at rate O(1/T^(1/L)) without boundedness or overparameterization assumptions.
-
Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse
Noisy SGD in the mean-field regime forces wide multivariate ReLU networks to an effective width of at most 2P-1, yielding a continuous piecewise-affine predictor whose hyperplanes are non-redundant with respect to the...
-
Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product
In double asymptotic limits, the squared singular value process of non-square matrix products obeys geometric Dyson Brownian motion whose T-transform solves a Burgers equation, producing the free log-normal law via fr...
-
Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
A learned dynamic chunking hierarchy lets byte-level language models match or beat BPE-tokenized Transformers at matched compute, with larger gains on Chinese, code, and DNA.
-
A Classical View on Benign Overfitting: The Role of Sample Size
The paper proves high-probability, non-asymptotic bounds showing that kernel ridge regression and two-layer ReLU networks in the NTK regime can achieve both arbitrarily small training and test error without assuming t...
-
Heterosynaptic Circuits Are Universal Gradient Machines
A two-signal (heterosynaptic) synaptic update rule reduces to preconditioned gradient descent at its stationary point, provided consistency scores across neurons share a sign, unifying Hebbian, anti-Hebbian and hetero...
-
Adaptive kernel predictors from feature-learning infinite limits of neural networks
Feature-learning infinite-width neural networks are kernel machines with data-dependent kernels, defined by a min-max saddle point (Bayesian/Langevin) or a DMFT fixed point (gradient flow with weight decay).
-
From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning
A multi-scale adaptive theory shows that kernel rescaling and directional feature adaptation are two approximations of the same posterior distribution, with differences appearing in output covariances and in non-linea...
-
Slowing Learning by Erasing Simple Features
QLEACE removes all quadratically available class information from a representation, reliably slows feedforward networks, but can inject higher-order information that lets stronger architectures learn faster.
-
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.
-
Reactivation: Empirical NTK Dynamics Under Task Shifts
Task transitions in continual learning cause abrupt, width-persistent changes in the Neural Tangent Kernel of past data, a phenomenon the authors call reactivation, which is modulated by semantic novelty of the new classes.
-
PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
PLoP selects LoRA adapter placement by ranking normalized feature norms and placing adapters on the lowest-scoring module types, using only forward passes.
-
Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
In overparameterized diffusion models, generalization happens first and memorization starts later, with the memorization time growing linearly with dataset size.
-
Block-Biased Mamba for Long-Range Sequence Processing
B2S6, a block-biased variant of Mamba's S6 unit, restores universal approximation, softens input-dependent forgetting, and beats S4/S4D on Long-Range Arena while keeping Mamba-like language perplexity.
-
Learning curves theory for hierarchically compositional data with power-law distributed features
On hierarchical grammar data with Zipf-distributed production rules, classification error decays as P^{-a/(1+a)} while next-token prediction retains a hierarchy-only asymptotic exponent.
-
Parameter Symmetry Potentially Unifies Deep Learning Theory
This position paper argues that parameter symmetry breaking and restoration unify three hierarchies in deep learning: learning dynamics, model complexity, and representation formation.
-
Physics of Skill Learning
The paper introduces Geometry, Resource, and Domino models that reproduce the sequential Domino effect in skill learning and link it to scaling laws, optimizers, and modularity.
-
Pre-Strings Lectures on Artificial Intelligence
Lecture notes define neural-network field theory and survey how it recovers known QFT/string results plus applied AI techniques for string problems.
-
Large-scale artificial intelligence with 41 million nanophotonic neurons on a metasurface
A 10 mm² metasurface with 41 million randomly designed meta-atoms acts as a giant random projector that, with a few thousand trained digital weights, matches ResNet/ViT-class models on medical image tasks.
-
PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model
PERFT inserts routed, parameter-efficient LoRA experts into MoE layers and reports consistent gains over attention-only LoRA on reasoning benchmarks.
-
Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens
Using NTK trace and effective rank, this paper shows that model and data scaling improve test loss at similar rates but drive internal dynamics in opposite directions, and estimates a feature-learning width limit well...
-
Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training
Weight-matrix probability densities in a toy autoencoder are evolved with the Fokker-Planck equation driven by the ADAM update, and the resulting output distributions roughly match training at epoch 5.
-
Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories
The paper reviews fixed-kernel neural network theory and proposes an over-parameterized Gaussian sequence model as a prototype for feature learning.
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
Discussion (0). Continue with ORCID to comment.