REVIEW 7 cited by
The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about which functions are learnable compared to single-pass gradient descent. In particular, multi-pass GD with finite stepsize is found to overcome the limitations of gradient flow and single-pass GD given by the information exponent (Ben Arous et al., 2021) and leap exponent (Abbe et al., 2023) of the target function. We show that upon re-using batches, the network achieves in just two time steps an overlap with the target subspace even for functions not satisfying the staircase property (Abbe et al., 2021). We characterize the (broad) class of functions efficiently learned in finite time. The proof of our results is based on the analysis of the Dynamical Mean-Field Theory (DMFT). We further provide a closed-form description of the dynamical process of the low-dimensional projections of the weights, and numerical experiments illustrating the theory.
Forward citations
Cited by 7 Pith papers
-
Online Learning of Neural Networks
Online mistake bounds for sign-activation feedforward networks are governed by a totally-separable packing number of the first layer margin, with dimension-free bounds under multi-index or all-layer margin assumptions.
-
Approximate Message Passing with Random Initialization for Phase Retrieval
Randomly initialized Bayes-optimal AMP provably achieves the weak-recovery threshold δ=1/2 and arbitrarily accurate recovery for δ>1.13 in proportional-regime noiseless phase retrieval.
-
Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributions
A random shift of Gaussian inputs forces the first Hermite coefficient of any non-linear target to be large, yielding near-linear sample complexity independent of the target's information exponent, and a similar resul...
-
Gradient dynamics for low-rank fine-tuning beyond kernels
In a student-teacher model with Gaussian inputs and a rank-1 teacher perturbation, online SGD converges to the teacher in d k^{O(1)} iterations, independent of the activation's Hermite information exponent.
-
Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
As ResNets grow deep and wide with fixed dropout rate, dropout training and random-gradient-masking training converge to the same limiting dynamics, and the common masking variants collapse to one limit.
-
Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networks
A three-layer network with layerwise gradient descent provably recovers the span of multiple quadratic features in O~(d^4) samples and then learns any polynomial link in the features.
-
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence
For orthogonal hidden directions, gradient flow provably sends each neuron to the nearest direction and a log-factor overparameterization suffices, but for equiangular directions with overlap above beta_c = (p*-2)/(k+...
Discussion (0). Continue with ORCID to comment.