REVIEW 11 cited by
On Learning Gaussian Multi-index Models with Gradient Flow
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data. Multi-index functions consist of a composition of an unknown low-rank linear projection and an arbitrary unknown, low-dimensional link function. As such, they constitute a natural template for feature learning in neural networks. We consider a two-timescale algorithm, whereby the low-dimensional link function is learnt with a non-parametric model infinitely faster than the subspace parametrizing the low-rank projection. By appropriately exploiting the matrix semigroup structure arising over the subspace correlation matrices, we establish global convergence of the resulting Grassmannian population gradient flow dynamics, and provide a quantitative description of its associated `saddle-to-saddle' dynamics. Notably, the timescales associated with each saddle can be explicitly characterized in terms of an appropriate Hermite decomposition of the target link function. In contrast with these positive results, we also show that the related \emph{planted} problem, where the link function is known and fixed, in fact has a rough optimization landscape, in which gradient flow dynamics might get trapped with high probability.
Forward citations
Cited by 11 Pith papers
-
Online Learning of Neural Networks
Online mistake bounds for sign-activation feedforward networks are governed by a totally-separable packing number of the first layer margin, with dimension-free bounds under multi-index or all-layer margin assumptions.
-
Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery
For multi-index models, the eigenvalues, eigenvector overlaps, and optimal preprocessing of spectral estimators are characterized exactly in the proportional asymptotics.
-
Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Gradient-based dataset distillation of two-layer ReLU nets on multi-index models encodes the r-dimensional principal subspace into synthetic data of memory complexity Θ̃(r²d+L) that recovers high generalization.
-
Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime
In the teacher-student setting, variable-projection training of two-layer networks is shown to match a weighted ultra-fast diffusion in the zero-regularization limit, giving linear convergence of the learned feature d...
-
Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributions
A random shift of Gaussian inputs forces the first Hermite coefficient of any non-linear target to be large, yielding near-linear sample complexity independent of the target's information exponent, and a similar resul...
-
The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning
Online SGD on the correlation loss recovers Multiscale Single-Index Model features at n=Õ(d^{K-1}) samples, matching Tensor PCA, while shallow nets cannot approximate the target under higher-chaos non-cancellation.
-
Joint Learning in the Gaussian Single Index Model
In Gaussian single-index models, joint gradient flow over direction and link function converges to the true regression function from either sign of initial alignment, with rate governed by the information exponent.
-
Optimal Spectral Transitions in High-Dimensional Multi-Index Models
Two linearized message-passing spectral estimators achieve the optimal weak-recovery threshold in Gaussian multi-index models, with a sharp BBP-like spectral phase transition at the critical sample complexity.
-
LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Initializing LoRA adapters from the SVD of the first full fine-tuning gradient yields subspace alignment and fast convergence in theory, and the resulting LoRA-One method beats standard LoRA on several LLM benchmarks.
-
Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networks
A three-layer network with layerwise gradient descent provably recovers the span of multiple quadratic features in O~(d^4) samples and then learns any polynomial link in the features.
-
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence
For orthogonal hidden directions, gradient flow provably sends each neuron to the nearest direction and a log-factor overparameterization suffices, but for equiangular directions with overlap above beta_c = (p*-2)/(k+...
Discussion (0). Continue with ORCID to comment.