Gradient descent from moderately small random initialization jointly trains both layers of a ReLU network and converges linearly to the global minimizer for a linear target at order-wise optimal sample complexity by progressing through alignment, growth, and refinement phases.
When Are Nonconvex Problems Not Scary?
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
In this note, we focus on smooth nonconvex optimization problems that obey: (1) all local minimizers are also global; and (2) around any saddle point or local maximizer, the objective has a negative directional curvature. Concrete applications such as dictionary learning, generalized phase retrieval, and orthogonal tensor decomposition are known to induce such structures. We describe a second-order trust-region algorithm that provably converges to a global minimizer efficiently, without special initializations. Finally we highlight alternatives, and open problems in this direction.
years
2026 3representative citing papers
The book presents principles from optimization and information theory to explain deep network architectures and enable new interpretable models.
citing papers explorer
-
When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks
Gradient descent from moderately small random initialization jointly trains both layers of a ReLU network and converges linearly to the global minimizer for a linear target at order-wise optimal sample complexity by progressing through alignment, growth, and refinement phases.
-
Principles and Practice of Deep Representation Learning: or a Mathematical Theory of Memory
The book presents principles from optimization and information theory to explain deep network architectures and enable new interpretable models.
- Convergence of difference inclusions: a diameter criterion and step-size conditions