In a sequential d,n→∞ then m→∞ limit with dm/n fixed, the spectrum of a deep linear Gaussian network's feature covariance converges to the free log-normal law, whose T-transform solves a Burgers equation.
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
8 Pith papers cite this work, alongside 187 external citations. Polarity classification is still indexing.
8
Pith papers citing it
187
external citations · Pith
abstract
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture of particles and a continuous-time gradient descent is performed on their weights and positions. This is an idealization of the usual way to train neural networks with a large hidden layer. We show that, when initialized correctly and in the many-particle limit, this gradient flow, although non-convex, converges to global minimizers. The proof involves Wasserstein gradient flows, a by-product of optimal transport theory. Numerical experiments show that this asymptotic behavior is already at play for a reasonable number of particles, even in high dimension.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Introduces structured DRO for learned inverse problem reconstructions with ambiguity sets aligned to the forward operator, yielding explicit dual representations and a worst-case bound that induces Tikhonov regularization on the operator Lipschitz constant.
Wasserstein policy gradient converges globally in entropy-regularized RL via Bellman-induced distributional PL geometry and uniform LSI, yielding geometric contraction up to discretization bias.
Introduces graph-to-image prediction of per-node dynamic stability landscapes in oscillator networks from topology, releases two 10k-graph datasets, and shows GNN-CNN models achieve good accuracy with cross-size generalization.
ALU uses public data to suppress unlearning cost quadratically while characterizing distribution mismatch effects, enabling mass unlearning with maintained utility.
At critical learning rate 1/N, the rescaled correlation of SGD in single-index models is claimed to converge to a mean-reverting Ornstein-Uhlenbeck process, but the diffusion coefficients are internally inconsistent.
Batch gradient descent achieves linear convergence to zero MSE with high probability for sufficiently wide shallow NNs with non-affine piecewise affine activations and distinct inputs.
Thesis uses statistical mechanics to study DAM and RBM models for understanding memorization, low-dimensional learning, and adversarial robustness in neural networks.
citing papers explorer
-
Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product
In a sequential d,n→∞ then m→∞ limit with dm/n fixed, the spectrum of a deep linear Gaussian network's feature covariance converges to the free log-normal law, whose T-transform solves a Burgers equation.