The paper establishes rigorous theorems proving the Frequency Principle holds for general deep neural networks at initial, intermediate, and final training stages.
Mean Field Analysis of Neural Networks: A Central Limit Theorem
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
We rigorously prove a central limit theorem for neural network models with a single hidden layer. The central limit theorem is proven in the asymptotic regime of simultaneously (A) large numbers of hidden units and (B) large numbers of stochastic gradient descent training iterations. Our result describes the neural network's fluctuations around its mean-field limit. The fluctuations have a Gaussian distribution and satisfy a stochastic partial differential equation. The proof relies upon weak convergence methods from stochastic analysis. In particular, we prove relative compactness for the sequence of processes and uniqueness of the limiting process in a suitable Sobolev space.
verdicts
UNVERDICTED 2representative citing papers
At the critical step-size scaling for SGD in high-dimensional single-layer networks, effective dynamics gain a diffusive correction term that changes the phase diagram and reduces to an Ornstein-Uhlenbeck process near fixed points, with the information exponent governing sample complexity.
citing papers explorer
-
Theory of the Frequency Principle for General Deep Neural Networks
The paper establishes rigorous theorems proving the Frequency Principle holds for general deep neural networks at initial, intermediate, and final training stages.
-
Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks
At the critical step-size scaling for SGD in high-dimensional single-layer networks, effective dynamics gain a diffusive correction term that changes the phase diagram and reduces to an Ornstein-Uhlenbeck process near fixed points, with the information exponent governing sample complexity.