REVIEW 4 cited by
On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We develop Banach spaces for ReLU neural networks of finite depth $L$ and infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the unit ball in the space for $L$-layer networks has low Rademacher complexity and thus favorable generalization properties. Functions in these spaces can be approximated by multi-layer neural networks with dimension-independent convergence rates. The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent dynamics for the associated multi-layer neural networks. We show that the path-norm increases at most polynomially under this continuous gradient flow dynamics.
Forward citations
Cited by 4 Pith papers
-
The Barron-Lipschitz Energy Gap and Depth Separation Phenomena in Scientific Machine Learning
Barron functions can fail to reach the Lipschitz-class infimum of certain variational energies—including a thin-shell folding energy where circular folds beat straight-line folds—while compositions of two Barron funct...
-
A First-Principles Theory of Slow Thinking and Active Perception
Active lifting of data distributions via latent-sequence sampling and max-rate uncertainty reduction formally derives slow-thinking LLMs and places them on representation and sampler hierarchies that can be climbed.
-
Convergence of Time-Averaged Mean Field Gradient Descent Dynamics for Continuous Multi-Player Zero-Sum Games
An exponentially discounted time-averaged mean-field gradient descent flow reaches the entropy-regularized mixed Nash equilibrium exponentially fast, and its annealed version reaches the unregularized equilibrium at r...
-
Solving the Poisson Equation with Dirichlet data by shallow ReLU$^\alpha$-networks: A regularity and approximation perspective
For half-plane Dirichlet data equal to ReLU^alpha, the harmonic solution lies in the ReLU^alpha-Barron space for 0<alpha<1, fails to lie in it for integer alpha, and admits approximations whose Barron norm grows only ...
Discussion (0). Continue with ORCID to comment.