The power of deeper networks for expressing natural functions
read the original abstract
It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons $m$ required to approximate natural classes of multivariate polynomials of $n$ variables grows only linearly with $n$ for deep neural networks, but grows exponentially when merely a single hidden layer is allowed. We also provide evidence that when the number of hidden layers is increased from $1$ to $k$, the neuron requirement grows exponentially not with $n$ but with $n^{1/k}$, suggesting that the minimum number of layers required for practical expressibility grows only logarithmically with $n$.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Fixed-order PCA: Theory for Overestimated Factor Models
Establishes asymptotic consistency of factor estimates and √T-normality in factor-augmented regressions for fixed R ≥ r using anisotropic local laws from random matrix theory.
-
MIDUS: Memory-Infused Depth Up-Scaling
MIDUS replaces duplicated FFN branches in depth up-scaling with head-wise memory layers using product-key retrieval and HIVE to deliver lightweight, head-conditioned residual capacity.
-
Complexity of Linear Regions in Self-supervised Deep ReLU Networks
Self-supervised ReLU networks form substantially fewer linear regions than supervised models for comparable accuracy, with contrastive methods rapidly expanding regions and self-distillation consolidating them, enabli...
-
Monetary Policy in the Media Spotlight: Sentiments, Signals, and Economic Impact
Media sentiment indicators from Canadian news, when added to a New Keynesian model with endogenous central-bank response, improve out-of-sample forecasts and account for part of monetary-policy propagation to output a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.