REVIEW 7 cited by
Expressivity and Approximation Properties of Deep Neural Networks with ReLU$^k$ Activation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In this paper, we investigate the expressivity and approximation properties of deep neural networks employing the ReLU$^k$ activation function for $k \geq 2$. Although deep ReLU networks can approximate polynomials effectively, deep ReLU$^k$ networks have the capability to represent higher-degree polynomials precisely. Our initial contribution is a comprehensive, constructive proof for polynomial representation using deep ReLU$^k$ networks. This allows us to establish an upper bound on both the size and count of network parameters. Consequently, we are able to demonstrate a suboptimal approximation rate for functions from Sobolev spaces as well as for analytic functions. Additionally, through an exploration of the representation power of deep ReLU$^k$ networks for shallow networks, we reveal that deep ReLU$^k$ networks can approximate functions from a range of variation spaces, extending beyond those generated solely by the ReLU$^k$ activation function. This finding demonstrates the adaptability of deep ReLU$^k$ networks in approximating functions within various variation spaces.
Forward citations
Cited by 7 Pith papers
-
Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations
An abstract framework for neural flows with composition and separation structures is proven to universally approximate any operator, recovering ResNet and plain architectures via discretization.
-
Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations
Neural flow operators with composition and separation structures are proven to universally approximate any operator in finite and infinite dimensions, recovering ResNet-type and plain architectures via time discretizations.
-
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
Shallow ReLU^s networks achieve specific improved approximation rates in L^p spaces and minimax-optimal generalization in nonparametric regression over Barron and Sobolev spaces with path-norm control.
-
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
Shallow ReLU^s networks achieve improved approximation rates in L^p spaces for p below a dimension-dependent threshold and minimax-optimal generalization bounds in Barron and Sobolev spaces under l1 path-norm control.
-
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
Derives approximation bounds for shallow ReLU^s networks in L^p and Sobolev spaces and shows path-norm regularized networks achieve minimax optimal generalization rates in regression.
-
Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
Derives explicit approximation rates for shallow ReLU^s networks in L^p and Sobolev spaces and shows path-norm regularized networks achieve minimax-optimal generalization rates with matching lower bounds.
-
Digital Twin Channel-Aided CSI Prediction: An Environment-Based Subspace Extraction Approach for Achieving Low Overhead and High Robustness
Using a digital-twin-derived channel subspace basis as a prior, the method predicts full spatial-frequency CSI from partial pilots, claiming up to 50% pilot overhead reduction and one-coherence-time-ahead prediction i...
Discussion (0). Sign in to comment.