REVIEW 6 cited by
On the Activation Function Dependence of the Spectral Bias of Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Neural networks are universal function approximators which are known to generalize well despite being dramatically overparameterized. We study this phenomenon from the point of view of the spectral bias of neural networks. Our contributions are two-fold. First, we provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. Second, based upon this theory we predict that switching the activation function to a piecewise linear B-spline, namely the Hat function, will remove this spectral bias, which we verify empirically in a variety of settings. Our empirical studies also show that neural networks with the Hat activation function are trained significantly faster using stochastic gradient descent and ADAM. Combined with previous work showing that the Hat activation function also improves generalization accuracy on image classification tasks, this indicates that using the Hat activation provides significant advantages over the ReLU on certain problems.
Forward citations
Cited by 6 Pith papers
-
Orthogonal greedy algorithm for linear operator learning with shallow neural network
Orthogonal greedy training of shallow ReLU networks is adapted to kernel estimation for linear operators, with stated convergence rates and large accuracy gains over neural operator baselines.
-
High precision PINNs in unbounded domains: application to singularity formulation in PDEs
A modular PINN recipe (sinh sampling, hard parity and nondegeneracy constraints, exact far-field asymptotics, self-scaled BFGS) yields much lower PDE residuals for self-similar blowup profiles of the 1D Burgers and 2D...
-
Leveraging KANs for Expedient Training of Multichannel MLPs via Preconditioning and Geometric Refinement
Training in a B-spline KAN basis is equivalent to preconditioned gradient descent on a multichannel ReLU MLP, and geometric refinement plus trainable knots accelerate and improve training.
-
Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations
Comparing fixed and learnable activations in PINNs across five PDEs shows learnable bases help in small networks, destabilize large ones, and low spectral bias does not guarantee accuracy.
-
Separated-Variable Spectral Neural Networks: A Physics-Informed Learning Approach for High-Frequency PDEs
A separable Fourier-feature neural network with learnable frequencies and a three-level frequency sampler is reported to solve high-frequency PDEs with far fewer parameters than vanilla PINNs.
-
Momentum-Accelerated Richardson(m) and Their Multilevel Neural Solvers
Neural networks predict the weights of momentum-accelerated long-step Richardson iterations, reducing iteration counts on anisotropic diffusion and Helmholtz problems versus Chebyshev-based iterations in numerical exp...
Discussion (0). Continue with ORCID to comment.