REVIEW 12 cited by
Towards Understanding the Spectral Bias of Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks. Finally, we provide numerical experiments to demonstrate the correctness of our theory. Our experimental results also show that our theory can tolerate certain model misspecification in terms of the input data distribution.
Forward citations
Cited by 12 Pith papers
-
Neural Spectral Bias and Conformal Correlators I: Introduction and Applications
Simple feed-forward neural networks trained on crossing symmetry plus a single anchor value reproduce CFT correlators to percent-level accuracy, and the authors conjecture this works because physical correlators are t...
-
SONAR: Spectral-Contrastive Audio Residuals for Generalizable Deepfake Detection
SONAR improves audio deepfake detection by explicitly aligning low- and high-frequency representations for real speech and repelling them for fakes, setting new benchmark EERs on ASVspoof 2021 and in-the-wild data.
-
Structure-Preserving Patch Decoding for Efficient Neural Video Representation
Splitting video frames with PixelUnshuffle into structure-preserving patches and decoding them with a global-to-local network improves INR video reconstruction over NeRV-style baselines.
-
Physics-informed neural network estimation of active material properties in time-dependent cardiac biomechanical models
A PINN-based method recovers active contractility parameters and detects fibrotic scars in simulated cardiac tissue from sparse displacement and strain data, with a few percent error in homogeneous tests.
-
Representation Learning for Tabular Data: A Comprehensive Survey
A comprehensive survey that categorizes deep tabular representation learning into specialized, transferable, and general models, with a feature/sample/objective taxonomy for specialized methods.
-
SNeRV: Spectra-preserving Neural Representation for Video
SNeRV decomposes frames with wavelet transforms, embeds only low-frequency content, and regenerates high-frequency details, outperforming prior NeRV models on reconstruction and interpolation.
-
F$^3$OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics
F3OCUS combines per-client LNTK layer importance scores with server-side meta-heuristic optimization of layer diversity to improve federated fine-tuning of vision-language models for medical tasks, and releases the 70...
-
SpiS-GAN: Spiral-Modulated Handwriting Synthesis with Star Operation
A GAN with elliptical-spiral feature mixing, star-operation blocks, and Sobel edge loss produces more realistic synthetic handwriting and lowers HTR error rates on English and Vietnamese datasets.
-
FW-GAN: Frequency-Driven Handwriting Synthesis with Wave-Modulated MLP Generator
FW-GAN combines wave-modulated MLP generation with frequency-domain losses and a wavelet-based discriminator to synthesize handwriting from one example, reporting lower FID scores than prior methods on IAM and HANDS-VNOnDB.
-
Split Complex-Valued Physics-Informed Neural Networks for Forward and Inverse Nonlinear PDEs
Split complex-valued PINNs achieve lower benchmark errors than real-valued PINNs, but the comparison is confounded by doubled parameters and unresolved internal error inconsistencies.
-
SUPER Module for Detail-Sensitive and Cost-Efficient U-Net Variant Decoders
A plug-in wavelet-domain decoder block improves thin-crack IoU on one self-baseline benchmark, while the abstract's flagship depth-estimation gains and decoder MAC reductions are absent from the main text.
-
Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting
The proposed spectral distillation method is not actually evaluated in the experiments, and the abstract's headline error reductions contradict the reported tables.
Discussion (0). Continue with ORCID to comment.