REVIEW 28 cited by
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
read the original abstract
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.
Forward citations
Cited by 28 Pith papers
-
Neural Spectral Bias and Conformal Correlators I: Introduction and Applications
Neural networks optimized solely on crossing symmetry reconstruct CFT correlators from minimal input data to few-percent accuracy across generalized free fields, minimal models, Ising, N=4 SYM, and AdS diagrams.
-
SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity
Inter-client gradient divergence in federated learning concentrates in low-frequency components; suppressing them via spectral or spatial high-pass filtering reduces client drift and raises accuracy under non-IID data.
-
Correcting Neural Operator Spectral Bias via Diffusion Posterior Sampling with Sparse Observations
FreqNO-DPS corrects neural operator spectral bias in 3D elastic wavefield prediction by frequency-dependent guidance in diffusion posterior sampling conditioned on sparse observations, achieving near-zero bias at 2-5%...
-
Learning on the Temporal Tangent Bundle for Physics-Informed Neural Networks
Parameterizing the temporal derivative in PINNs and reconstructing via Volterra integral yields 100-200x lower errors on advection, Burgers, and Klein-Gordon equations while proving equivalence to the original PDE.
-
FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filtering
Low-frequency components of client-side SAM perturbations carry most inter-client disagreement; high-pass filtering them yields more consistent federated updates and higher accuracy under non-IID data.
-
Spectral Born machines: classically trainable quantum generative models for discrete data
Spectral Born machines are Fourier-phase quantum generative models over Z_d^n that train classically via graph-spectral MMD and show reduced parameters plus apparent overfitting resistance on integer data.
-
Calibrated Harmonic Overlaid Implicit Neural Representations for Multi-Dimensional Data
CHOIR uses coordinated harmonic superposition instead of function composition and perceptual spectrum calibration to improve stability and reduce bias in implicit neural representations for multi-dimensional data recovery.
-
Fourier Feature Pyramids for Physics-Informed Neural Networks
beignet replaces random Fourier feature embeddings in PINNs with a trainable multi-resolution Fourier feature pyramid, achieving higher accuracy on PDE benchmarks with fewer parameters and near machine precision resid...
-
PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting
PnP-Corrector decouples physics simulation from error correction via a plug-and-play agent, cutting error by 29% in 300-day global ocean-atmosphere forecasts.
-
PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting
PnP-Corrector decouples physics simulation from error correction to counter reciprocal error amplification in coupled spatiotemporal forecasting, cutting error by 29% in a 300-day ocean-atmosphere test.
-
Neural Networks Reveal a Universal Bias in Conformal Correlators
Neural networks trained on crossing symmetry accurately reconstruct conformal correlators from minimal inputs due to alignment between their spectral bias and CFT smoothness.
-
Understanding Latent Diffusability via Fisher Geometry
Latent diffusability is quantified by decomposing the MMSE rate along diffusion trajectories into Fisher Information and Fisher Information Rate, with three geometric penalties (dimensional compression, tangential dis...
-
A Greedy PDE Router for Blending Neural Operators and Classical Methods
An approximate greedy router for hybrid PDE solvers that mimics optimal selection without true error access and shows faster, more stable error reduction on test equations.
-
Deep Learning for Subspace Regression
Neural networks regress oversized subspaces for parametric problems using subspace-specific losses, with theory and experiments showing improved accuracy and smoother mappings.
-
WebSailor: Navigating Super-human Reasoning for Web Agent
WebSailor trains open-source web agents to match proprietary performance on complex information-seeking tasks by generating high-uncertainty scenarios and using a new RL method called DUPO.
-
Sinc Kolmogorov-Arnold network and its application for solving PDEs with singularities
SincKANs integrate Sinc interpolation into KAN activations and report better empirical results than alternatives on function approximation and PINN tasks.
-
Theory of the Frequency Principle for General Deep Neural Networks
The paper establishes rigorous theorems proving the Frequency Principle holds for general deep neural networks at initial, intermediate, and final training stages.
-
Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation
SpectraMB performs target-oriented representation purification via dynamic spectral filtering and reliability-aware fusion via global-context attention to address intra-behavior entanglement and inter-behavior heterog...
-
An alternating learning-based collocation method for solving inverse elliptic problems
ALBC solves inverse elliptic problems via alternating updates with adaptive sinusoidal shallow networks, proving convergence and outperforming standard collocation while matching or exceeding PINNs at lower cost on be...
-
Full-field prediction for engineering-scale three-dimensional aircraft with multigrid-hierarchical learning
MHLF combines multigrid geometry representation with hierarchical learning to predict full flow fields for engineering-scale 3D aircraft, accelerating CFD convergence 3-8x across subsonic to supersonic regimes without...
-
Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation
Introduces DCD, a wavelet-based stage-wise distillation technique that preserves structural details in efficient 3D multi-modal MRI segmentation models.
-
Random Neural Network Expressivity for Non-Linear Partial Differential Equations
Random neural networks achieve a dimension-free approximation rate of 1/2 for sufficiently regular time-dependent Sobolev functions and can efficiently approximate solutions to Porous Medium Equations and Compressible...
-
Parity Supervision as a Driver of Generalization in Quantum Generative Modeling
Parity supervision improves exact KL fit and recovery of unseen high-value states in IQP Born machines beyond MSE training or max-entropy controls via parity-moment evidence transfer.
-
Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes
HRGrad resolves gradient conflicts in multi-task learning for asymptotic-preserving neural networks by encoding small parameters and using a gradient alignment metric, enabling stable training across all Knudsen numbe...
-
Spectral methods: crucial for machine learning, natural for quantum computers?
Quantum computers may enable more natural manipulation of Fourier spectra in ML models via the Quantum Fourier Transform, potentially leading to resource-efficient spectral methods.
-
Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysis
Beta Sampling uses spectral analysis to select critical denoising steps in diffusion models, outperforming uniform sampling on FID and IS metrics.
-
PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting
PnP-Corrector decouples pre-trained physics engines from a correction agent to mitigate reciprocal error amplification in coupled spatiotemporal forecasting, cutting error by 28% on a 300-day ocean-atmosphere task.
-
A Practitioner's Guide to Kolmogorov-Arnold Networks
A systematic review of Kolmogorov-Arnold Networks that maps their relation to Kolmogorov superposition theory, MLPs, and kernels, examines basis-function design choices, summarizes performance advances, and supplies a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.