FreqNO-DPS corrects neural operator spectral bias in 3D elastic wavefield prediction by frequency-dependent guidance in diffusion posterior sampling conditioned on sparse observations, achieving near-zero bias at 2-5% sensor coverage.
hub Canonical reference
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
Canonical reference. 100% of citing Pith papers cite this work as background.
24
Pith papers citing it
Background
100% of classified citations
abstract
We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.
hub tools
citation-role summary
background 5
citation-polarity summary
fields
cs.LG 8 cs.CV 3 math.NA 3 quant-ph 3 hep-th 2 cs.AI 1 cs.CL 1 cs.IR 1 physics.flu-dyn 1 stat.ME 1roles
background 5polarities
background 5representative citing papers
Parameterizing the temporal derivative in PINNs and reconstructing via Volterra integral yields 100-200x lower errors on advection, Burgers, and Klein-Gordon equations while proving equivalence to the original PDE.
Spectral Born machines are Fourier-phase quantum generative models over Z_d^n that train classically via graph-spectral MMD and show reduced parameters plus apparent overfitting resistance on integer data.
CHOIR uses coordinated harmonic superposition instead of function composition and perceptual spectrum calibration to improve stability and reduce bias in implicit neural representations for multi-dimensional data recovery.
beignet replaces random Fourier feature embeddings in PINNs with a trainable multi-resolution Fourier feature pyramid, achieving higher accuracy on PDE benchmarks with fewer parameters and near machine precision residuals on the inviscid Burgers blowup using Adam.
Parity-moment supervision improves forward-KL fit and unseen high-value-state recovery over coordinate-wise MSE in a controlled 12-qubit IQP Born-machine benchmark.
Simple feed-forward neural networks trained on crossing symmetry plus a single anchor value reproduce CFT correlators to percent-level accuracy, and the authors conjecture this works because physical correlators are the smoothest allowed functions.
Simple neural networks trained on crossing symmetry and one anchor point reproduce conformal correlators to within a few percent across many CFTs.
Latent diffusability is quantified by decomposing the MMSE rate along diffusion trajectories into Fisher Information and Fisher Information Rate, with three geometric penalties (dimensional compression, tangential distortion, curvature injection) identified as sources of failure.
An approximate greedy router for hybrid PDE solvers that mimics optimal selection without true error access and shows faster, more stable error reduction on test equations.
Neural networks regress oversized subspaces for parametric problems using subspace-specific losses, with theory and experiments showing improved accuracy and smoother mappings.
WebSailor trains open-source web agents to match proprietary performance on complex information-seeking tasks by generating high-uncertainty scenarios and using a new RL method called DUPO.
SpectraMB performs target-oriented representation purification via dynamic spectral filtering and reliability-aware fusion via global-context attention to address intra-behavior entanglement and inter-behavior heterogeneity in multi-behavior recommendation.
ALBC solves inverse elliptic problems via alternating updates with adaptive sinusoidal shallow networks, proving convergence and outperforming standard collocation while matching or exceeding PINNs at lower cost on benchmarks with up to 20% noise.
MHLF combines multigrid geometry representation with hierarchical learning to predict full flow fields for engineering-scale 3D aircraft, accelerating CFD convergence 3-8x across subsonic to supersonic regimes without accuracy loss.
Introduces DCD, a wavelet-based stage-wise distillation technique that preserves structural details in efficient 3D multi-modal MRI segmentation models.
Random neural networks achieve a dimension-free approximation rate of 1/2 for sufficiently regular time-dependent Sobolev functions and can efficiently approximate solutions to Porous Medium Equations and Compressible Navier-Stokes Equations.
HRGrad resolves gradient conflicts in multi-task learning for asymptotic-preserving neural networks by encoding small parameters and using a gradient alignment metric, enabling stable training across all Knudsen numbers for BGK and linear transport equations.
Quantum computers may enable more natural manipulation of Fourier spectra in ML models via the Quantum Fourier Transform, potentially leading to resource-efficient spectral methods.
Beta Sampling uses spectral analysis to select critical denoising steps in diffusion models, outperforming uniform sampling on FID and IS metrics.
PnP-Corrector decouples pre-trained physics engines from a correction agent to mitigate reciprocal error amplification in coupled spatiotemporal forecasting, cutting error by 28% on a 300-day ocean-atmosphere task.
A systematic review of Kolmogorov-Arnold Networks that maps their relation to Kolmogorov superposition theory, MLPs, and kernels, examines basis-function design choices, summarizes performance advances, and supplies a practitioner's selection guide plus open challenges.
citing papers explorer
-
Correcting Neural Operator Spectral Bias via Diffusion Posterior Sampling with Sparse Observations
FreqNO-DPS corrects neural operator spectral bias in 3D elastic wavefield prediction by frequency-dependent guidance in diffusion posterior sampling conditioned on sparse observations, achieving near-zero bias at 2-5% sensor coverage.
-
Learning on the Temporal Tangent Bundle for Physics-Informed Neural Networks
Parameterizing the temporal derivative in PINNs and reconstructing via Volterra integral yields 100-200x lower errors on advection, Burgers, and Klein-Gordon equations while proving equivalence to the original PDE.
-
Spectral Born machines: classically trainable quantum generative models for discrete data
Spectral Born machines are Fourier-phase quantum generative models over Z_d^n that train classically via graph-spectral MMD and show reduced parameters plus apparent overfitting resistance on integer data.
-
Calibrated Harmonic Overlaid Implicit Neural Representations for Multi-Dimensional Data
CHOIR uses coordinated harmonic superposition instead of function composition and perceptual spectrum calibration to improve stability and reduce bias in implicit neural representations for multi-dimensional data recovery.
-
Fourier Feature Pyramids for Physics-Informed Neural Networks
beignet replaces random Fourier feature embeddings in PINNs with a trainable multi-resolution Fourier feature pyramid, achieving higher accuracy on PDE benchmarks with fewer parameters and near machine precision residuals on the inviscid Burgers blowup using Adam.
-
Parity Supervision as a Driver of Generalization in Quantum Generative Modeling
Parity-moment supervision improves forward-KL fit and unseen high-value-state recovery over coordinate-wise MSE in a controlled 12-qubit IQP Born-machine benchmark.
-
Neural Spectral Bias and Conformal Correlators I: Introduction and Applications
Simple feed-forward neural networks trained on crossing symmetry plus a single anchor value reproduce CFT correlators to percent-level accuracy, and the authors conjecture this works because physical correlators are the smoothest allowed functions.
-
Neural Networks Reveal a Universal Bias in Conformal Correlators
Simple neural networks trained on crossing symmetry and one anchor point reproduce conformal correlators to within a few percent across many CFTs.
-
Understanding Latent Diffusability via Fisher Geometry
Latent diffusability is quantified by decomposing the MMSE rate along diffusion trajectories into Fisher Information and Fisher Information Rate, with three geometric penalties (dimensional compression, tangential distortion, curvature injection) identified as sources of failure.
-
A Greedy PDE Router for Blending Neural Operators and Classical Methods
An approximate greedy router for hybrid PDE solvers that mimics optimal selection without true error access and shows faster, more stable error reduction on test equations.
-
Deep Learning for Subspace Regression
Neural networks regress oversized subspaces for parametric problems using subspace-specific losses, with theory and experiments showing improved accuracy and smoother mappings.
-
WebSailor: Navigating Super-human Reasoning for Web Agent
WebSailor trains open-source web agents to match proprietary performance on complex information-seeking tasks by generating high-uncertainty scenarios and using a new RL method called DUPO.
-
Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation
SpectraMB performs target-oriented representation purification via dynamic spectral filtering and reliability-aware fusion via global-context attention to address intra-behavior entanglement and inter-behavior heterogeneity in multi-behavior recommendation.
-
An alternating learning-based collocation method for solving inverse elliptic problems
ALBC solves inverse elliptic problems via alternating updates with adaptive sinusoidal shallow networks, proving convergence and outperforming standard collocation while matching or exceeding PINNs at lower cost on benchmarks with up to 20% noise.
-
Full-field prediction for engineering-scale three-dimensional aircraft with multigrid-hierarchical learning
MHLF combines multigrid geometry representation with hierarchical learning to predict full flow fields for engineering-scale 3D aircraft, accelerating CFD convergence 3-8x across subsonic to supersonic regimes without accuracy loss.
-
Detail Consistent Stage-Wise Distillation for Efficient 3D MRI Segmentation
Introduces DCD, a wavelet-based stage-wise distillation technique that preserves structural details in efficient 3D multi-modal MRI segmentation models.
-
Random Neural Network Expressivity for Non-Linear Partial Differential Equations
Random neural networks achieve a dimension-free approximation rate of 1/2 for sufficiently regular time-dependent Sobolev functions and can efficiently approximate solutions to Porous Medium Equations and Compressible Navier-Stokes Equations.
-
Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes
HRGrad resolves gradient conflicts in multi-task learning for asymptotic-preserving neural networks by encoding small parameters and using a gradient alignment metric, enabling stable training across all Knudsen numbers for BGK and linear transport equations.
-
Spectral methods: crucial for machine learning, natural for quantum computers?
Quantum computers may enable more natural manipulation of Fourier spectra in ML models via the Quantum Fourier Transform, potentially leading to resource-efficient spectral methods.
-
Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysis
Beta Sampling uses spectral analysis to select critical denoising steps in diffusion models, outperforming uniform sampling on FID and IS metrics.
-
PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting
PnP-Corrector decouples pre-trained physics engines from a correction agent to mitigate reciprocal error amplification in coupled spatiotemporal forecasting, cutting error by 28% on a 300-day ocean-atmosphere task.
-
A Practitioner's Guide to Kolmogorov-Arnold Networks
A systematic review of Kolmogorov-Arnold Networks that maps their relation to Kolmogorov superposition theory, MLPs, and kernels, examines basis-function design choices, summarizes performance advances, and supplies a practitioner's selection guide plus open challenges.
- Sinc Kolmogorov-Arnold network and its application for solving PDEs with singularities
- Theory of the Frequency Principle for General Deep Neural Networks