Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
read the original abstract
We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on $L^2$ bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and the law of this sampling scheme. We then apply our results to the homotopy method of arXiv:1907.05600 and provide theoretical justification for its empirical success.
This paper has not been read by Pith yet.
Forward citations
Cited by 12 Pith papers
-
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
PACE is a clipped per-coordinate controller added to AdamW that improves the limiting error of the returned iterate average in both quadratic analysis and LM experiments.
-
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
PACE is an AdamW wrapper derived from optimal control that improves the limiting error of the returned exponential-moving-average model in both theory and LM experiments.
-
Tessellations of Semi-Discrete Flow Matching
Semi-discrete Flow Matching produces terminal assignment regions that are topologically simple (open, simply connected, homeomorphic to the ball under assumption) yet geometrically distinct from optimal transport Lagu...
-
Improved Sample Complexity For Diffusion Model Training Without Empirical Risk Minimizer Access
The paper establishes an O(ε^{-4}) sample complexity bound for score estimation in diffusion models without requiring access to the empirical risk minimizer.
-
From Score Matching to Diffusion: A Fine-Grained Error Analysis in the Gaussian Setting
In the Gaussian setting the Wasserstein error of score-matching-plus-diffusion sampling equals a kernel norm of the data power spectrum whose kernel is determined by the four error sources and the algorithm parameters.
-
Catastrophic Compositional Generation: Why Vanilla Diffusion Models Fail to Extrapolate
Vanilla diffusion models fail at compositional generation because score estimation error dominates and cannot be fixed by inference-time techniques when targets are out-of-distribution.
-
A unified perspective on fine-tuning and sampling with diffusion and flow models
A unified framework for exponential tilting in diffusion and flow models that includes bias-variance decompositions showing finite gradient variance for some methods, norm bounds on adjoint ODEs, and adapted losses wi...
-
Diffusion Model for Manifold Data: Score Decomposition, Curvature, and Statistical Complexity
Diffusion models on manifold-supported data admit score decompositions whose statistical rates are controlled by intrinsic dimension and curvature.
-
On the Limits of Latent Reuse in Diffusion Models
Reusing source latent spaces in diffusion models under distribution shift produces target score error set by principal-angle misalignment and diffusion-time-amplified ambient noise.
-
Structured drift design for denoising diffusion models
Proposes GOU process with anisotropic drift to embed data geometry in diffusion models, claiming better mode separation, correlation preservation, and convergence than isotropic baselines.
-
Statistical Properties of Training & Generalization
Neural scaling laws in deep learning interact with physics constraints and inductive biases beyond classical statistics.
-
Statistical Properties of Training & Generalization
Review of neural scaling laws and their relation to constraints and inductive biases when applying machine learning to physics problems.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.