REVIEW 6 cited by
The Hidden Linear Structure in Score-Based Models and its Application
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
The Hidden Linear Structure in Score-Based Models and its Application
read the original abstract
Score-based models have achieved remarkable results in the generative modeling of many domains. By learning the gradient of smoothed data distribution, they can iteratively generate samples from complex distribution e.g. natural images. However, is there any universal structure in the gradient field that will eventually be learned by any neural network? Here, we aim to find such structures through a normative analysis of the score function. First, we derived the closed-form solution to the scored-based model with a Gaussian score. We claimed that for well-trained diffusion models, the learned score at a high noise scale is well approximated by the linear score of Gaussian. We demonstrated this through empirical validation of pre-trained images diffusion model and theoretical analysis of the score function. This finding enabled us to precisely predict the initial diffusion trajectory using the analytical solution and to accelerate image sampling by 15-30\% by skipping the initial phase without sacrificing image quality. Our finding of the linear structure in the score-based model has implications for better model design and data pre-processing.
Forward citations
Cited by 6 Pith papers
-
Diffusion Models, Denoiser Architecture and Creativity
Creativity in diffusion models stems from denoiser architecture interacting with the target distribution, shown via explicit generated-sample distributions for linear, polynomial, and bottleneck denoisers plus UNET ab...
-
Geometry-Aware Discretization Error of Diffusion Models
First-order asymptotic expansions of weak and Fréchet discretization errors in diffusion sampling are derived, explicit under Gaussian data through covariance geometry and robust to other data geometries.
-
Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging
Diffusion trajectory distillation is reframed as operator merging, yielding an optimal variance-driven merging strategy via Pareto dynamic programming in the linear Gaussian case and unavoidable approximation errors f...
-
From Score Matching to Diffusion: A Fine-Grained Error Analysis in the Gaussian Setting
In the Gaussian setting the Wasserstein error of score-matching-plus-diffusion sampling equals a kernel norm of the data power spectrum whose kernel is determined by the four error sources and the algorithm parameters.
-
Diffusion Models, Denoiser Architecture and Creativity
Diffusion models generate novel samples due to the interaction between denoiser architecture inductive bias and target distribution, with explicit generated distributions derived for linear, polynomial, and bottleneck...
-
The two clocks and the innovation window: When and how generative models learn rules
Generative models learn rules before memorizing data, creating an innovation window whose width depends on dataset size and rule complexity, observed in both diffusion and autoregressive architectures.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.