Bayesian diffusion models memorize training data when mutual information between restricted observations and training data exceeds log dataset size, and generalize otherwise.
arXiv preprint arXiv:2305.14712 , year=
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
abstract
The diffusion probabilistic generative models are widely used to generate high-quality data. Though they can synthetic data that does not exist in the training set, the rationale behind such generalization is still unexplored. In this paper, we formally define the generalization of the generative model, which is measured by the mutual information between the generated data and the training set. The definition originates from the intuition that the model which generates data with less correlation to the training set exhibits better generalization ability. Meanwhile, we show that for the empirical optimal diffusion model, the data generated by a deterministic sampler are all highly related to the training set, thus poor generalization. This result contradicts the observation of the trained diffusion model's (approximating empirical optima) extrapolation ability (generating unseen data). To understand this contradiction, we empirically verify the difference between the sufficiently trained diffusion model and the empirical optima. We found, though obtained through sufficient training, there still exists a slight difference between them, which is critical to making the diffusion model generalizable. Moreover, we propose another training objective whose empirical optimal solution has no potential generalization problem. We empirically show that the proposed training objective returns a similar model to the original one, which further verifies the generalization ability of the trained diffusion model.
representative citing papers
Filtered Posterior Mean Collections unify prior models of diffusion generalization and demonstrate performance gains through soft relaxations and source distribution augmentations on image datasets.
Higher-order Langevin dynamics reduce memorization in diffusion models by governing data trajectories with a low-pass-filtered score whose smoothness increases with order.
Generative models learn rules before memorizing data, creating an innovation window whose width depends on dataset size and rule complexity, observed in both diffusion and autoregressive architectures.
A PDE framework using Li-Yau inequalities proves well-posedness and sharp stability for score-based Fokker-Planck dynamics, with reverse-time trajectories concentrating on compactly supported data manifolds at rate sqrt(t).
In the oracle continuous-time setting, stochastic interpolation models recover training samples exactly, with deviations controlled by discretization and estimation errors, leading to theoretical definitions of overfitting and underfitting.
citing papers explorer
-
An exact information theory of generalization phase transitions in Bayesian diffusion models
Bayesian diffusion models memorize training data when mutual information between restricted observations and training data exceeds log dataset size, and generalize otherwise.
-
Filtered Posterior Mean Collections: A Unified Framework for Analytical Models of Diffusion Generalization
Filtered Posterior Mean Collections unify prior models of diffusion generalization and demonstrate performance gains through soft relaxations and source distribution augmentations on image datasets.
-
Reducing Diffusion Model Memorization with Higher Order Langevin Dynamics
Higher-order Langevin dynamics reduce memorization in diffusion models by governing data trajectories with a low-pass-filtered score whose smoothness increases with order.
-
The two clocks and the innovation window: When and how generative models learn rules
Generative models learn rules before memorizing data, creating an innovation window whose width depends on dataset size and rule complexity, observed in both diffusion and autoregressive architectures.
-
A PDE Perspective on Generative Diffusion Models
A PDE framework using Li-Yau inequalities proves well-posedness and sharp stability for score-based Fokker-Planck dynamics, with reverse-time trajectories concentrating on compactly supported data manifolds at rate sqrt(t).
-
A Theoretical Analysis of Memory and Overfitting Phenomena in Stochastic Interpolation Models
In the oracle continuous-time setting, stochastic interpolation models recover training samples exactly, with deviations controlled by discretization and estimation errors, leading to theoretical definitions of overfitting and underfitting.