Bayesian diffusion models memorize training data when mutual information between restricted observations and training data exceeds log dataset size, and generalize otherwise.
Dynamical regimes of diffusion models
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.
representative citing papers
The paper establishes a reverse-time quantum diffusion framework that generates complex quantum ensembles from simple distributions by deriving and learning a feedback Hamiltonian from forward trajectory data.
Flow models reach 99.2% Sudoku accuracy in 7 passes and 96.1% on out-of-distribution Sudoku-Extreme by selecting dynamically stable candidates and training with self-conditioning plus DPO to avoid failed outputs.
SOW uses MLLMs and attention to selectively control unidirectional diffusion for pixel-level fidelity and contextual coherence in text-vision-to-image tasks.
citing papers explorer
-
An exact information theory of generalization phase transitions in Bayesian diffusion models
Bayesian diffusion models memorize training data when mutual information between restricted observations and training data exceeds log dataset size, and generalize otherwise.
-
Generating quantum ensembles via reverse-time quantum diffusions
The paper establishes a reverse-time quantum diffusion framework that generates complex quantum ensembles from simple distributions by deriving and learning a feedback Hamiltonian from forward trajectory data.
-
Flow Reasoning Models: Scaling Reasoning Through Iterative Self-Refinement
Flow models reach 99.2% Sudoku accuracy in 7 passes and 96.1% on out-of-distribution Sudoku-Extreme by selecting dynamically stable candidates and training with self-conditioning plus DPO to avoid failed outputs.
-
SOWing Information: Cultivating Contextual Coherence with MLLMs in Image Generation
SOW uses MLLMs and attention to selectively control unidirectional diffusion for pixel-level fidelity and contextual coherence in text-vision-to-image tasks.