REVIEW 8 cited by
Memorization and Regularization in Generative Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.
Forward citations
Cited by 8 Pith papers
-
Tessellations of Semi-Discrete Flow Matching
Semi-discrete Flow Matching produces terminal assignment regions that are topologically simple (open, simply connected, homeomorphic to the ball under assumption) yet geometrically distinct from optimal transport Lagu...
-
On The Hidden Biases of Flow Matching Samplers
Empirical flow matching introduces coupled biases from plug-in estimation, including altered statistical targets, non-gradient minimizers, and non-unique dynamics via flux-null fields, with base distribution controlli...
-
When and how can inexact generative models still sample from the data manifold?
Inexact generative models stay on the data manifold because infinitesimal learning errors perturb the density only along the manifold, when top Lyapunov vectors align with the support boundary.
-
PAC-DP: PAC-Bayesian Diffusion Policy Learning
Adding a PAC-Bayes KL regularizer to Bayesian diffusion policies improves denoising, likelihood bounds, and robotic success rates most in low-data and complex tasks.
-
Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
Introduces the Invariant Contamination Ratio (ICR), a Fisher-based metric, to evaluate how diffusion models balance invariant representations with residual variation and to detect the onset of memorization during training.
-
On the Memorization of Consistency Distillation for Diffusion Models
Consistency distillation reduces memorization in diffusion models by suppressing unstable feature directions associated with memorization while preserving stable generalizable modes.
-
Conditional flow matching for physics-constrained inverse problems with finite training data
Conditional flow matching learns a velocity field to sample from measurement-conditioned posteriors in physics inverse problems, with early stopping to prevent variance collapse and selective memorization under finite...
-
A Kinetic Energy Perspective of Flow Matching
A per-sample kinetic-energy score along flow-matching trajectories tracks semantic quality and data rarity, with an extreme-energy regime that predicts memorization, and a two-phase inference-time shaping that improve...
Discussion (0). Sign in to comment.