REVIEW 22 cited by
On Memorization in Diffusion Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Due to their capacity to generate novel and high-quality samples, diffusion models have attracted significant research interest in recent years. Notably, the typical training objective of diffusion models, i.e., denoising score matching, has a closed-form optimal solution that can only generate training data replicating samples. This indicates that a memorization behavior is theoretically expected, which contradicts the common generalization ability of state-of-the-art diffusion models, and thus calls for a deeper understanding. Looking into this, we first observe that memorization behaviors tend to occur on smaller-sized datasets, which motivates our definition of effective model memorization (EMM), a metric measuring the maximum size of training data at which a learned diffusion model approximates its theoretical optimum. Then, we quantify the impact of the influential factors on these memorization behaviors in terms of EMM, focusing primarily on data distribution, model configuration, and training procedure. Besides comprehensive empirical results identifying the influential factors, we surprisingly find that conditioning training data on uninformative random labels can significantly trigger the memorization in diffusion models. Our study holds practical significance for diffusion model users and offers clues to theoretical research in deep generative models. Code is available at https://github.com/sail-sg/DiffMemorize.
Forward citations
Cited by 22 Pith papers
-
An exact information theory of generalization phase transitions in Bayesian diffusion models
Bayesian diffusion models memorize training data when mutual information between restricted observations and training data exceeds log dataset size, and generalize otherwise.
-
An analytic theory of creativity in convolutional diffusion models
Convolutional diffusion models generate novel images by assembling locally consistent patch mosaics of training patches, and this mechanism is captured by an analytic score machine that predicts individual model outputs.
-
Secrets Everywhere: Auditing Memorization in Mobility Prediction Models
Mobility prediction models systematically assign higher likelihood to training trajectories than to behaviorally similar unseen ones, with effects varying by user regularity and model architecture.
-
Filtering Memorization from Parameter-Space in Diffusion Models
Base-Anchored Filtering suppresses weakly backbone-aligned LoRA spectral channels to cut memorization while preserving or improving generation quality, without data or re-training.
-
Generalization and Memorization in Rectified Flow
Rectified Flow models peak in membership-inference vulnerability at the flow midpoint under uniform training; U-shaped timestep sampling suppresses memorization without harming FID.
-
Finding DoRI: Discovery of Retained Images in Diffusion Models
Adversarially optimized text embeddings re-trigger supposedly removed memorized images in pruned diffusion models, showing memorization is distributed rather than local.
-
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
In one-dimensional denoising score matching with two-layer ReLU networks, a large SGD learning rate provably prevents the learned score from getting close to the empirical optimal score, mitigating memorization.
-
Memorization and Regularization in Generative Diffusion Models
The exact minimizer of the empirical score-matching loss makes reverse diffusion trajectories converge to training samples, and certain regularizers prevent that collapse.
-
NAMESAKES: Probing Identity Memorization in Text-to-Image Models
A two-signal probe — generation consistency and centroid distinctiveness — separates memorized from fabricated identities in black-box text-to-image models, tested on the new NAMESAKES benchmark.
-
Fitting Image Diffusion Models on Video Datasets
A shared-noise temporal consistency regularizer for image diffusion training accelerates convergence and lowers FID on the HandCo video dataset.
-
On the Collapse Errors Induced by the Deterministic Sampler for Diffusion Models
Deterministic diffusion samplers induce collapse errors, where samples overly concentrate locally, caused by low-noise score learning degrading high-noise score accuracy.
-
Diffusion models under low-noise regime
Diffusion models trained on disjoint data converge at high noise but diverge near the data manifold, and they fail to denoise very small perturbations accurately.
-
A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective
A small subset of training samples drives most memorization in tabular diffusion models, and pruning them based on early memorization signals reduces measured leakage, though the evaluation metric makes part of the ga...
-
FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings
Probabilistically adding high-intensity noise to individual token embeddings during fine-tuning reduces replication in Stable Diffusion by up to 28.78% in the paper's experiments, with unchanged or improved FID.
-
Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives
Diffusion policies trained on small robot demonstration sets act as action lookup tables, and a simple nearest-neighbor policy with a contrastive encoder matches their performance at a fraction of the cost.
-
Understanding and Mitigating Memorization in Generative Models via Sharpness of Probability Landscapes
Memorized outputs in diffusion models sit in sharp probability peaks, which can be detected at the first sampling step and avoided by optimizing initial noise.
-
LoyalDiffusion: A Diffusion Model Guarding Against Data Replication
Selectively replacing U-Net skip connections 3 and 4 with a 3x3 convolution, applied only for large diffusion timesteps, reduces measured training-data replication by about half with little FID loss.
-
Towards a Mechanistic Explanation of Diffusion Model Generalization
Diffusion model denoisers appear to generalize through localized patch-based denoising, a mechanism a training-free patch composite (PSPC) can reproduce across architectures and datasets.
-
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.
-
Generation Properties of Stochastic Interpolation under Finite Training Set
For finite training sets, stochastic interpolation models reproduce training samples under deterministic generation and output noise-perturbed copies under stochastic generation.
-
Protecting patient privacy in clinical foundation models: Technical and legal perspectives
This review proposes a two-dimensional privacy risk framework for clinical foundation models, then analyzes whether current US and EU law can handle the leakage risks.
-
Grounding Intelligence in Movement
Movement should be treated as a first-class AI modeling modality, and a unified, biomechanically grounded movement foundation model built from aggregated data across species and sensors is the proposed path forward.
Discussion (0). Continue with ORCID to comment.