Pith. sign in

REVIEW 2 major objections 8 minor 58 references

Exact threshold found for memorization vs. generalization in diffusion models

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 01:11 UTC pith:KT7C6ZM4

load-bearing objection Clean info-theoretic phase boundary for memorization vs. generalization in diffusion models; theory is solid but tail conditions unverified and neural-net agreement is early-training only the 2 major comments →

arxiv 2607.08041 v1 pith:KT7C6ZM4 submitted 2026-07-09 cs.LG cond-mat.dis-nn

An exact information theory of generalization phase transitions in Bayesian diffusion models

Henry Hunt , Mason Kamb , Surya Ganguli This is my paper
classification cs.LG cond-mat.dis-nn
keywords informationmodelsdiffusiontrainingbirddatarestrictedbayesian
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces BIRD (Bayesian Information Restricted Diffusion) models, a class of analytically tractable diffusion models in which each pixel of an image observes only restricted information about the noisy image and then performs Bayesian inference to guess which training sample produced its observation. The central result is an exact information-theoretic criterion for when such models memorize versus generalize: the model memorizes if and only if the mutual information between the pixel's restricted observation and the true data distribution exceeds the logarithm of the number of training samples, and generalizes otherwise. This yields a phase boundary in the joint space of dataset size, time in the reverse generative process, and degree of information restriction. The authors prove this using a Random Energy Model saddle-point analysis in the large-data limit, under mild tail assumptions on the posterior. They show that spatially local BIRD models (where restriction takes the form of limiting each pixel to a local patch) closely predict the individual outputs of trained UNets and diffusion transformers early in training, achieving high agreement across standard image datasets. For natural images with power-law spectral statistics, the theory predicts that the critical patch scale at the phase boundary scales linearly with noise level, matching the spectral scale needed for effective denoising, and that the dataset size required to avoid memorization scales only exponentially in the image linear dimension raised to a small exponent, not in the full pixel dimension, explaining how the curse of dimensionality is circumvented.

Core claim

The memorization-generalization phase transition in BIRD models is exactly governed by the equation ln|D| = I(φ; C_{x,t}), where |D| is the number of training samples and I(φ; C_{x,t}) is the mutual information between the true data distribution and a pixel's restricted observation under the forward diffusion process. When mutual information exceeds ln|D|, the Bayesian posterior concentrates on a single training sample and the model memorizes; below this threshold, the posterior retains entropy and the model generalizes. This is proven via a Random Energy Model analysis that becomes exact in the large-dataset limit.

What carries the argument

The BIRD model framework, in which each pixel x makes a restricted observation C_{x,t} of the noisy image and computes a Bayesian posterior P_train(φ|C_{x,t}) over training samples; the Random Energy Model mapping from posterior energies to a Boltzmann distribution, enabling a saddle-point analysis of the posterior entropy; and the mutual information I(φ; C_{x,t}) as the order parameter for the phase transition.

Load-bearing premise

The exact phase transition formula requires that the Bayesian posterior over training samples, viewed as an energy distribution, has sub-exponential tails with a well-behaved rate function. If these tail conditions fail for certain data distributions or observation channels, the sharp transition may become smooth or shift, and the exact equality between ln|D| and mutual information would no longer hold precisely.

What would settle it

Construct a data distribution and observation channel where the posterior energy distribution violates the sub-exponential tail assumption (e.g., heavy-tailed energies with many local extrema), and show that the posterior entropy does not exhibit a sharp transition at ln|D| = I(φ; C_{x,t}) but instead changes gradually, invalidating the exact phase boundary prediction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The phase boundary predicts that successful generation proceeds along the edge of memorization: as noise decreases in the reverse process, the model must progressively restrict information (e.g., shrink patch size) to stay in the generalizing phase, which is confirmed experimentally in both BIRD models and early-training neural diffusion models.
  • For scale-invariant natural images (power spectral density ~ k^{-2}), the dataset size required to avoid memorization does not grow with image dimension at all, while for near-scale-invariant images (exponent 2-ε with ε ~ 0.1-0.3), it grows only as exp(L_I^ε) where L_I is the image linear size, far below the exponential in full dimensionality that naive theory would predict.
  • The Gaussian upper bound on mutual information, computable from second-order data statistics alone, closely predicts the critical patch scale on real datasets, providing a practical tool for estimating memorization risk without computing exact mutual information.
  • The consistent generalization phenomenon (independent models trained on disjoint data subsets producing nearly identical outputs) is shown to occur precisely when the BIRD model is in the generalizing phase, linking robustness to training data realization directly to the information-theoretic criterion.
  • The theory provides a principled basis for understanding why local inductive biases in diffusion model architectures promote generalization: spatial locality restricts information, lowering mutual information below the memorization threshold.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. This paper introduces Bayesian Information Restricted Diffusion (BIRD) models, a class of analytically tractable diffusion models in which each pixel observes only restricted information about a noisy image and performs Bayesian inference over the training set to reverse the diffusion process. The central theoretical result is an exact phase boundary between memorization and generalization: under a Random Energy Model (REM) analysis in the large-dataset limit, the BIRD model memorizes when the mutual information I(φ; C_{x,t}) between the true data distribution and the pixel's restricted observation exceeds ln|D|, and generalizes otherwise (Eq. 5, Thm D.1). The authors show that spatially local BIRD models predict the outputs of trained UNets and DiTs early in training (r² ~ 0.85–0.93), and that the critical patch scale L_c(σ_t) tracks the spectral scale L_spec of natural images, implying that data requirements scale as exp(L_I^ε) rather than exponentially in full ambient dimension, thereby circumventing the curse of dimensionality for near-scale-invariant images.

Significance. The paper makes a substantive contribution by providing a first-principles, parameter-free criterion for the memorization–generalization transition in a general class of Bayesian diffusion models. The derivation via the REM saddle-point (Thm D.1, App. D.5) is internally coherent and yields a clean, falsifiable prediction: ln|D| = I(φ; C_{x,t}). The experimental validation across four datasets and two architectures (Tables 1–2, Fig. 3) is encouraging, and the scaling analysis for power-law images (Sec. 5.2, Thm F.3) provides a concrete mechanism for evading the curse of dimensionality. The connection between the critical scale and the spectral scale (Fig. 4b) is a notable empirical finding. The framework generalizes prior work on local score models [13, 14] and extends the collapse condition of [6] to arbitrary channels and data distributions.

major comments (2)
  1. App. D.5, Thm D.1: The proof of the exact phase boundary ln|D| = I(φ; C_{x,t}) relies on the Random Energy Model saddle-point approximation, which requires that the energy distribution P(E|C_{x,t}) admits a rate function with a continuous first derivative and finitely many extrema at O(1) energy (Eq. 72), plus sub-exponential tails. These conditions are never verified for any concrete data distribution or channel—not for Gaussian data, not for CIFAR10, not for CelebA. The paper should either (a) verify these conditions for at least the Gaussian case (where the energy is a quadratic form and the rate function can be computed explicitly), or (b) explicitly acknowledge that the 'exact' claim is conditional on unverified tail assumptions and discuss what happens when they fail (e.g., whether the transition becomes smooth). Without this, the gap between 'exact under unverified conditions' and
  2. Sec. 4, Eq. (4)–(5) vs. App. D.5, Thm D.1: The main text states the phase boundary is 'exactly given by' ln|D| = I(φ; C_{x,t}), but Thm D.1 actually proves the pointwise condition S[P_train] = max(0, ln|D| − D_KL(P_test(φ|C_{x,t}) || P_0(φ))) (Eq. 65), which involves the KL divergence for a specific observation, not the mutual information. The mutual information version (Eq. 5) is obtained only after averaging over C_{x,t} drawn from the test distribution (Thm D.2, Eq. 103). The paper should clarify in the main text that Eq. (5) is an averaged condition, while the pointwise condition involves the KL divergence, and that these coincide only under the concentration assumption √Var(D_KL) = o(1) in Thm D.2. This distinction matters for interpreting the experimental validation in Fig. 3, where the entropy deficit is plotted for specific observations.
minor comments (8)
  1. Title: The phrase 'exact information theory' is somewhat misleading given the O(ln ln|D|/ln|D|) corrections in Thm D.1 and the unverified tail conditions. Consider softening to 'An information-theoretic theory...' or similar.
  2. Fig. 1b: The schematic shows three arrows (noise, observation capacity, dataset size) but the phase boundary is drawn as a simple curve. A 2D projection (L vs. σ_t) with |D| as a parameter would be clearer than the 3D sketch.
  3. Sec. 3, Tables 1–2: The r² values peak at 10–30 epochs and then decline. The paper attributes this to BIRD models describing 'early training,' but does not discuss what causes the divergence at later epochs (e.g., emergence of nonlocal features, attention effects). A brief discussion would strengthen the narrative.
  4. App. D.5, between Eqs. (60) and (61): The transition from the CLT-based approximation to the max(·, 0) formula is stated informally ('we can guess'). The REM proof that follows justifies this, but the narrative flow could be improved by stating upfront that the REM analysis will make this rigorous.
  5. Sec. 5.2, Eq. (8): The scaling ln|D| ~ L_I^ε is derived under the assumption that L_c ~ L_spec at all noise levels. The paper notes this holds for ε ~ 0.1–0.3, but does not discuss whether the proportionality constant between L_c and L_spec depends on ε or on the dataset. Fig. 4b suggests close agreement, but a more quantitative comparison (e.g., ratio L_c/L_spec vs. σ_t) would help assess robustness at the most demanding t.
  6. App. F.6: The subcritical threshold uses d* = 10 as an 'effective sample size.' The sensitivity of the results to this choice is not discussed. A brief robustness check (e.g., d* = 5, 20) would strengthen the empirical claims.
  7. App. H.1: The calibration procedure selects patch scales by best validation loss, which naturally selects scales near L_c. This could introduce circularity in the claim that trained models 'track the phase boundary.' The authors should note that the calibration is independent of the neural network outputs, and ideally provide an independent check (e.g., using the spectral scale directly without calibration).
  8. References: Several 2026 arXiv preprints are cited (e.g., [20], [21], [24], [26], [36], [43]). These should be verified for availability and updated with DOIs/journal references upon publication.

Circularity Check

0 steps flagged

No significant circularity; the phase boundary is derived from the REM without fitting to the target result, and the mutual information is computed analytically or bounded independently.

full rationale

The central claim—the phase boundary ln|D| = I(φ; C_{x,t})—is derived from first principles via a Random Energy Model saddle-point argument (Thm D.1, App. D.5). The proof maps the posterior entropy to a thermodynamic free energy computation and obtains the max(0, ln|D| - D_KL) formula without fitting any parameter to the memorization outcome. The mutual information I(φ; C_{x,t}) is computed analytically for Gaussian data (Thm E.1) and upper-bounded via second-order statistics (Gaussian bound, Thm E.2) for real datasets, not calibrated to memorization behavior. The critical scale prediction L_c ~ σ_t √(ln|D|) is derived from the entropy power inequality (Thm F.3), not fitted. The self-citations to [13, 14] (Kamb & Ganguli) are used to define the LS/ELS model class that BIRD generalizes, but the phase transition theory itself is self-contained: it does not depend on any unverified result from those prior works. The experimental validation (Fig. 3) compares the theoretically predicted boundary to numerically computed posterior entropies, which is an independent check, not a fit renamed as prediction. The patch scale calibration in experiments (App. H.1) uses validation loss, not the memorization outcome, so it does not force the phase boundary prediction. The only mild concern is that the REM tail conditions (sub-exponential tails, finite extrema of the rate function) are not verified for real image data, but this is a correctness/assumption concern, not circularity—the derivation does not assume its conclusion. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The theory has no free parameters fitted to the memorization transition itself. The core result ln|D| = I(φ; C_{x,t}) is parameter-free given the data distribution and observation channel. Experimental calibration choices (patch scale L, d*) are used only for neural network comparison, not for testing the phase boundary. The power-law exponent ε is a measured property of natural images, not a fitted constant.

free parameters (3)
  • Patch scale L
    Selected by cross-validation on a held-out set to minimize denoising loss, not a free parameter in the theory but a calibration choice for neural network comparison experiments.
  • Power law exponent ε = 0.1-0.3
    Empirically measured from natural image statistics, not fitted to the memorization transition. Enters the scaling law ln|D| ~ L^ε as a data property.
  • d* (subcritical threshold parameter) = 10
    Used only in the empirical analysis of critical vs spectral scale (App. F.6), not in the core theory. Chosen as effective sample size for a subcritical threshold.
axioms (5)
  • domain assumption Training data points φ ∈ D are drawn i.i.d. from the true data distribution P_0(φ).
    Standard in learning theory. Invoked in Thm D.1 (App. D.5) to apply the REM framework where energies of different training points are independent.
  • domain assumption The posterior P_train(φ|C_{x,t}) has sub-exponential tails with a rate function having a finite number of maxima/minima at O(1) energy.
    Stated in Thm D.1 (App. D.5). Required for the saddle-point approximation to be exact in the large |D| limit. Not verified for specific datasets but argued to hold under mild conditions.
  • domain assumption The denoising loss decouples across pixels, allowing per-pixel analysis.
    Invoked in App. C (Eq. 22) to justify analyzing each pixel's denoiser independently. This is exact for the squared loss but the BIRD framework's restriction to per-pixel observations is a modeling choice.
  • domain assumption Natural images have translationally-invariant statistics with power-law PSD P(k) ~ |k|^{-2-ε}.
    Used in Sec. 5.2 and Thm F.3 (App. F.3) to derive the scaling of critical and spectral scales. Well-established empirically [39] but an idealization of real datasets.
  • domain assumption The variance of the KL divergence D_KL(P_test(φ|C_{x,t}) || P_0(φ)) is o(1) for test-set averaging.
    Stated in Thm D.2 (App. D.5). Required to replace the annealed entropy with its mean when averaging over C_{x,t}.
invented entities (1)
  • BIRD (Bayesian Information Restricted Diffusion) models independent evidence
    purpose: A class of diffusion models where each pixel observes restricted information C_{x,t} about the noisy image and uses Bayesian posterior inference over training data to denoise.
    The BIRD framework is a theoretical construct, but it is falsifiable: the paper tests whether spatially local BIRD models predict neural network outputs (Figs. 2, 6, Tables 1-2) and whether the phase boundary matches empirical posterior entropies (Fig. 3). The framework generalizes existing LS/ES/ELS models [13] which are special cases.

pith-pipeline@v1.1.0-glm · 51236 in / 3410 out tokens · 347252 ms · 2026-07-10T01:11:50.269759+00:00 · methodology

0 comments
read the original abstract

How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. This model class generalizes existing analytical diffusion models that use spatially local information restriction. We show that spatially local BIRD models closely approximate trained diffusion models \textit{early in training}, across different architectures such as UNets and DiTs. Under minimal assumptions on the data distribution, we identify an information-theoretic phase boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality.

Figures

Figures reproduced from arXiv: 2607.08041 by Henry Hunt, Mason Kamb, Surya Ganguli.

Figure 1
Figure 1. Figure 1: A schematic figure showing different transitions [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The consistent and robust generalization of diffusion models early in training, across [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of theory and experiment for the memorization-generalization phase tran [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of varying the patch scale L (for square patches) in a spatially local BIRD model. (a) The top (bottom) shows the outcome of denoising of a single test (train) image using BIRD models of different patch sizes L starting from noise σt = 1. Optimal denoising occurs when L = 7 (in general this optimal denoising scale coincides with Lc). For L = 5 < Lc, from theory we know the denoiser is robust, but de… view at source ↗
Figure 5
Figure 5. Figure 5: A plot of the mutual information I(ϕΩ; φ) on a Gaussianized version of the BWCeleba32 dataset, as a function of both the noise level σ 2 and the patch length L, relative to the prior for a) the theoretical distribution b) a finite sample of 104 examples, as a function of noise level and patch scale. Plotted as well is the critical threshold I = ln |D|. To test the concordance between the critical scale and… view at source ↗
Figure 6
Figure 6. Figure 6: Plots of the Median r 2 metric (left) and the MSE metric (right) averaged over comparisons between several independently trained DiTs (top two panels) and UNets (bottom two panels) and their corresponding analytical models, across four datasets Celeba64, FashionMNIST, CIFAR10, and MNIST. Each model is trained on a reduced dataset of 104 samples. The Median r 2 metric starts high across all datasets and gra… view at source ↗
Figure 7
Figure 7. Figure 7: Further samples comparing DiTs and UNets early in training, trained on two disjoint [PITH_FULL_IMAGE:figures/full_fig_p048_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Further samples comparing DiTs and UNets early in training, trained on two disjoint [PITH_FULL_IMAGE:figures/full_fig_p048_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Further samples comparing DiTs and UNets early in training, trained on two disjoint [PITH_FULL_IMAGE:figures/full_fig_p049_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Further samples comparing DiTs and UNets early in training, trained on two disjoint [PITH_FULL_IMAGE:figures/full_fig_p049_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: As training progresses, models evolve from ‘patch mosaic’ style outputs [13] towards se [PITH_FULL_IMAGE:figures/full_fig_p050_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 21 internal anchors

  1. [1]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. PMLR, 2015

    work page 2015

  2. [2]

    Denoising diffusion probabilistic models.Ad- vances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Ad- vances in neural information processing systems, 33:6840–6851, 2020

    work page 2020

  3. [3]

    Denoising Diffusion Implicit Models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020. 9

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  4. [4]

    Generalization in diffusion models arises from geometry-adaptive harmonic representations

    Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and St ´ephane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation.arXiv preprint arXiv:2310.02557, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Cambridge University Press, 2014

    Shai Shalev-Shwartz and Shai Ben-David.Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014

    work page 2014

  6. [6]

    Dynamical Regimes of Diffusion Models

    Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Marc M ´ezard. Dynamical regimes of diffusion models.arXiv preprint arXiv:2402.18491, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Convergence of denoising diffusion models under the manifold hypothesis

    Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothe- sis.arXiv preprint arXiv:2208.05314, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    Mem- orization and generalization in generative diffusion under the manifold hypothesis.Journal of Statistical Mechanics: Theory and Experiment, 2025(7):073401, 2025

    Beatrice Achilli, Luca Ambrogioni, Carlo Lucibello, Marc M´ezard, and Enrico Ventura. Mem- orization and generalization in generative diffusion under the manifold hypothesis.Journal of Statistical Mechanics: Theory and Experiment, 2025(7):073401, 2025

    work page 2025

  9. [9]

    Adapting to unknown low-dimensional structures in score-based diffusion models.Advances in Neural Information Processing Systems, 37:126297–126331, 2024

    Gen Li and Yuling Yan. Adapting to unknown low-dimensional structures in score-based diffusion models.Advances in Neural Information Processing Systems, 37:126297–126331, 2024

    work page 2024

  10. [10]

    Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data

    Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data. InInternational Conference on Machine Learning, pages 4672–4712. PMLR, 2023

    work page 2023

  11. [11]

    A Good Score Does not Lead to A Good Generative Model

    Sixu Li, Shi Chen, and Qin Li. A good score does not lead to a good generative model.arXiv preprint arXiv:2401.04856, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Closed-Form Diffusion Models

    Christopher Scarvelis, Haitz S ´aez de Oc´ariz Borde, and Justin Solomon. Closed-form diffusion models.arXiv preprint arXiv:2310.12395, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  13. [13]

    An analytic theory of creativity in convolutional diffusion models

    Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  14. [14]

    Towards a mechanistic explanation of diffusion model generalization

    Matthew Niedoba, Berend Zwartsenberg, Kevin Patrick Murphy, and Frank Wood. Towards a mechanistic explanation of diffusion model generalization. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  15. [15]

    On the Generalization of Diffusion Model

    Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model.arXiv preprint arXiv:2305.14712, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    On Memorization in Diffusion Models

    Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memoriza- tion in diffusion models.arXiv preprint arXiv:2310.02664, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Memorization to generalization: Emergence of diffusion models from associative memory.arXiv preprint arXiv:2505.21777, 2025

    Bao Pham, Gabriel Raya, Matteo Negri, Mohammed J Zaki, Luca Ambrogioni, and Dmitry Krotov. Memorization to generalization: Emergence of diffusion models from associative memory.arXiv preprint arXiv:2505.21777, 2025

    work page arXiv 2025

  18. [18]

    On the interpolation effect of score smoothing in diffusion models

    Zhengdao Chen. On the interpolation effect of score smoothing in diffusion models. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps:// openreview.net/forum?id=O33LAUliUF

    work page 2026

  19. [19]

    The Unreasonable Effectiveness of Gaussian Score Approximation for Diffusion Models and its Applications

    Binxu Wang and John J Vastola. The unreasonable effectiveness of gaussian score approxima- tion for diffusion models and its applications.arXiv preprint arXiv:2412.09726, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    A Random Matrix Theory Perspective on the Consistency of Diffusion Models

    Binxu Wang, Jacob Zavatone-Veth, and Cengiz Pehlevan. A random matrix theory perspective on the consistency of diffusion models.arXiv preprint arXiv:2602.02908, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold

    Ye He, Yitong Qiu, and Molei Tao. Diffusion model’s generalization can be characterized by inductive biases toward a data-dependent ridge manifold.arXiv preprint arXiv:2602.06021, 2026. 10

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    Diffusion models learn low-dimensional distributions via subspace clustering

    Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. In2025 IEEE 10th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pages 211–215. IEEE, 2025

    work page 2025

  23. [23]

    Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models

    Alessandro Favero, Antonio Sclocchi, and Matthieu Wyart. Bigger isn’t always memorizing: Early stopping overparameterized diffusion models.arXiv preprint arXiv:2505.16959, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Why diffusion models don’t memorize: The role of implicit dynamical regularization in training

    Tony Bonnaire, Rapha ¨el Urfin, Giulio Biroli, and Marc Mezard. Why diffusion models don’t memorize: The role of implicit dynamical regularization in training. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems, 2026. URLhttps: //openreview.net/forum?id=BSZqpqgqM0

    work page 2026

  25. [25]

    How compositional generalization and creativity improve as diffusion models are trained

    Alessandro Favero, Antonio Sclocchi, Francesco Cagnetta, Pascal Frossard, and Matthieu Wyart. How compositional generalization and creativity improve as diffusion models are trained. InInternational Conference on Machine Learning, pages 16286–16306. PMLR, 2025

    work page 2025

  26. [26]

    A theory of learning data statistics in diffusion models, from easy to hard

    Lorenzo Bardone, Claudia Merger, and Sebastian Goldt. A theory of learning data statistics in diffusion models, from easy to hard.arXiv preprint arXiv:2603.12901, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Shallow diffusion networks provably learn hidden low-dimensional structure

    Nicholas M Boffi, Arthur Jacot, Stephen Tu, and Ingvar Ziemann. Shallow diffusion networks provably learn hidden low-dimensional structure.arXiv preprint arXiv:2410.11275, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    A solvable model of learning generative diffusion: theory and insights.arXiv preprint arXiv:2501.03937, 2025

    Hugo Cui, Cengiz Pehlevan, and Yue M Lu. A solvable model of learning generative diffusion: theory and insights.arXiv preprint arXiv:2501.03937, 2025

    work page arXiv 2025

  29. [29]

    Diffusion probabilistic mod- els generalize when they fail to memorize

    TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic mod- els generalize when they fail to memorize. InICML 2023 workshop on structured probabilistic inference{\&}generative modeling, 2023

    work page 2023

  30. [30]

    Analysis of learning a flow-based generative model from limited sample complexity

    Hugo Cui, Florent Krzakala, Eric Vanden-Eijnden, and Lenka Zdeborov´a. Analysis of learning a flow-based generative model from limited sample complexity. InInternational Conference on Learning Representations, volume 2024, pages 51929–51955, 2024

    work page 2024

  31. [31]

    On the edge of memorization in diffusion models.Advances in Neural Information Processing Systems, 38:96113–96157, 2026

    Sam Buchanan, Druv Pai, Yi Ma, and Valentin De Bortoli. On the edge of memorization in diffusion models.Advances in Neural Information Processing Systems, 38:96113–96157, 2026

    work page 2026

  32. [32]

    Locality in image diffusion models emerges from data statistics.arXiv preprint arXiv:2509.09672, 2025

    Artem Lukoianov, Chenyang Yuan, Justin Solomon, and Vincent Sitzmann. Locality in image diffusion models emerges from data statistics.arXiv preprint arXiv:2509.09672, 2025

    work page arXiv 2025

  33. [33]

    Local mechanisms of compositional generalization in conditional diffusion

    Arwen Bradley. Local mechanisms of compositional generalization in conditional diffusion. arXiv preprint arXiv:2509.16447, 2025

    work page arXiv 2025

  34. [34]

    Origins of creativity in attention-based diffusion models.arXiv preprint arXiv:2506.17324, 2025

    Emma Finn, T Anderson Keller, Manos Theodosis, and Demba E Ba. Origins of creativity in attention-based diffusion models.arXiv preprint arXiv:2506.17324, 2025

    work page arXiv 2025

  35. [35]

    Local Diffusion Models and Phases of Data Distributions

    Fangjun Hu, Guangkuo Liu, Yifan Zhang, and Xun Gao. Local diffusion models and phases of data distributions.arXiv preprint arXiv:2508.06614, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Models

    Luca Ambrogioni. How out-of-equilibrium phase transitions can seed pattern formation in trained diffusion models.arXiv preprint arXiv:2603.20092, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [37]

    Local- ized diffusion models.arXiv preprint arXiv:2505.04417, 2025

    Georg A Gottwald, Shuigen Liu, Youssef Marzouk, Sebastian Reich, and Xin T Tong. Local- ized diffusion models.arXiv preprint arXiv:2505.04417, 2025

    work page arXiv 2025

  38. [38]

    An analytic theory of convolutional neural network inverse problems solvers

    Minh Hai Nguyen, Quoc Bao Do, Edouard Pauwels, and Pierre Weiss. An analytic theory of convolutional neural network inverse problems solvers.arXiv preprint arXiv:2601.10334, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  39. [39]

    Statistics of natural images: Scaling in the woods

    Daniel Ruderman and William Bialek. Statistics of natural images: Scaling in the woods. Advances in neural information processing systems, 6, 1993

    work page 1993

  40. [40]

    Diffusion is spectral autoregression, 2024

    Sander Dieleman. Diffusion is spectral autoregression, 2024. URLhttps://sander.ai/ 2024/09/02/spectral-autoregression.html. 11

    work page 2024

  41. [41]

    The MIT press, 1949

    Norbert Wiener.Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications. The MIT press, 1949

    work page 1949

  42. [42]

    Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35: 26565–26577, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35: 26565–26577, 2022

    work page 2022

  43. [43]

    Theory of speciation transitions in diffusion models with general class structure, 2026

    Beatrice Achilli, Marco Benedetti, Giulio Biroli, and Marc M ´ezard. Theory of speciation transitions in diffusion models with general class structure, 2026. URLhttps://arxiv. org/abs/2602.04404

    work page arXiv 2026

  44. [44]

    B. Derrida. Random-energy model: Limit of a family of disordered models.Phys. Rev. Lett., 45:79–82, Jul 1980. doi: 10.1103/PhysRevLett.45.79. URLhttps://link.aps.org/doi/ 10.1103/PhysRevLett.45.79

    work page doi:10.1103/physrevlett.45.79 1980

  45. [45]

    Ox- ford University PressOxford, January 2009

    Marc M ´ezard and Andrea Montanari.Information, Physics, and Computation. Ox- ford University PressOxford, January 2009. ISBN 9780191718755. doi: 10.1093/ acprof:oso/9780198570837.001.0001. URLhttp://dx.doi.org/10.1093/acprof:oso/ 9780198570837.001.0001

    work page doi:10.1093/acprof:oso/ 2009

  46. [46]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas.Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954

    work page 2006

  47. [47]

    Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

    Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni. Manifolds, random matrices and spectral gaps: The geometric phases of generative diffusion. arXiv preprint arXiv:2410.05898, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  48. [48]

    Towards a Mechanistic Explanation of Diffusion Model Generalization

    Matthew Niedoba, Berend Zwartsenberg, Kevin Murphy, and Frank Wood. Towards a mecha- nistic explanation of diffusion model generalization.arXiv preprint arXiv:2411.19339, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  49. [49]

    Unmasking bias in diffusion model training

    Hu Yu, Li Shen, Jie Huang, Hongsheng Li, and Feng Zhao. Unmasking bias in diffusion model training. InEuropean Conference on Computer Vision, pages 374–390. Springer, 2024

    work page 2024

  50. [50]

    Easing Color Shifts in Score-Based Diffusion Models

    Katherine Deck and Tobias Bischoff. Easing color shifts in score-based diffusion models. arXiv preprint arXiv:2306.15832, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  51. [51]

    Perception prioritized training of diffusion models

    Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11472–11481, 2022

    work page 2022

  52. [52]

    Progressive Distillation for Fast Sampling of Diffusion Models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 12 Appendices A Notation and background on diffusion models. A.1 Notation conventions We use the following notation: •φ∈R d denotes an example from the training set. For images or patches of sizeLpixels byLpixels byCcolor ch...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  53. [53]

    Replace the average over the true data distributionP 0(φ)with an empirical average over a finite training set

    work page

  54. [54]

    Minimize the loss in (14) to obtain the learned neural networkM[t, θ](ϕ t)

    work page

  55. [55]

    Replace the posterior meanE[φ|ϕ t]in Tweedie’s formula in (13) with its approximation Mt,θ[ϕt], thereby obtaining an approximate score function

    work page

  56. [56]

    Insert this approximate score into the deterministic ODE representing the reverse process in (12). In practice, for purposes of variance reduction, rather than predicting the clean dataφfrom the noisy sampleϕ t, one can also predict the noiseηor the velocityv t through the following two losses respectively: Lη t = 1 2 Eφ∼P0,η∼N(0,I) [∥η−M t,θ[ϕt(φ, η)]∥2]...

    work page

  57. [57]

    under a particular correlated choice of targetT x and channelC x,t. [13] consider the setting of a model that isequivariantunder a groupG, which means that for anyg∈Gthe modelM[ϕ] satisfiesM[U gϕ] =U gM[ϕ], whereU g is a unitary representation of the groupG. Under the following choice of correlated channel and target: g∼Haar(G)(30) Cx,t =U † g ϕt (31) Tx ...

    work page

  58. [58]

    thermodynamic

    for a nice pedagogical treatment). The REM is solvable by a saddle point approximation which becomes exact in a “thermodynamic” limit, which in our case, corresponds to a large amount of training data, or large|D|. We define P(E ′|Cx,t) = Z dφ δ(E ′ −E(φ|C x,t))P0(φ)(66) This is simply the distribution of energiesE ′ we expect to see conditioned on a fixe...

    work page