{"id":"1c7d6577-1d61-4d81-9d7e-73ea81569195","arxiv_id":"2504.12532","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Diffusion model generalization is explained by the covariance of the noisy score-matching target, which adds state-dependent noise to sampling and fills gaps in the training distribution.","lead":"This paper develops a mathematical theory for why diffusion models can create new-looking samples instead of just repeating their training data. It argues that the noisy training objective used for diffusion models acts like a built-in smoothing mechanism that fills in gaps between training examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a second-order cumulant truncation that is unjustified for finite-P estimators; Prop. 4.1's naive estimator is explicitly non-Gaussian, so the V-kernel alone may not determine the learned distribution.","rationale":"The reader's weakest_assumption identifies the second-order cumulant truncation, and this is exactly the load-bearing concern in my reading. The paper's central object is the averaged path integral [q(x0|xT)], and the reduction to an effective SDE with only a mean and a V-kernel is exact only when the score estimator field is Gaussian. The authors explicitly assume this in Sec. 3 and Appendix D.3 but do not justify it for Prop. 4.1, where the estimator is a discrete mixture over the posterior and has no asymptotic parameter to make it Gaussian. The linear/NTK asymptotic results are better protected because the CLT applies as P -> infinity with F/P fixed, so the reader's concern is the right one, though it applies more to the general claims and Prop. 4.1 than to the main asymptotic theorems. I also note two related secondary issues: the effective 'SDE' with a nonlocal V is not an ordinary Markovian SDE, and the infinite-training-time NTK result in Eq. 17 has a discretization-dependent prefactor. These do not change my recommendation. The paper is honest about its limitations, gives a detailed derivation, and has reproducible code; those count in its favor. The correct response is to keep the reader's CONDITIONAL verdict and require either a bound on neglected cumulants or a demonstration that they do not affect the claimed gap-filling behavior in the non-Gaussian regimes.","tokens_in":35410,"tokens_out":9373,"duration_ms":115183,"concrete_test":"Implement Prop. 4.1 for a two-point mixture x0 = ±μ with κ = O(1): simulate the exact interleaved training-sampling dynamics for many noise seeds, and compare the resulting empirical [q] against the Gaussian effective SDE obtained from V = κ D_t C(x,t) D_t δ(t-t'). If the two distributions differ beyond Monte Carlo error, higher-order cumulants are material in a regime the paper claims is described by V alone. Repeat at smaller κ to check whether the discrepancy scales as κ^{3/2}, as expected if a third cumulant is the leading correction. A separate consistency check: for a linear model with small P (e.g., F=8, P=16), compare the true estimator distribution to the Gaussian V-kernel prediction and verify the deviation vanishes with 1/sqrt(P).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 and the headline 'generalization occurs iff V != 0' depend on truncating the cumulant expansion of the score estimator at second order, as the paper states in Sec. 3 ('Assuming higher-order terms can be neglected') and Appendix D.3. This is not a harmless technicality. For an arbitrary finite-P estimator, proxy-score fluctuations are non-Gaussian; the naive estimator of Prop. 4.1 is a discrete mixture over the M training examples at each time step, with no large-P or small-fluctuation limit that would suppress its third and higher cumulants. In that regime the path-integral average contains terms beyond -1/2 p^T V p, and the effective SDE (8) with only savg and V does not describe [q]. The linear and NTK results are on firmer ground: when P -> infinity with F/P held fixed, the CLT does suppress higher cumulants, so those asymptotic V-kernels are justified. But Sec. 3 and Prop. 4.1 are presented as general statements, and the paper does not bound the neglected cumulants or test any regime where they are large. The authors candidly flag the assumption, but the claim that covariance is the mechanism of generalization is incomplete without evidence that non-Gaussian corrections are negligible in the regimes where the central statements are made.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that diffusion models generalize because the denoising score matching (DSM) objective trains against a noisy \"proxy score\" whose expectation is the true score but whose state-dependent covariance—the \"V-kernel\"—perturbs the reverse-time dynamics. Using a Martin-Siggia-Rose path integral, the paper derives an effective SDE for the ensemble-averaged learned distribution (Prop. 3.1), then computes the V-kernel for three tractable cases: an interleaved naive estimator that uses proxy-score samples at each Euler step (Prop. 4.1), expressive linear feature models (Prop. 5.1), and lazy infinite-width neural networks in the NTK regime (Prop. 5.2). The paper argues that this covariance-driven noise fills gaps in the training distribution, explains capacity-dependent memorization/generalization transitions, and interacts with feature-related inductive biases. The theory is illustrated with 1D and 2D simulations of linear models.","tokens_in":35680,"tokens_out":14295,"duration_ms":159686,"significance":"If the proposed mechanism is correct, it offers an original and potentially influential explanation of a genuinely puzzling phenomenon: even though the DSM optimum equals the true score, finite-sample diffusion models learn smoothed distributions that interpolate the training data. The paper's main strengths are the self-contained path-integral formalism (Appendices D-G), explicit V-kernel formulas showing dependence on the capacity ratio F/P and on feature kernels, and reproducible toy experiments (code is provided). The qualitative predictions—capacity-driven memorization-to-generalization transitions, boundary smearing, and feature-dependent gap filling—are falsifiable. The main caveat is that the central object is an annealed average of the learned distribution, and the key approximations are not quantitatively controlled; the paper is more convincing as a mechanism proposal than as a theorem.","major_comments":[{"comment":"The paper defines [q(x0|xT)] as the arithmetic mean E_theta[q(x0|xT;theta)] and calls it the \"typical\" learned distribution. Since q is a nonlinear functional of the score estimator, the annealed average E_theta[q] differs from the quenched/typical value exp(E_theta log q); the V-kernel noise in Prop. 3.1 describes the smoothed mixture of models, not necessarily the distribution of any single trained model. This distinction is load-bearing because the statement \"generalization occurs iff V != 0\" refers to this averaged object. The paper should either justify that the annealed average is the appropriate observable (e.g., by showing self-averaging or by restricting to a regime where annealed and quenched averages agree to leading order) or explicitly reframe the central claim as being about the average learned distribution rather than the typical one.","section":"Section 3, Eq. (7), Prop. 3.1"},{"comment":"The effective SDE in Eq. (8) is derived by dropping all cumulants above second order in the characteristic function of the score estimator. The paper flags this assumption but does not bound the neglected terms. For the naive estimator of Prop. 4.1, the per-time-step estimator is a discrete mixture over the M training examples and is not Gaussian. A direct estimate under the 1/sqrt(Delta t) scaling in Eq. (55) suggests that the third and higher cumulants of the integrated action are O(Delta t^{3/2}) and vanish in the continuum limit, so the approximation is plausibly valid; however, the paper does not present this or any equivalent smallness condition, and for finite Delta t or large kappa the higher cumulants need not be negligible. Since the headline claim rests on this truncation, the manuscript should provide the missing calculation or state the validity conditions explicitly.","section":"Section 3, Eq. (52), Appendix D.3, Prop. 4.1"},{"comment":"The linear-model V-kernel is computed as the covariance over x0 draws conditional on fixed x,t samples, and the empirical moments mu-hat and Sigma-hat in the kernel Q are treated as deterministic. The general V-kernel in Sec. 3 averages over all training-sample realizations. Because Q in Eq. (74) depends on mu-hat and Sigma-hat, fluctuations of the x,t samples also contribute to the score-estimator covariance. The paper should show that those contributions are subleading relative to the F/P term (or equivalently redefine the ensemble in Prop. 5.1 so that it matches Sec. 3). Without this, Prop. 5.1 and Prop. 5.2 compute a slightly different object from the one used in the central argument.","section":"Appendix F.4-F.5, Eq. (78), Prop. 5.1"}],"minor_comments":[{"comment":"The symbol k is reused for the feature kernel in Prop. 5.1 and for the NTK feature kernel in Prop. 5.2, and Appendix G also uses K and k for the Gram matrix and an input-dependent vector. The footnote acknowledges the overload, but the notation is still hard to follow; consider renaming one of these kernels.","section":"Section 5, Eqs. (12)-(15), Appendix G"},{"comment":"The infinite-training-time V-kernel contains a prefactor kappa(Delta z) = F Delta z / P that depends on an unspecified discretization volume Delta z; the paper should define Delta z and state explicitly how kappa(Delta z) is held fixed in the F,P -> infinity limit.","section":"Eq. (17), Appendix G.3"},{"comment":"The caption states \"Average learned distribution (N = 100)\" without defining N in the main text; please state that N is the number of models in the ensemble average.","section":"Fig. 2 caption"},{"comment":"The early text says that optimizing J0 or J1 \"reproduces training examples\"; this is only true in the limit epsilon -> 0, since for finite epsilon the PF-ODE reproduces p(x0|epsilon). The paper later acknowledges this, but the early phrasing could mislead.","section":"Sec. 2 and Appendix A"},{"comment":"The semiclassical approximation is described as a \"small noise\" approximation, but the actual small parameter is kappa (proportional to F/P); please make this explicit where Eq. (18) is introduced.","section":"Sec. 6, Eq. (18), Appendix K"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an ambitious theory paper with honest limitations, and the appendices are careful. My main reservation is the gap between the annealed quantity actually computed and the 'typical learned distribution' language used throughout; this should be addressed before the paper can be accepted, as it affects the interpretation of the central claim. The novelty and fit with the journal are otherwise appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the V-kernel idea: the DSM objective's target is a noisy proxy score, and the covariance of the learned estimator acts as state-dependent noise in the effective PF-ODE, filling gaps in the training distribution. The path-integral averaging of the PF-ODE is a neat trick, and the explicit V-kernel computations for linear models and lazy infinite-width NTKs (Props. 5.1 and 5.2) are new, reproducible, and consistent with the CLT. The derivations in the appendices are careful and the authors are honest about many limitations. I also appreciate that they treat the noisy target as a feature rather than a bug, and that they connect the mechanism to memorization, outliers, and the F/P capacity ratio.\n\nThe soft spot is exactly where the stress-test note lands. The effective SDE in Prop. 3.1 and the \"generalization iff V != 0\" slogan depend on truncating the cumulant expansion at second order. For the naive estimator of Prop. 4.1, that truncation is not justified: the estimator is a discrete mixture over the M training examples, and nothing suppresses its third and higher cumulants in the construction. The paper flags the assumption, but it does not bound the neglected terms or show a regime where they are small. So the general claim is too strong. The linear and NTK results are on firmer ground because the P → ∞ limit with F/P held fixed does justify Gaussianity via the CLT, but those are asymptotic, lazy-regime results. The numerics also lack error bars, and the promised F/P scaling is not tested on a realistic architecture.\n\nFor a reader working on diffusion theory, this is worth engaging: the V-kernel formalism is likely to be useful even if the strong general claim needs qualification. I would send it to peer review, but I would ask the authors to either restrict the central claim to the asymptotic regimes or provide a bound on the higher-order cumulants, add error bars to the empirical figures, and ideally demonstrate the F/P scaling on at least one real architecture. My own verdict is conditional: the mechanism is plausible and the asymptotic calculations are solid, but the paper currently claims more than it proves.","headline":"A genuinely new mechanism for diffusion-model generalization via the covariance of the DSM proxy score, with clean asymptotic results for linear and NTK models, but the general claim leans on an unverified second-order cumulant truncation.","tokens_in":36242,"tokens_out":1537,"would_cite":true,"duration_ms":19049,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Noise in the training target, not memorized scores, is what lets diffusion models generalize, a path-integral theory argues.","keywords":["diffusion models","denoising score matching","generalization","inductive bias","score estimator variance","path integral","V-kernel","neural tangent kernel"],"falsifier":"Measure the skewness (third cumulant) of the DSM proxy-score target at a boundary point for finite $P$, and compare a model trained on that target against the effective-SDE prediction: if the third cumulant is non-negligible, the distribution predicted from Eq. (8) will deviate from the observed one, falsifying the Gaussian truncation. A simpler version is to scan $F/P$ and $\\epsilon$ in the one-dimensional three-point linear model of Fig. 2 and check whether the measured average learned distribution matches the semiclassical formula (18); systematic mismatch that grows with the third cumulant would show that the V-kernel alone does not carry the inductive bias.","tokens_in":35128,"feed_emoji":"🖼️","tokens_out":9010,"duration_ms":86308,"temperature":0.7,"pith_summary":"The paper claims that diffusion models generalize, filling gaps in the training set rather than merely reproducing it, because the denoising score matching objective trains against a noisy 'proxy score' that only equals the true score in expectation. Averaging over training-realization randomness turns the probability-flow ODE into an effective stochastic differential equation whose noise has covariance given by a 'V-kernel', built from the covariance of the score estimator; with an unbiased estimator and finitely many training examples, generalization occurs if and only if this V-kernel is nonzero. The paper derives explicit V-kernels for a naive memorizing estimator, for linear feature models, and for lazy infinite-width networks, and shows the kernel concentrates in boundary regions between training examples, which is why learned distributions resemble the training data with gaps filled in. This matters because it converts a puzzling empirical fact into a mechanism that can be computed from the DSM objective, the forward process, the model's features, and the training-set geometry.","feed_headline":"Noise in the training target drives diffusion generalization","feed_subtitle":"A V-kernel theory says diffusion models fill gaps exactly where the noisy score's variance is high","key_machinery":"The load-bearing object is the V-kernel, the covariance kernel of the learned score estimator's fluctuations, $V(x,t;x',t') = D_t \\operatorname{Cov}_\\theta[\\hat{s}(x,t),\\hat{s}(x',t')] D_{t'}$, defined for any score estimator and computed in closed form for three model classes. The paper's other central device is a time-reversed Martin-Siggia-Rose path integral representation of the probability-flow ODE, which turns the ensemble average of the learned distribution into a Gaussian average over score fluctuations; keeping only the mean and second cumulant yields the effective SDE of Proposition 3.1. The V-kernel carries the argument because it is the only place where finite-sample randomness of the DSM proxy score enters the typical dynamics, and its proxy-score covariance factor $C(x,t) = S_t^{-1} + \\nabla^2 \\log p(x|t)$ is what localizes generalization to boundary regions.","core_discovery":"The central discovery is a mechanism, not just a phenomenon: the noisy target in denoising score matching has a state-dependent covariance $C(x,t)$, and this covariance, propagated through the diffusion tensor and the model's feature kernel, becomes the noise term of an effective reverse SDE that describes the typical distribution a diffusion model learns. The paper shows that for an unbiased score estimator with finitely many training samples, the only thing separating true probability-flow dynamics from the model's typical sampling dynamics is this V-kernel noise. Concretely, Proposition 5.1 and Proposition 5.2 give V-kernels of the form $\\lim_{P\\to\\infty} (F/P) D_t \\mathbb{E}_{z''}[\\lambda_{t''}^2/\\mathbb{E}_t[\\lambda_t]^2\\, k(z;z'') C(z'') k(z'';z')] D_{t'}$ for linear models and lazy neural networks, while the naive estimator gives the local form $V = \\kappa D_t C(x,t) D_t \\delta(t-t')$. Because $C$ is large in boundary regions between nearby training points and small near isolated points or far from data, the typical learned distribution interpolates dense regions of the training set, preserves data dimensionality, and leaves outliers memorized.","pith_inferences":["Editorial inference: if the V-kernel is the mechanism, then the time weighting $\\lambda_t$ of the DSM objective could be engineered to sculpt which gaps are filled, for instance by down-weighting times where $C$ is large in undesired regions to suppress unwanted mode blending.","Editorial inference: the theory yields a falsifiable quantitative prediction for simple linear models: the measured average learned distribution should match the effective-SDE prediction over a range of $F/P$ and $\\epsilon$, with discrepancies appearing exactly when the score estimator's third cumulant becomes non-negligible.","Editorial inference: since $C$ is the Fisher information of the posterior over $x_0$, the same V-kernel formalism could be adapted to other training objectives such as sliced score matching, denoiser prediction, or noise prediction by swapping the target covariance, giving a unified treatment of generalization across training losses.","Editorial inference: the temporal autocorrelation structure of $V$ in the NTK case means the effective dynamics are non-Markovian in time, which may be a mechanism by which sampling spends extra time in boundary regions, an effect that could show up in measured trajectory statistics."],"forward_implications":["Even a model that memorizes its training samples at every reverse step produces a nontrivial V-kernel and hence generalization, as Proposition 4.1 shows for the naive interleaved estimator.","Generalization disappears as $P\\to\\infty$ unless the number of features $F$ grows with $P$; the ratio $F/P$ controls the strength of the V-kernel, predicting a sharp memorization-to-generalization transition as capacity outstrips sample count.","Asymmetries in the forward process's noise matrix $G_t$ enter $V$ through the diffusion tensor, so the forward process is not just a sampler detail but an inductive-bias dial.","Feature-related spectral biases multiply the proxy-score covariance in the V-kernel, which explains why the same data distribution is generalized differently, for example square versus cross gap-filling, for Gaussian versus Fourier features and different data orientations.","Outliers, isolated points, and regions far from training data have near-zero proxy-score covariance, so generalization through variance is a boundary-smearing mechanism rather than an extrapolation-to-novelty mechanism."],"supporting_citations":[{"why":"Supplies the Martin-Siggia-Rose path integral used to convert ensemble-averaged PF-ODE dynamics into an effective SDE.","marker":"Martin et al., 1973"},{"why":"Shows the DSM objective shares the true score as its optimum, so the proxy score is a noisy target with the same expectation.","marker":"Vincent, 2011"},{"why":"Connects DSM to score-based generative modeling and identifies the training-distribution score as the objective's optimum.","marker":"Song & Ermon, 2019"},{"why":"Introduces the probability-flow ODE and VP-SDE reverse process that the paper's effective dynamics generalize.","marker":"Song et al., 2021"},{"why":"Documents the large low-noise variance of the DSM target and supplies the EDM forward-process parameterization used throughout.","marker":"Karras et al., 2022"},{"why":"Defines the neural tangent kernel whose spectral structure underlies the NTK V-kernel of Proposition 5.2.","marker":"Jacot et al., 2018"},{"why":"Defines the lazy-training regime in which the NTK result is derived.","marker":"Chizat et al., 2019"}],"fun_headline_variants":["Diffusion models fill gaps where score noise is high","Variance in the target shapes diffusion inductive bias","Noisy score target explains diffusion generalization","Score noise, not score accuracy, drives diffusion gaps","Effective reverse SDE reveals diffusion's gap-filling bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the randomness in the learned score is nearly Gaussian, so that only its mean and covariance matter; if the score estimator's fluctuations are significantly non-Gaussian, the V-kernel picture of generalization would be incomplete.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion models fill gaps where score noise is high","Variance in the target shapes diffusion inductive bias","Noisy score target explains diffusion generalization","Score noise, not score accuracy, drives diffusion gaps","Effective reverse SDE reveals diffusion's gap-filling bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1465,"prompt_tokens":1002,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":618,"tokens_out":463,"duration_ms":4762,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:29:33.539110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the skewness (third cumulant) of the DSM proxy-score target at a boundary point for finite $P$, and compare a model trained on that target against the effective-SDE prediction: if the third cumulant is non-negligible, the distribution predicted from Eq. (8) will deviate from the observed one, falsifying the Gaussian truncation. A simpler version is to scan $F/P$ and $\\epsilon$ in the one-dimensional three-point linear model of Fig. 2 and check whether the measured average learned distribution matches the semiclassical formula (18); systematic mismatch that grows with the third cumulant would show that the V-kernel alone does not carry the inductive bias.","supporting_citations":[{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the neural tangent kernel whose spectral structure underlies the NTK V-kernel of Proposition 5.2."},{"cited_title":"On lazy training in differentiable programming","cited_arxiv_id":null,"evidence_quote":"Defines the lazy-training regime in which the NTK result is derived."}],"review_version":1}