From Score Approximation to Distribution Approximation in Score-Based Diffusion Models

Lan V. Truong

REVIEW 3 major objections 4 minor 13 references

Accurate L2 score approximation in a diffusion model guarantees the generated distribution is close to the target, up to a prior-mismatch term.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:29 UTC pith:D3PJHXJ3

load-bearing objection A clean, explicit KL bound connecting score approximation to distribution approximation, but the proof has a false claim about terminal laws and an unproven Novikov-existence step; the core result is likely correct and worth publishing after revision. the 3 major comments →

arxiv 2607.22199 v1 pith:D3PJHXJ3 submitted 2026-07-24 cs.LG cs.ITmath.ITstat.ML

From Score Approximation to Distribution Approximation in Score-Based Diffusion Models

Lan V. Truong This is my paper

classification cs.LG cs.ITmath.ITstat.ML MSC 60J6068T0762B10

keywords score-based diffusion modelsscore approximationKL divergenceGirsanov theoremuniversal approximationreverse-time SDEgenerative modelsdata processing inequality

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Score-based diffusion models generate data by reversing a forward noising process, and the quality of generation depends on how well a neural network approximates the unknown score function. Classical universal approximation results guarantee score accuracy in function space, but not that the generated probability distribution is accurate. This paper proves the missing link: if the score network is accurate in L2 with error below ε, then the reverse-diffusion output is accurate in Kullback-Leibler divergence with error below δ + ½ g²max ε, where δ is the mismatch between the forward terminal distribution and the chosen prior. The proof works by combining the universal approximation theorem, Girsanov's theorem on path space, and the data processing inequality for relative entropy. The result gives a direct theoretical justification that improving score approximation really does improve the generated distribution.

Core claim

The paper's central claim is that score approximation and distribution approximation are quantitatively equivalent for score-based diffusion models: the KL divergence between the true data distribution µ and the generated distribution µ_θ is bounded by the prior mismatch KL(p_T∥p_prior) plus a constant times the squared L2 score error measured under the forward marginals. Explicitly, for every ε>0 a neural network can be chosen so that ∥sθ−s∥²_{L2(µ)}<ε and KL(µ∥µ_θ)<δ+½g²_max ε. The proof establishes an intermediate identity for path measures: KL(P∥Q_θ) equals the prior mismatch plus ½∫g²(t)E_{p_t}∥sθ−s∥² dt, and then shows the marginal KL is no larger than the path KL. This transfers unive

What carries the argument

The central object is the score function s(x,t)=∇_x log p_t(x) and the reverse-time SDE it defines. The key identity is KL(P∥Q_θ)=KL(p_T∥p_prior)+½∫_0^T g²(t)E_{p_t}∥sθ(X_t,t)−s(X_t,t)∥² dt, obtained from Girsanov's theorem; this converts score error into path-space divergence. The data processing inequality for relative entropy then converts path-space divergence into a bound on the marginal KL divergence at the data time. Universal approximation closes the loop by supplying an arbitrary small L2 score error.

Load-bearing premise

The load-bearing premise is that the approximating network satisfies the Novikov exponential-integrability condition while being L2-accurate; the paper verifies this in examples such as sub-Gaussian data with linear-growth networks, but does not prove it for every data distribution.

What would settle it

Find or construct a data distribution and an L2-approximating score network whose score error is small but whose exponential integral ∫ g²∥sθ−s∥² along true diffusion paths is infinite (so Girsanov cannot apply), and show that the generated distribution still has large KL divergence. Simpler: numerically estimate the right-hand side δ+½g²maxε on a heavy-tailed dataset with a trained linear-growth network; if the observed KL exceeds the bound, the theorem's assumptions are not being met and mark exactly where.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the prior matches the forward terminal distribution, arbitrarily good score approximation yields arbitrarily small KL divergence between generated and true distributions.
The bound is explicit and monotone: halving the L2 score error cuts the score-dependent part of the KL error by half, so score training effort translates directly into distribution quality.
The theorem applies to variance-preserving and variance-exploding diffusions when data are sub-Gaussian or have finite exponential moments and the score network has linear growth, because those settings satisfy the Novikov condition.
The result holds for arbitrary Euclidean data distributions whose score lies in L2 and satisfies the regularity conditions; no low-dimensional structural assumption is needed for the approximation step.
The prior-mismatch term δ is irreducible in this analysis, so the bound cannot go below δ even with a perfect score network.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If δ is non-negligible, the bound predicts that improving the score network beyond the point where ½g²maxε<δ yields diminishing returns; the operative bottleneck becomes the choice of prior distribution.
For heavy-tailed data where exponential integrability of the score error is not assured, the theorem's assumptions may fail even though L2 approximation succeeds; a bounded or localized score approximation could restore the Girsanov step and is a natural testable extension.
The identity suggests a diagnostic: measure the L2 score error and the terminal prior KL during training; if KL(µ∥µ_θ) exceeds δ+½g²maxε significantly, the discrepancy points to a breakdown in the Novikov condition or in the regularity assumptions rather than in score accuracy.
Transferring the same proof to other divergences (Wasserstein or total variation) would require replacing Girsanov's Radon-Nikodym control; the paper flags this as future work, and a first step would be bounding those metrics in terms of the same L2 score error under additional tail conditions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A clean, explicit KL bound connecting score approximation to distribution approximation, but the proof has a false claim about terminal laws and an unproven Novikov-existence step; the core result is likely correct and worth publishing after revision.

read the letter

Colleague,

The paper proves a clean explicit bound: if a neural score network matches the true score in L2(µ) to within ε, then KL between generated and true data distributions is below KL(p_T∥p_prior) + ½g_max²ε. The proof runs through the standard path-space KL identity for reverse-time SDEs, Hornik's universal approximation, and DPI. The high-level statement is what you'd expect; the contribution is making it explicit and unconditional on statistical estimation. That is worth something.

The path-space identity in Proposition 1 is classical, and the final bound is a direct corollary of it plus Hornik. So novelty is real but modest. The paper is honest about that, and the writing is clear. The examples in Section 4 showing when Assumption 2 holds are useful.

Now the soft spots. There are two.

First, the proof of Proposition 1 contains a false claim. It constructs an auxiliary process Y starting from X_T (the true terminal state) and then asserts Y has the same terminal distribution as the learned process Xθ. That is not the case: Xθ is initialized from p_prior, while X_T has law p_T. The final KL formula is still correct—you can get it directly from Girsanov with the initial-density ratio—but as written this step is wrong. It needs fixing.

Second, Theorem 1 asserts the existence of a neural network satisfying Assumption 2 (Novikov) via Hornik. Hornik guarantees L2(µ) approximation, not exponential integrability of the score mismatch. For arbitrary data distributions with heavy tails, L2 closeness does not imply the exponential moment is finite. The paper gives sufficient conditions (sub-Gaussian data, linear-growth networks) but not a general existence proof. The theorem overclaims generality. This is also fixable—for instance by requiring a bounded activation and imposing tail conditions on µ—but it's a real gap.

Neither flaw sinks the core idea. The bound is true and worth having. But the paper needs revision before it is a solid reference. For a reader who works on diffusion theory, it's a useful discussion piece; for a general ML audience, it's a modest footnote.

My recommendation: send it to peer review but expect heavy revision. The proof defects are addressable, and the result, if properly stated, deserves a place in the literature.

Referee Report

3 major / 4 minor

Summary. The paper aims to prove a quantitative transfer from score-function approximation to distribution approximation in score-based diffusion models. Building on Hornik's universal approximation theorem, Girsanov's theorem, and the data-processing inequality, the author derives an explicit KL bound: for any ε > 0 there should exist a neural network sθ with L2(µ) score error below ε and satisfying the Novikov condition, such that KL(µ∥µθ) < KL(p_T∥p_prior) + (1/2) g_max² ε. The proof proceeds through a path-space KL identity (Proposition 1) and a projection step (Proposition 2), then combines them with Hornik's density result (Theorem 1). The paper also discusses VP/VE diffusion examples and possible extensions.

Significance. If fully established, this result would provide a clean approximation-theoretic justification for why accurate score networks should yield accurate generated distributions, complementing statistical-learning analyses such as Chen et al. The explicit bound, including the unavoidable prior-mismatch term δ, is conceptually attractive and the choice of ingredients is natural. The paper is also careful to identify δ as irreducible. However, the proof as written contains a false claim in the path-measure comparison, an invalid Jensen step in the projection argument, and an unsupported existence assertion for a network satisfying the Novikov condition. These are central to the main theorem, so the manuscript currently overclaims and requires substantial revision. The underlying strategy is likely correct under additional explicit hypotheses, and the gaps appear fixable; the paper merits a major revision rather than rejection.

major comments (3)

[§5, Proposition 1 (proof)] The auxiliary process Y is defined with terminal condition Y_T = X_T, so its terminal law is p_T, while the learned reverse process Xθ has terminal law p_prior. The statement that Y has 'exactly the same ... terminal distribution as the learned reverse-time SDE' is false unless p_prior = p_T. Therefore the assertion L(Y) = L(Xθ), and the conclusion that Y induces Qθ, is not valid. The path-space identity (12) may be recoverable by comparing P and Qθ directly via conditional Girsanov, but the proof as written is invalid and the central link to Theorem 1 is not established.
[§7, Theorem 1 and Eq. (24)] The proof claims that a network satisfying both Assumptions 1 and 2 with the desired L2 error exists and that this is 'guaranteed by (24)'. However, Eq. (24) is Hornik's theorem, which only provides a globally Lipschitz network with L2(µ) score error < ε — i.e., Assumption 1. It does not imply the Novikov condition (Assumption 2): L2 closeness alone does not ensure exponential integrability of ∫ g²(t)∥sθ−s∥² dt. Moreover, Assumption 2 is a condition that involves sθ, so the theorem's phrasing that 's satisfies Assumption 2' is ill-posed. The existence part of Theorem 1 requires additional hypotheses (e.g., exponential moments and bounded network growth); without such hypotheses the theorem is unproven.
[§6, Proposition 2 (Eq. (22))] The claimed Jensen inequality is wrong. Since log is concave, Jensen gives log E_Qθ[R|X0] ≥ E_Qθ[log R|X0], not the displayed ≤ E_P[log R|X0] in Eq. (22). Thus the proof of KL(µ∥µθ) ≤ KL(P∥Qθ) is invalid as written. The inequality is a standard data-processing inequality, but the paper's derivation does not establish it. Because Theorem 1 relies on this projection step, the proof must be corrected, either by citing the DPI directly or by a valid argument.

minor comments (4)

[§3.1 and §5] The symbol µ is used both for the data distribution on R^d and for the measure pt(x)dx dt on R^d × [0,T]; this is confusing and should be changed, e.g., to ν.
[§5, Lemma 1] The reverse-time SDE is written as a forward SDE with a condition at time t=T. To apply the standard strong-existence theorem, one should introduce the time-reversal variable s=T−t or explicitly clarify the convention.
[§4, Assumption 2] Assumption 2 is not an assumption on the data alone but a condition involving the learned network sθ. Its role should be clarified at the point of introduction: it is a hypothesis to be verified for the chosen approximating network, not a global assumption that can be imposed before sθ is constructed.
[§7, Remark 3] Remark 3 states 'if the score network approximates the true score function with an L2 error of at most ε...' This is informal: Theorem 1 is existential, not a statement about an arbitrary network. The wording could mislead readers into thinking the bound holds for every network with small score error.

Circularity Check

0 steps flagged

No significant circularity: the KL bound is a conditional composition of Girsanov, the data-processing inequality, and Hornik's theorem; the Assumption 2 gap is a proof gap, not circularity.

full rationale

Walking the derivation chain: Section 5, Proposition 1 derives an exact path-space KL identity from Girsanov's theorem, conditional on Assumptions 1 and 2. Section 6, Proposition 2 applies the data-processing inequality to obtain KL(µ∥µθ) ≤ KL(P∥Qθ). Section 7 invokes Hornik's theorem to produce an L2(µ)-accurate network satisfying Assumption 1, then substitutes this into the composition to obtain the stated KL bound. The quantities ε, δ, and g_max are exogenous inputs; no parameter is fitted to the target KL divergence, and no target-dependent constant is smuggled into the hypotheses. The conclusion is genuinely conditional: if a network is L2-accurate and satisfies Novikov's condition, then the KL bound follows from the cited external results (Anderson, Hornik, Girsanov/Øksendal, Cover-Thomas). None of the load-bearing results are self-citations of the present author, and Hornik's theorem is an independent, standard approximation result. The proof does contain a substantive omitted-proof gap: the statement 'whose existence is guaranteed by (24)' in the proof of Theorem 1 overclaims, because Hornik's theorem supplies L2 accuracy and Lipschitz regularity but does not by itself establish Assumption 2 (the Novikov condition). This is a rigor/correctness concern about the existence assertion, not a circularity: the theorem does not define score approximation in terms of KL divergence, nor does it fit any parameter to the predicted distribution. Therefore the honest circularity finding is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No new parameters, entities, or fitted constants; the theory rests on standard stochastic calculus and regularity assumptions. The main undeclared load is Assumption 2, whose satisfiability by the constructed network is asserted rather than proved.

axioms (5)

domain assumption Global Lipschitz continuity of f, s, sθ (Assumption 1)
Needed for strong existence/uniqueness of the reverse SDEs; fails for arbitrary data distributions, though it holds for Gaussian/log-concave margins.
domain assumption Novikov condition for the score-error exponential martingale (Assumption 2)
Required for Girsanov. The paper argues it holds for sub-Gaussian data and linear-growth networks but does not prove existence of an L2-accurate network satisfying it in general.
standard math s ∈ L^2(µ)^d and Hornik's universal approximation theorem applies
Needed to find sθ with arbitrarily small L2 score error; Hornik's theorem is cited as [9], and the measure µ has finite total mass T.
domain assumption Absolute continuity/finite KL between p_T and p_prior (δ finite)
The bound uses KL(p_T∥p_prior); if infinite, the bound is vacuous.
standard math Conditional bridge law of a diffusion depends only on drift/diffusion, not terminal marginal
Used implicitly to justify the Girsanov comparison in Proposition 1 when the terminal laws of the two processes differ.

pith-pipeline@v1.3.0-alltime-deepseek · 10126 in / 22119 out tokens · 215897 ms · 2026-08-01T05:29:39.764805+00:00 · methodology

0 comments

read the original abstract

Score-based diffusion models have achieved remarkable empirical success in generative modeling, yet their approximation-theoretic foundations remain incomplete. In particular, although classical universal approximation theorems guarantee that neural networks can approximate score functions, it remains unclear whether such approximation guarantees translate into approximation of the probability distributions generated by reverse diffusion processes. In this paper, we establish a rigorous quantitative connection between these two notions. Specifically, we prove that if a neural network approximates the true score function sufficiently accurately, then the probability distribution generated by the corresponding reverse diffusion model is close to the target data distribution in Kullback-Leibler (KL) divergence, up to an irreducible mismatch between the terminal distribution of the forward diffusion process and the prior used to initialize the reverse process. More precisely, we derive an explicit upper bound on the distribution approximation error in terms of the score approximation error, the diffusion noise schedule, and the terminal prior mismatch. Our analysis combines Hornik's universal approximation theorem, Girsanov's theorem on path space, and the data processing inequality for relative entropy. Complementary to recent work that studies score approximation under finite-sample statistical settings and structural assumptions on the data distribution, our work develops an approximation-theoretic analysis based on classical neural network approximation theory. The resulting theorem provides a simple and explicit guarantee linking neural network approximation of score functions to approximation of the probability distributions generated by reverse diffusion models.

discussion (0)

Reference graph

Works this paper leans on

13 extracted references

[1]

Brian D. O. Anderson. Reverse-time diffusion equation models.Stochastic Processes and their Applications, 12(3):313–326, 1982

1982
[2]

Diffusion schr¨ odinger bridge with applications to score-based generative modeling

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schr¨ odinger bridge with applications to score-based generative modeling. InAdvances in Neural Informa- tion Processing Systems (NeurIPS), 2021

2021
[3]

Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data. InInternational Conference on Machine Learning (ICML), 2023

2023
[4]

Cover and Joy A

Thomas M. Cover and Joy A. Thomas.Elements of Information Theory. Wiley, 2 edition, 2006

2006
[5]

Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

1989
[6]

U. G. Haussmann and E. Pardoux. Time reversal of diffusion processes. InStochastic Differ- ential Systems: Filtering and Control, volume 69 ofLecture Notes in Control and Information Sciences, pages 176–182. Springer, 1985

1985
[7]

U. G. Haussmann and E. Pardoux. Time reversal of diffusions.The Annals of Probability, 14(4):1188–1205, 1986

1986
[8]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020

2020
[9]

Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

Kurt Hornik. Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

1991
[10]

Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics

Ioannis Karatzas and Steven E. Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics. Springer, New York, 2 edition, 1991

1991
[11]

Springer, 5 edition, 1998

Bernt Øksendal.Stochastic Differential Equations: An Introduction with Applications. Springer, 5 edition, 1998

1998
[12]

Generative modeling by estimating gradients of the data distribution

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

2019
[13]

Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021. 18

2021

[1] [1]

Brian D. O. Anderson. Reverse-time diffusion equation models.Stochastic Processes and their Applications, 12(3):313–326, 1982

1982

[2] [2]

Diffusion schr¨ odinger bridge with applications to score-based generative modeling

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schr¨ odinger bridge with applications to score-based generative modeling. InAdvances in Neural Informa- tion Processing Systems (NeurIPS), 2021

2021

[3] [3]

Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estima- tion and distribution recovery of diffusion models on low-dimensional data. InInternational Conference on Machine Learning (ICML), 2023

2023

[4] [4]

Cover and Joy A

Thomas M. Cover and Joy A. Thomas.Elements of Information Theory. Wiley, 2 edition, 2006

2006

[5] [5]

Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

George Cybenko. Approximation by superpositions of a sigmoidal function.Mathematics of Control, Signals and Systems, 2(4):303–314, 1989

1989

[6] [6]

U. G. Haussmann and E. Pardoux. Time reversal of diffusion processes. InStochastic Differ- ential Systems: Filtering and Control, volume 69 ofLecture Notes in Control and Information Sciences, pages 176–182. Springer, 1985

1985

[7] [7]

U. G. Haussmann and E. Pardoux. Time reversal of diffusions.The Annals of Probability, 14(4):1188–1205, 1986

1986

[8] [8]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020

2020

[9] [9]

Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

Kurt Hornik. Approximation capabilities of multilayer feedforward networks.Neural Networks, 4(2):251–257, 1991

1991

[10] [10]

Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics

Ioannis Karatzas and Steven E. Shreve.Brownian Motion and Stochastic Calculus, volume 113 ofGraduate Texts in Mathematics. Springer, New York, 2 edition, 1991

1991

[11] [11]

Springer, 5 edition, 1998

Bernt Øksendal.Stochastic Differential Equations: An Introduction with Applications. Springer, 5 edition, 1998

1998

[12] [12]

Generative modeling by estimating gradients of the data distribution

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

2019

[13] [13]

Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021. 18

2021