Pith. sign in

REVIEW 3 major objections 8 minor 15 references

Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling

T0 review · 3 major / 8 minor · reviewed 2026-07-10 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small Score Error Can Still Break Diffusion Sampling

desk verdict Clean counterexample showing on-path score error can't certify Euler-Maruyama stability; positive projection result is a nice bonus. read the letter →

arxiv 2607.08757 v1 pith:QUO434V5 submitted 2026-07-09 stat.ML cs.LGcs.NAmath.NAmath.PR

Yiwei Zhou This is my paper
classification stat.MLcs.LGcs.NAmath.NAmath.PR MSC 65C3060H3560J6068T07
keywords score-baseddiffusionnumericalstabilityEuler-MaruyamaWassersteinconvergencescorematchingdenoiserprojectionreverse-timeSDE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Score-based diffusion models are trained to minimize the average discrepancy between the learned score and the true score, where the average is taken over the forward noising process. This paper proves that making that average error arbitrarily small does not guarantee that the discrete numerical sampler used at inference time is stable. The author constructs a single smooth score field whose forward-marginal error is below any chosen threshold, whose continuous-time reverse process is nonexplosive with all moments finite and arbitrarily close to the exact reverse process in total variation, yet whose Euler-Maruyama discretizations produce samples whose every positive moment diverges. The mechanism is a superlinear perturbation placed in a remote tail region that carries negligible mass under the forward marginals. The continuous-time process almost never reaches this region, so its path law stays close to the exact one. But a rare numerical trajectory can overshoot into the region, after which each explicit Euler step compounds the superlinear growth faster than the entry probability decays. The result is that the discretized samples converge in probability to the correct endpoint while simultaneously diverging in every Wasserstein distance. The same obstruction is shown to persist within a single fixed neural architecture: a family of bounded, globally Lipschitz denoisers in one fixed GELU network can have vanishing forward-marginal error and vanishing path-space total variation distance, while their Euler-Maruyama endpoints diverge in every W_p. As a positive counterweight, the author shows that if the data support lies in a known bounded closed convex set, projecting the learned denoiser onto that set before forming the score preserves pointwise accuracy, prevents the superlinear amplification cascade, and restores Wasserstein convergence under mild regularity.

What carries the argument

Variance-exploding forward diffusion; Tweedie's identity relating score to denoiser; Euler-Maruyama discretization of the reverse-time SDE; path-space total variation via Girsanov change of measure; Lyapunov function V_m(x) = (1 + x^2)^m for nonexplosion; nested geometric grids t_j = delta * exp((j/K) log(T/delta)); rare-event amplification recursion z_j >= c_0 * z_{j+1}^{1+alpha}; denoiser projection onto bounded closed convex set C; explicit chain representation via telescoping products of contraction ratios

What would settle it

If one could exhibit a forward marginal and grid family for which the rare-event entry probability always dominates the superlinear amplification, the moment divergence would fail and the counterexample would not apply to that setting.

Watch

Extended reading notes

Core claim

The paper's central object is the gap between forward-marginal score accuracy and trajectory-wise numerical stability. The author identifies a specific mechanism that creates this gap: a superlinear perturbation hidden in a low-mass tail region is invisible to the forward-marginal error budget and harmless to the continuous-time reverse process, but an explicit Euler step that lands in that region triggers a self-reinforcing cascade of superlinear amplification. The rare-event cost of entering the region grows like exp(-C K^{2/alpha}) while the amplification grows like (1+alpha)^K, so the amplification dominates and every positive moment of the discrete endpoint diverges. This separation of弱

Load-bearing premise

The moment-divergence proof requires that the rare-event probability of a numerical trajectory entering the amplification region (which decays roughly like exp(-C K^{2/alpha}) for a grid of K steps) is outweighed by the super-exponential growth of the amplification itself (which grows like (1+alpha)^K). This separation of scales is the structural linchpin of the counterexample and depends on the Gaussian tail of the forward marginal and the use of geometric grids.

Editorial extensions

If this is right

  • Standard score-matching loss bounds alone cannot certify that a diffusion sampler produces finite-moment samples; any such certificate must control the score along states the numerical sampler can reach, not just states visited by the forward process.
  • Two score fields with identical forward-marginal error budgets can have opposite numerical stability properties, so the scalar error budget does not rank samplers by stability.
  • Denoiser projection onto a known data-supporting convex set is a provably stabilizing intervention that does not increase approximation error, giving a theoretical justification for the common practice of clipping predicted clean samples in image diffusion.
  • The obstruction extends to probability-flow ODEs and to arbitrary dimensions, so it is not specific to stochastic samplers or to one-dimensional constructions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The result suggests that adaptive or implicit discretization schemes might avoid the amplification cascade by not committing to explicit superlinear updates, though the paper does not analyze alternative integrators.
  • The separation between rare-event cost and amplification depends on the Gaussian tail of the forward marginal; heavier-tailed forward processes might change the balance and could be a setting where the obstruction weakens.
  • For Schrodinger bridge methods, where drifts are learned along bridge marginals and then discretized, an analogous gap between bridge-marginal accuracy and numerical stability may arise; the author flags this as a direction but does not pursue it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper identifies and rigorously demonstrates a fundamental gap between forward-marginal score accuracy and numerical stability of Euler-Maruyama discretizations in diffusion sampling. The central result (Theorem 3.1) constructs a single smooth score field with arbitrarily small on-path L² error, whose continuous-time reverse process is nonexplosive with all moments finite and arbitrarily close in path-space total variation to the exact reverse process, yet whose Euler-Maruyama endpoints converge in probability while every positive moment diverges—so W_p diverges for all p≥1. The mechanism is a superlinear perturbation placed in a remote tail region carrying negligible forward-marginal mass: rare numerical trajectories enter this region and trigger repeated superlinear amplification that outweighs the rare-event probability cost. The paper extends this to fixed finite neural architectures (Theorem 3.5, Corollary 3.6) and provides a positive result showing that projecting the learned denoiser onto a known bounded convex set containing the data support restores grid-uniform moment bounds and Wasserstein convergence (Section 4). Experiments with a small DiT-style network provide a constructive realization test.

Significance. This is a significant contribution that identifies a genuine and previously unarticulated obstruction in diffusion model theory. The key insight—that on-path score accuracy under forward marginals is blind to off-path numerical instability because discretized trajectories can enter poorly controlled regions—is both mathematically precise and practically relevant. The counterexample construction is explicit and the proof is verifiable step by step: the separation between the rare-event cost (exp(−C·K^{2/α})) and the super-exponential amplification ((1+α)^K) is a mathematical consequence of the construction, not an assumption. The extension to fixed finite neural architectures (Theorem 3.5) within a concrete DiT-S/2 configuration (Corollary 3.6) substantially strengthens the practical relevance. The positive result on denoiser projection (Theorem 4.3) provides a clean stability interpretation of the clipping heuristic used in practice. The paper ships falsifiable experimental predictions and constructive proofs with explicit constants.

major comments (3)
  1. [Theorem 3.1, Step 5] The moment divergence proof hinges on the scale separation between the rare-event probability exp(−C₃(K^{2/α}+K)) and the amplification term of order (1+α)^K. This separation is correct for the stated construction with geometric grids and Gaussian-tailed forward marginals. However, the paper does not discuss the robustness of this separation to grid choice or tail behavior. While the theorem is existential and the construction is valid as stated, a brief remark acknowledging that the divergence depends on the geometric grid structure (where h_j ~ 1/K) and Gaussian tails would strengthen the paper and preempt questions about whether the obstruction persists under adaptive or non-uniform grids.
  2. [Section 5 and Appendix C] The stress-initialized protocol (Appendix C) explicitly states it 'does not estimate the natural entry probability.' This is honest, but it leaves open whether the amplification mechanism can be triggered under natural (non-stress-initialized) sampling. The paper's theoretical results show that the rare-event probability is nonzero but exponentially small. A brief discussion of whether the experimental setup could be extended to estimate natural entry rates—even at a qualitative level—would help calibrate the practical severity of the obstruction. As presented, the experiments confirm the mechanism exists but do not speak to its frequency under normal operation.
  3. [Theorem 3.5] The neural architecture result constructs a sequence of parameter values θ_m within one fixed DiT architecture. The proof relies on approximating the ideal remote-tail denoiser F_m(x) = −λΓ_{R_m}(x)x³ on the tube [−Q_m, Q_m] with accuracy ζ_m = Q_m^{-1}. The global output bound (Eq. 31) grows polynomially in Q_m, and the Gaussian tail beyond Q_m decays exponentially. This argument is sound. However, the paper could clarify whether the polynomial growth bound (Eq. 31) is tight or merely sufficient for the argument. If it is not tight, stating this would avoid giving the impression that the network must exhibit polynomial growth in practice.
minor comments (8)
  1. The paper states (end of Section 3.1) that the same counterexample can be expressed in VP/DDPM coordinates 'by a standard time-dependent rescaling.' A one-line formula for this rescaling would help readers verify the claim.
  2. In Theorem 3.1, Step 5, the induction giving Eq. (16) requires c₀^{1/α}A > 1 and c₀A^{1+α} ≥ A. The text says 'Taking A still larger if necessary,' but the dependence of c₀ on λ, α, a, b, δ, M is stated earlier. A brief note that A is chosen after these constants are fixed would clarify the induction in Eq. (16).
  3. The stress-initialized protocol uses perturbations sampled uniformly from [−0.35, 0.35]. The choice of 0.35 is not motivated. A sentence explaining why this range was selected would strengthen this claim.
  4. Equation (3) defines the perturbed score using χ_R(x), but the smoothness of χ_R is only stated as 'even cutoff χ_R ∈ C^∞(R; [0,1]).' An explicit reference to a standard construction or a brief description would help.
  5. The notation ŝ^{safe} and ŝ^{bad} in Corollary 3.4 uses superscripts that are typeset differently from the main text's ŝ_t notation. Consider unifying.
  6. In Proposition 4.1, the Gaussian term G_k is stated as N(0, σ²_k I_d), but the dimension d is not explicitly introduced in the proposition statement (it is implicit from the d-dimensional setting in Eq. (2)). A brief mention would help.
  7. The reference list could include recent work on adaptive step-size controllers for diffusion SDEs, which are directly relevant to the grid-dependence discussion the paper identifies.
  8. Figure 1 and Figure 2 are referenced but not visible in the text provided. Ensure axis labels, legends, and captions are clear in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the counterexample is a self-contained mathematical construction and the positive result uses standard external tools.

full rationale

The paper's central claim is existential: it constructs a specific smooth score field (Eq. 3) and proves, step by step, that it has small on-path L² error, stable continuous-time dynamics, close path-space TV, yet divergent Euler-Maruyama moments. Each step of Theorem 3.1 uses standard mathematical tools: dominated convergence (Step 1), Lyapunov/coercivity arguments (Step 2), pathwise uniqueness coupling (Step 3), truncation-based localization (Step 4), and explicit rare-event vs. amplification scale separation (Step 5). No step reduces to a fitted parameter or a self-definitional loop. The positive result (Theorem 4.3) relies on Tweedie's identity, Girsanov's theorem, Pinsker's inequality, and the geometric fact that conditional expectation of a compactly supported variable lies in the convex hull — all standard, externally verifiable results. The neural extension (Theorem 3.5) constructs explicit DiT parameters approximating the ideal cubic profile, with approximation quality verified by Taylor expansion of GELU/LayerNorm. No self-citation is load-bearing for the logical chain. The derivation is self-contained against external mathematical benchmarks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is a mathematical construction with no invented physical entities. The free parameters (λ, α, R, A) are proof artifacts chosen to demonstrate the mechanism, not fitted to empirical data. The axioms are standard results from stochastic calculus and Bayesian inference.

free parameters (4)
  • λ (perturbation amplitude) = arbitrary positive constant
    Controls the strength of the superlinear perturbation. Not fitted to data; chosen as an arbitrary positive constant to demonstrate the mechanism.
  • α (superlinear growth exponent) = arbitrary positive constant
    Controls the degree of superlinearity in the perturbation. Not fitted; any positive value works for the construction.
  • R (cutoff radius) = chosen large enough to satisfy error tolerance ε
    Determines where the perturbation activates. Chosen based on the desired forward-marginal error bound ε, not fitted to data.
  • A (shell constant) = chosen large enough for amplification estimate
    Used in the rare-event construction to ensure the amplification recursion holds. Not fitted; chosen for the proof to work.
assumptions (4)
  • standard math Tweedie's identity: s_t(x) = (D_t(x) - x)/t
    Used in Section 2 to relate the score to the denoiser. Standard result in Bayesian inference.
  • domain assumption Data support is compact: supp(μ_0) ⊂ B(0, M)
    Used throughout to bound the exact denoiser and score. Stated in Section 2. Necessary for the linear growth bound on the exact score.
  • standard math Standard Euler-Maruyama convergence for globally Lipschitz drifts (Higham et al. 2002, [15])
    Invoked in Step 4 of Theorem 3.1 proof for the truncated drift. Standard numerical analysis result.
  • standard math Girsanov's theorem and Pinsker's inequality
    Used in Theorem 3.5 and Theorem 4.3 to bound path-space TV distance via KL divergence. Standard stochastic calculus tools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling." pith.science (2026). https://pith.science/paper/QUO434V5

@misc{pith2026260708757,
  author       = {Pith},
  title        = {Pith review of: Score Accuracy Along the Forward Diffusion Does Not Certify Numerical Stability in Diffusion Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUO434V5}},
  note         = {Machine review of arXiv:2607.08757}
}
abstract

Score matching controls average error under the forward marginals, but a discretized reverse-time sampler evaluates the learned score along its own trajectory. We show that small forward-marginal error does not guarantee numerical stability. We construct a single smooth score field with arbitrarily small forward-marginal $L^2$ error. The learned reverse-time process is nonexplosive, has moments of every order, and can be arbitrarily close to the exact reverse-time process in path-space total variation. Yet its Euler--Maruyama discretizations converge in probability while every positive moment diverges. Thus weak convergence can hold even though every Wasserstein distance $W_p$, $p\ge1$, diverges. The same failure can occur within one fixed finite neural architecture. We construct a family of bounded, globally Lipschitz denoisers for which both the forward-marginal error and the path-space total variation distance tend to zero, while their Euler--Maruyama endpoints diverge in every $W_p$. For compactly supported data, we also give a simple positive result. Projecting the learned denoiser onto a known bounded closed convex set containing the support preserves pointwise accuracy, gives grid-uniform moment bounds, and yields Wasserstein convergence under mild local regularity. Experiments with a small fixed DiT-style network show large growth along rare numerical trajectories and its suppression by denoiser projection, while overall trajectory errors remain small.

Figures

Figures reproduced from arXiv: 2607.08757 by the authors.

Figure 1
Figure 1. Stepwise coordinatewise RMS magnitude rk = (Eb|Yk| 2 ) 1/2 for the neural and paired target-profile chains. The trajectories also show a simple feedback mechanism: once the chain reaches a region where forward-marginal error gives little control, one inaccurate update can move later score queries farther from the reference path and amplify the deviation. Here this produces repeated the amplification. 5.2 Projection … view at source ↗
Figure 2
Figure 2. Stepwise coordinatewise RMS magnitude for the paired raw and projected chains under [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

  1. [1]

    Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling through Stochastic Differential Equations.International Conference on Learning Representations, 2021

    work page 2021

  2. [2]

    S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.International Conference on Learning Representations, 2023

    work page 2023

  3. [3]

    H. Lee, J. Lu, and Y. Tan. Convergence of Score-Based Generative Modeling for General Data Distributions.Proceedings of the 34th International Conference on Algorithmic Learning Theory, PMLR 201:946–985, 2023

    work page 2023

  4. [4]

    Benton, V

    J. Benton, V. De Bortoli, A. Doucet, and G. Deligiannidis. Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization.International Conference on Learning Representations, 2024

    work page 2024

  5. [5]

    X. Gao, H. M. Nguyen, and L. Zhu. Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models.Journal of Machine Learning Research, 26(43):1–54, 2025

    work page 2025

  6. [6]

    Bruno and S

    S. Bruno and S. Sabanis. Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradients.Transactions on Machine Learning Research, 2025

    work page 2025

  7. [7]

    D. Y. Cao, A. Y. Chen, K. Sridharan, and Y. Wu. On the Robustness of Langevin Dynamics to Score Function Error. arXiv:2603.11319, 2026

    work page Pith review arXiv 2026

  8. [8]

    Hutzenthaler, A

    M. Hutzenthaler, A. Jentzen, and P. E. Kloeden. Strong and weak divergence in finite time of Euler’s method for stochastic differential equations with non-globally Lipschitz continuous coefficients.Proceedings of the Royal Society A, 467:1563–1576, 2011

    work page 2011

Show all 15 references
  1. [9]

    N. B. Khelifa, R. E. Turner, and R. Venkataramanan. Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors. arXiv:2606.06179, 2026

    2026 arXiv

  2. [10]

    Tang and H

    W. Tang and H. Zhao. Contractive Diffusion Probabilistic Models. arXiv:2401.13115, 2024

    2024 arXiv

  3. [11]

    Conforti, A

    G. Conforti, A. Durmus, and M. Gentiloni Silveri. KL Convergence Guarantees for Score Diffusion Models under Minimal Data Assumptions.SIAM Journal on Mathematics of Data Science, 7(1):86–109, 2025

    2025

  4. [12]

    Peebles and S

    W. Peebles and S. Xie. Scalable Diffusion Models with Transformers.Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 26

    2023

  5. [13]

    Saharia, W

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv:2205.11487, 2022

    2022 arXiv

  6. [14]

    Lou and S

    A. Lou and S. Ermon. Reflected Diffusion Models.Proceedings of the 40th International Conference on Machine Learning, PMLR 202:22675–22701, 2023

    2023

  7. [15]

    D. J. Higham, X. Mao, and A. M. Stuart. Strong convergence of Euler-type methods for nonlinear stochastic differential equations.SIAM Journal on Numerical Analysis, 40(3):1041– 1063, 2002. 27

    2002

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.