REVIEW 3 major objections 4 minor 18 references
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Large learning rates prevent memorization in denoising score matching by keeping the learned score away from the exact empirical score.
desk verdict A genuinely new conditional theorem linking large learning rates to non-memorization in denoising score matching, but the paper's headline conclusion reaches beyond what the theorem actually proves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs through three coupled objects. First, writing the empirical optimal score $s^*$ through a soft-assignment random variable $W(y)$ makes its small-noise behavior precise: the variance of $W$ controls $s^{*\prime}$, and this yields a lower bound on the weighted total variation $\mathrm{TV}_\pi^{(1)}(s^*)$. Second, for two-layer ReLU networks the largest eigenvalue of the neural tangent kernel is lower bounded by $(2/m)\mathrm{TV}_\pi^{(1)}(s_\theta)$, tying the network's representational complexity to the irregularity of the score it implements. Third, the linear-stability condition for SGD turns a large learning rate into an upper bound on the Hessian at a stable minimum, $\lambda_{\max}(\nabla^2 R_n(\theta^*)) \le 2/(m\eta)$, which in turn caps $\mathrm{TV}_\pi^{(1)}(s_{\theta^*})$ from above. Matching these lower and upper bounds on total variation forces a strictly positive excess risk whenever the learning rate is large.
What would settle it
Train a two-layer ReLU network on one-dimensional data with $\sigma$ small and $\eta$ above the theorem's threshold, then measure the excess risk $R_n(\theta_{\mathrm{final}}) - R_n(s^*)$ by Monte Carlo. If, for some data configuration satisfying the minimum-spacing condition, the excess risk can be made smaller than $\frac{\pi n^5 \mu^3 \Delta^3}{2^{36} e^{1/2} A^4 \sigma^4}$ while the final iterate is linearly stable, Theorem 8 is false.
Extended reading notes
Core claim
For one-dimensional data, let $R_n$ be the denoising score matching risk over two-layer ReLU networks with bounded outer weights, and let $s^*$ be the empirical optimal score. The paper proves (Theorem 8) that if $\theta^*$ is a linearly stable local minimum of $R_n$ and the noise level $\sigma$ is small enough, then whenever $\eta > \frac{2^{12}\sigma^2}{\mu n^2 \Delta}$, the excess risk satisfies $R_n(\theta^*) - R_n(s^*) > \frac{\pi n^5 \mu^3 \Delta^3}{2^{36} e^{1/2} A^4 \sigma^4}$. Since the excess risk equals a weighted $L^2$ distance between $s_{\theta^*}$ and $s^*$, the learned score cannot be arbitrarily close to the empirical optimal score, and therefore full memorization of the training sample is avoided. The discovery is that the large learning rate itself acts as an implicit regularizer, and the proof shows this is not an artifact of stochastic gradient noise, because the same argument works for gradient descent.
Load-bearing premise
The whole argument assumes that SGD actually settles at a twice-differentiable local minimum $\theta^*$ that is linearly stable in the sense of the linearized update, and that $\theta^*$ lies strictly inside the parameter constraint set; this is plausible for the smoothed score-matching objective but is not proven.
Editorial extensions
If this is right
- If Theorem 8 is correct, a network trained with learning rate above the stated threshold cannot converge to the empirical optimal score, so it cannot reproduce the training data exactly.
- In the diffusion setting the bound applies to small $\sigma$, which corresponds to the final denoising steps where memorization is known to be most dangerous.
- The same proof applies to gradient descent on the population risk, so the effect is a bias of the step size itself, not a byproduct of stochastic gradient noise.
- The upper bound on total variation implies that very large learning rates push the score toward linear functions, and the learned generative distribution then approaches a Gaussian approximation of the data.
- Empirically, the same qualitative pattern appears in dimensions 2 and 10 and in a full diffusion generation loop, suggesting the one-dimensional analysis captures a general mechanism.
Reading between the lines
- An unproved but testable corollary is that the critical learning rate should grow as the minimum data spacing shrinks, so in higher dimension, with fixed sample size, memorization avoidance should become easier; the paper's dimension experiments are consistent with this, but no multivariate theorem is given.
- The total-variation quantity at the center of the proof could be used as an explicit penalty: adding $\mathrm{TV}_\pi^{(1)}(s_\theta)$ to the risk would likely reproduce the memorization-prevention effect without requiring a large step size.
- The bound does not say that larger learning rates improve generation quality; it only bounds how close the score can come to the memorizing solution, so any benefit to fidelity must come from a separate mechanism or from a carefully scheduled learning rate.
- The proof carries over to any learning-rate schedule bounded below by a positive constant, which suggests that annealing the learning rate could keep the regularization active while still allowing fine-grained fitting at large noise levels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies denoising score matching in the one-dimensional setting with two-layer ReLU networks and asks why practical diffusion models do not fully memorize the training set. It first shows that the empirical optimal score s⋆ is highly irregular for small noise: its derivative has large weighted total variation (Theorem 4), while its risk is exponentially small (Corollary 3). It then proves, conditional on θ⋆ being a linearly stable local minimum of the semi-empirical risk in the sense of Mulayoff et al., that if the learning rate η exceeds a threshold of order σ², the excess risk Rn(θ⋆) − Rn(s⋆) is bounded below by a positive quantity, so the learned score cannot be arbitrarily close to s⋆ (Theorem 8). The paper concludes that large learning rates act as an implicit regularizer against memorization and supports this with experiments in one and higher dimensions, measuring both excess risk and memorization via MMD.
Significance. If the conditional theorem is accepted, this is a novel theoretical mechanism for implicit regularization in denoising score matching: it connects the learning rate, the flatness of stable minima, and the irregularity of the empirical optimal score. The proofs are detailed, the main theorem is derived from independently established stability and approximation results, and no parameters are fitted to obtain the conclusion. The experiments go beyond the one-dimensional theory and directly measure the learning-rate dependence of memorization. The main reservation is that the advertised practical claim is inferred from a theorem whose key stability assumption is not proven for SGD on this objective.
major comments (3)
- [§6, Theorem 8; §2, p. 3] The paper's central conclusion is obtained by applying Theorem 8 to minima reached by SGD. However, Theorem 8 is conditional on θ⋆ being a linearly stable local minimum of Rn, and the manuscript does not prove that SGD with large η actually converges to such a point. Lemma 5 only proves that Rn is twice differentiable everywhere; it says nothing about convergence of the SGD iterates, stationarity of the limit, or stability of the linearized dynamics in (7). The sentence in Section 2 claiming that the noise in the objective 'guarantees the necessary assumption that SGD reaches a twice-differentiable local minimum' therefore overstates what is established. I ask the authors to either prove the required convergence and stability property under explicit assumptions, or to state the result as conditional and adjust the abstract and Theorem 1 accordingly; the experiments should also verify that the final parameters satisfy the stationarity and interior-point conditions used in the proof.
- [§6, Theorem 8; §3] Theorem 8 gives a lower bound on the excess risk Rn(θ⋆) − Rn(s⋆), but no result in the paper quantifies how this lower bound translates into reduced memorization of the generated distribution. Li et al. (2024a) show that the exact score s⋆ memorizes, whereas the manuscript does not prove continuity of the sampling distribution with respect to the score under the weighted L2 metric used in the excess-risk identity. Closeness to s⋆ could in principle still produce near-memorization, so the 'thereby mitigating memorization' conclusion is an extrapolation. Please either prove a quantitative statement for the one-dimensional setting or make explicit that the memorization conclusion is a heuristic supported by the experiments in Section 7.
- [§4, A = Cn/σ^6; §6, Theorem 8] The quantitative strength of Theorem 8 depends critically on the constraint radius A. The authors choose A = Cn/σ^6 so that s⋆ is approximable; substituting this choice into the lower bound of Theorem 8 gives a bound of order σ^20, and the learning-rate threshold is of order σ^2. Both quantities vanish as σ → 0. This does not invalidate the pointwise statement for fixed σ, but the 'small-noise' asymptotic is not uniform, and the separation from s⋆ is much weaker than the informal statement in Theorem 1 might suggest. Please state this scaling explicitly and clarify what remains true as σ → 0.
minor comments (4)
- [Appendix A, proof of Proposition 10] The matrix Φ(ξ) is said to belong to R^{3m×n}, but the parameter vector θ is in R^{2m} and the displayed gradient ∇θsθ is 2m-dimensional; the dimension should be R^{2m×n} unless the fixed inner weights are being included in the parameter vector, which would contradict Section 4.
- [Appendix D] The phrase 'population risk Rn' is confusing because Rn in (5) is conditional on the training sample; the term 'semi-empirical risk' used earlier in Section 3 is more accurate, and the discussion of GD should use that terminology consistently.
- [§4, Lemma 12] The constant Cn used in the choice A = Cn/σ^6 is not defined before it is used; please define it explicitly in terms of μ and the sample (for instance via the bound in Lemma 12) before invoking it to justify the approximation of s⋆.
- [§7, paragraph 'Learning rate and memorization effect'] There is a typo in 'at thet end of training'; it should read 'at the end of training'.
Circularity Check
No significant circularity: the main result is a conditional theorem built on external stability and approximation results.
full rationale
The derivation chain is not circular. Theorem 4 establishes irregularity of the empirical optimal score s* directly from its closed form (6) and elementary Gaussian estimates; the lower bound on TV(1)_pi(s*) is a genuine analytic result, not an input assumption. The second leg imports the stability-to-Hessian bound lambda_max <= 2/(m eta) from Mulayoff et al. (2021, Lemma 1); this is an external, independently published theorem, and it is not a self-citation of the present authors. Proposition 6 combines that external bound with a new Hessian decomposition to upper-bound TV(1)_pi(s_theta*), while Proposition 7 gives a separate low-risk lower bound; Theorem 8 is the contrapositive combination of these inequalities. No parameter is fitted to a data subset and then renamed as a prediction, and no conclusion is equivalent by construction to its own hypothesis. The only notable weakness is the unproven premise that SGD actually reaches a linearly stable, twice-differentiable local minimum: the text in Section 2 claims the noise regularizing effect 'guarantees' this, but Lemma 5 only proves twice differentiability everywhere and does not prove convergence to such a minimum. That is a correctness or applicability gap, not circularity, because Theorem 8 is explicitly conditional on the stability hypothesis. The paper also relies on external results for the memorization property of s* (Li et al. 2024a) and for ReLU approximation (Bach 2024), both independent of the present authors. No self-citation is load-bearing. The experimental section is supportive rather than part of the proof chain. Overall, the central claim has independent mathematical content and does not reduce to its inputs.
Assumptions & free parameters
assumptions (5)
- domain assumption Linearly stable local minima satisfy lambda_max(Hessian R_n(theta*)) <= 2/(m eta) (Mulayoff et al. 2021, Lemma 1).
- domain assumption The empirical optimal score s* can be approximated by the two-layer ReLU class S with outer weight bound A >= C_n/sigma^6.
- domain assumption If the learned score equals the empirical optimal score, the backward diffusion generates the empirical distribution (full memorization).
- domain assumption The training data is one-dimensional with positive minimum spacing Delta > 0, and the small-noise condition Delta >= 8 sigma/mu holds.
- domain assumption The network's inner weights are fixed at +/-1 and the outer weights lie in a bounded ball; this does not reduce expressivity due to ReLU homogeneity.
Cite this review
Pith. "Pith review of Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization." pith.science (2026). https://pith.science/paper/OXKMP2V2
@misc{pith2026250203435,
author = {Pith},
title = {Pith review of: Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization},
year = {2026},
howpublished = {\url{https://pith.science/paper/OXKMP2V2}},
note = {Machine review of arXiv:2502.03435}
}
read the original abstract
Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising score matching--leads to memorization, where generated samples replicate the training data. Yet, in practice, only a moderate degree of memorization is observed, even without explicit regularization. In this paper, we investigate this phenomenon by uncovering an implicit regularization mechanism driven by large learning rates. Specifically, we show that in the small-noise regime, the empirical optimal score exhibits high irregularity. We then prove that, when trained by stochastic gradient descent with a large enough learning rate, neural networks cannot stably converge to a local minimum with arbitrarily small excess risk. Consequently, the learned score cannot be arbitrarily close to the empirical optimal score, thereby mitigating memorization. To make the analysis tractable, we consider one-dimensional data and two-layer neural networks. Experiments validate the crucial role of the learning rate in preventing memorization, even beyond the one-dimensional setting.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
On the other hand, for any y ∈ R, π(y; µ, σ) ⩽ µ(xn − x1)
2 ∆. On the other hand, for any y ∈ R, π(y; µ, σ) ⩽ µ(xn − x1). Proof Recall that π−(y; µ, σ) = P(µU < y)2E(y − µU |µU < y). Lower bound. Let x = y µ. Then, clearly, y ∈ [µxi, µxi+1] is equivalent to x ∈ [xi, xi+1]. Hence, P(µU < y)2 = P(U < x)2 = i2 n2 and E[y − µU |µU < y] = µE[x − U |U < x] = µ x − 1 i iX i′=1 xi′ . Therefore, π−(µx; µ, σ) = µi2 n2 x −...
work page 1941
-
[3]
URL https://scoste.fr/posts/diffusion/ #denoising_score_matching. Accessed on 2025-01-20. Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informa...
work page 2025
-
[5]
A good score does not lead to a good generative model
Sixu Li, Shi Chen, and Qin Li. A good score does not lead to a good generative model. arXiv:2401.04856, 2024a. Xiang Li, Yixiang Dai, and Qing Qu. Understanding generalizability of diffusion models requires rethinking the hidden Gaussian structure. InAdvances in Neural Information Processing Systems, volume
-
[8]
Hierarchical text- conditional image generation with clip latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv:2204.06125,
-
[13]
17 WU MARION BIAU BOYER TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K. Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling,
work page 2023
-
[14]
On copyright risks of text-to-image diffusion models
Yang Zhang, Teoh Tze Tzun, Lim Wei Hern, Haonan Wang, and Kenji Kawaguchi. On copyright risks of text-to-image diffusion models. arXiv:2311.12803,
-
[16]
2 ∆. Upper bound. Again, let x = y µ. Observe that, for y ∈ [µx1, µxn], we have x ∈ [x1, xn]. Therefore, E[y − µU |µU < y] = µE[x − U |U < x] ⩽ µ(x1 − xn). So, π−(y; µ, σ) ⩽ µ(xn − x1). We may also upper bound π+(y; µ, σ) with the same value. By taking the expectation, we have π(y; µ, σ) ⩽ µ(xn − x1). 20 LARGE LEARNING RATES IN DENOISING SCORE MATCHING PR...
work page 2021
-
[17]
If, for all y ∈ [µxi − µ∆ 2 , µxi], one has sθ⋆(y) ⩽ 0, then nRn(θ⋆) ⩾ EY ∼N (µxi,σ2) (sθ⋆(Y ) − 1 σ2 (µxi − Y ))2 ⩾ 1√ 2πσ2 Z µxi µxi− µ∆ 2 sθ⋆(y) − 1 σ2 (µxi − y) 2e− (y−µxi)2 2σ2 dy ⩾ 1√ 2πσ2 Z µxi µxi− µ∆ 2 1 σ4 (µxi − y)2e− (y−µxi)2 2σ2 dy = 1 σ4 √ 2πσ2 Z 0 − µ∆ 2 y2e− y2 2σ2 dy = 1 σ2 √ 2πσ2 µ∆ 2 e− µ2∆2 8σ2 + Z 0 − µ∆ 2 e− y2 2σ2 dy ⩾ 1 σ2 √ 2πσ2 µ...
work page 1941
Show all 18 references
-
[18]
Similarly, Z ∞ 2µ(xn)+ |s⋆′′(y; µ, σ)|dy ⩽ 4µ2(n − 1)(xn − x1)3 σ4∆ . Putting everything together, we have Z R |s⋆′′(y; µ, σ)|dy ⩽ 4µ4(xn − x1)3((xn)+ − (x1)−) σ6 + 8µ2(n − 1)(xn − x1)3 σ4∆ ⩽ 4µ2(xn − x1)3 σ6 µ2((xn)+ − (x1)−) + 2(n − 1)σ2 ∆ , which is the desired result. The ...
1941
-
[1941]
On memorization in diffusion models
Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv:2310.02664,
-
[2015]
Stable minima cannot overfit in univariate ReLU networks: Generalization by large step sizes.arXiv:2406.06838,
Dan Qiao, Kaiqi Zhang, Esha Singh, Daniel Soudry, and Yu-Xiang Wang. Stable minima cannot overfit in univariate ReLU networks: Generalization by large step sizes.arXiv:2406.06838,
-
[2016]
Venkatakrishnan, Charles A
Singanallur V . Venkatakrishnan, Charles A. Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pages 945–948. IEEE,
2013
-
[2018]
Denoising: A powerful building-block for imaging, inverse problems, and machine learning
15 WU MARION BIAU BOYER Peyman Milanfar and Mauricio Delbracio. Denoising: A powerful building-block for imaging, inverse problems, and machine learning. arXiv:2409.06219,
-
[2019]
Teodoro, Jos´e M
Afonso M. Teodoro, Jos´e M. Bioucas-Dias, and M ´ario A.T. Figueiredo. Image restoration and re- construction using variable splitting and class-adapted image priors. In 2016 IEEE International Conference on Image Processing, pages 3518–3522. IEEE,
2016
-
[2021]
On the generalization of diffusion model
Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model. arXiv:2305.14712,
-
[2022]
Trainability and accuracy of artificial neural networks: An interacting particle system approach
Grant Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics , 75:1889–1935,
1935
-
[2023]
SegDiff: Image segmentation with diffusion probabilistic models
13 WU MARION BIAU BOYER Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. SegDiff: Image segmentation with diffusion probabilistic models. arXiv:2112.00390,
-
[2024]
Memorization and regularization in generative diffusion models
Ricardo Baptista, Agnimitra Dasgupta, Nikola B Kovachki, Assad Oberai, and Andrew M Stuart. Memorization and regularization in generative diffusion models. arXiv preprint arXiv:2501.15785,
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.