{"id":"deb4e202-7cab-4597-9619-3bafe1c3726d","arxiv_id":"2502.00885","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SGD with momentum has a provably worse generalization bound than plain SGD under heavy-tailed noise for quadratic losses, according to a comparison that is flawed as written.","lead":"This paper proves theoretical bounds on how much stochastic gradient descent with momentum can overfit when gradient noise has heavy tails. It argues that, for quadratic losses, the provable generalization bound for momentum is always worse than the bound for plain SGD.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 6's stochastic-integral bound inherits an initial-condition factor that vanishes at Y0=0, contradicting distinct stationary laws and leaving Corollary 7 unsupported.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing error: the proof of Theorem 6 carries the deterministic initial-condition factor ||Y0||/n into the stochastic-integral term, although that term is independent of Y0. This makes the upper bound vanish at Y0 = 0, which is impossible because the two stationary measures are distinct whenever the datasets differ. Since Corollary 7 and the 'momentum is harmful' claim are direct corollaries of Theorem 6 and Proposition 8, the central comparative conclusion is unsupported. This is an internal inconsistency, not merely a disagreement with prior consensus. I find no need to manufacture additional objections: the general non-convex stability bound (Theorem 3) and the uniform-in-time discretization result (Theorem 12) are separate contributions and may be valuable, but they do not rescue the quadratic-loss comparison. The appropriate verdict remains REJECT, matching the reader's assessment.","tokens_in":50746,"tokens_out":3191,"duration_ms":34914,"concrete_test":"Recompute Theorem 6 for a scalar quadratic case with d = 1, n = 1, X = 1, hatX = 2, fixed γ > 0, and Y0 = 0. Solve or numerically simulate the two stable Ornstein-Uhlenbeck processes in (16)-(17) and estimate W_p(μ, μhat) for p ∈ [1, α). If the estimated distance is clearly positive while formula (20) evaluates to zero, Theorem 6 is refuted. Separately, re-derive the small-jump bound in (56) using an operator-norm estimate for e^{-A(t-s)} - e^{-hatA(t-s)} that does not contain ||Y0||; if that bound is positive at Y0 = 0, the proof step in (53)-(55) is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison claim rests on Theorem 6, whose proof in Appendix B.1 bounds the stochastic integral in (53) by applying the deterministic estimate (54) — which contains the factor ||Y0||/n — to the integrand (e^{-A(t-s)} - e^{-hatA(t-s)})Σz. That integrand has no dependence on Y0; only the vector difference e^{-At}Y0 - e^{-hatAt}Y0 is controlled by (54). Consequently the displayed bound in (20) is exactly zero whenever Y0 = 0, even though W_p(μ, μhat) is strictly positive because A ≠ hatA gives two different stationary distributions. This is not mere looseness: a valid upper bound cannot vanish for a strictly positive quantity. Replacing (54) with a correct operator-norm estimate for the matrix difference would introduce a factor measuring ||A - hatA|| (hence the data perturbation) rather than ||Y0||, and the resulting bound need not be ordered as in Corollary 7. Since Corollary 7 and Proposition 8 are the only bridge from stability bounds to the paper's headline 'momentum is harmful' conclusion, the quadratic-loss comparison is invalid as proven. The general non-convex bound and the uniform-in-time discretization result may still stand, but they do not support the central claim without a corrected Theorem 6.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies algorithmic stability and generalization bounds for stochastic gradient descent with momentum (SGDm) under heavy-tailed noise. It analyzes the continuous-time limit as a degenerate Lévy-driven SDE, proves a Wasserstein stability bound for a class of potentially non-convex losses (Theorem 3), and derives a generalization error bound (Corollary 4). For quadratic losses, it claims a p-Wasserstein stability estimate (Theorem 6) and, combining it with Proposition 8, concludes that SGDm always has a worse generalization bound than SGD without momentum (Corollary 7). The paper also proves a uniform-in-time discretization error bound (Theorem 12) and derives discrete-time stability and generalization bounds. The central quadratic comparison claim is the main advertised contribution.","tokens_in":51003,"tokens_out":5326,"duration_ms":59898,"significance":"If valid, the claim that momentum provably harms generalization under heavy-tailed noise would be a notable and broadly interesting result for optimization and learning theory. The paper also contains a substantial technical component in the uniform-in-time Wasserstein discretization bound for degenerate stable SDEs (Theorem 12), which appears to be new and of independent interest. The general non-convex stability bound (Theorem 3) is plausible and built on sound external tools. However, the headline negative result for momentum rests on Theorem 6, and the proof of that theorem contains a load-bearing error. The comparison claim is therefore not established by this manuscript.","major_comments":[{"comment":"The stochastic-integral estimate is not supported by the displayed inequality. Equation (54) bounds the norm of e^{-At}Y0 - e^{-\\hat A t}Y0 by a multiple of ||Y0||/n; it does not bound the operator norm ||e^{-A(t-s)} - e^{-\\hat A(t-s)}||. In the stochastic integrals in (55), the integrand is (e^{-A(t-s)} - e^{-\\hat A(t-s)})Σz, so the required estimate is an operator-norm bound applied to the arbitrary jump vector z. The proof instead substitutes the right-hand side of (54) into the integrand, introducing the factor ||Y0||. Setting Y0 = 0 makes the displayed bound in (20) exactly zero, while W_p(μ, μhat) is strictly positive whenever A ≠ \\hat A and the two stationary laws differ. A valid upper bound cannot vanish for a strictly positive quantity, so Theorem 6 is unsound as written.","section":"Appendix B.1, Eq. (53)-(57)"},{"comment":"The stationary bound in (20) depends on the initial condition Y0, but the invariant measures μ and μhat are unique and do not depend on Y0. After passing to t → ∞, the left-hand side W_p(μ, μhat) is independent of Y0, while the right-hand side contains ||Y0||. This is not merely a looseness issue: for Y0 = 0 the right-hand side is zero, contradicting the fact that W_p(μ, μhat) > 0 in the setting of the theorem. Consequently, the comparison in Corollary 7 and the statement that the SGDm bound is always larger than the SGD bound are unsupported. Proposition 8 establishes only the ordering σmin ≤ θmin; without a valid Theorem 6 it cannot carry the generalization comparison. A corrected stochastic-integral estimate would have to be based on an operator-norm bound for e^{-A(t-s)} - e^{-\\hat A(t-s)}, which would introduce a factor depending on the difference of the drift matrices rather than on ||Y0||, and there is no reason that the resulting bound would satisfy the same monotone comparison.","section":"Theorem 6 and Corollary 7"}],"minor_comments":[{"comment":"The dataset name is written as 'CIF AR10' in a few places; it should be 'CIFAR-10'.","section":"Abstract and Section 6"},{"comment":"The word 'erdogicity' should be 'ergodicity'.","section":"Appendix B.1"},{"comment":"The sentence 'On the other hand, Liu et al. (2023), showed that ...' contains a comma splice; it should read 'On the other hand, Liu et al. (2023) showed that ...'.","section":"Section 7"},{"comment":"It would help the reader if the text explicitly stated that (54) is an estimate for the fixed vector Y0, not an operator-norm estimate, since the subsequent application is the source of the error in Theorem 6.","section":"Equation (54)"}],"recommendation":"reject","confidential_remarks":"The central claimed contribution — that momentum is always worse than SGD for quadratic losses under heavy-tailed noise — rests on Theorem 6, whose proof is invalid. The error is not a presentation issue: the stochastic integral is independent of the initial condition Y0, yet the proof controls it with an estimate that carries ||Y0||. The resulting bound vanishes for Y0 = 0, which is impossible for distinct stationary measures. Since Corollary 7 is the only bridge to the headline conclusion, the comparison claim is unsupported. The uniform-in-time discretization result and the general non-convex bound may still have value, but they do not justify the advertised conclusion. I recommend rejection rather than major revision because the quadratic comparison needs a new proof technique and the conclusion may not survive a correct estimate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that the paper contains real technical work, but its central claim about momentum is unsupported as proven. The quadratic-loss comparison (Theorem 6, Corollary 7, Proposition 8) has a load-bearing flaw: the bound in (20) multiplies everything by ||Y0||, including the stochastic integral term, which is independent of Y0. Setting Y0=0 makes the right-hand side vanish, but A and A-hat differ, giving distinct stationary laws, so the Wasserstein distance is strictly positive. The proof applies the deterministic estimate (54) to the stochastic integrand, which is not justified. This is not looseness; a valid upper bound cannot be zero for a positive quantity. Since Corollary 7 is the only bridge to the 'momentum is harmful' conclusion, that conclusion collapses.\n\nWhat stands: Theorem 3 gives a quantitative W1 stability bound for a general non-convex class under heavy-tailed noise, extending the Raj et al. framework to the underdamped (momentum) SDE. The constants are explicit, and the Lindeberg/semigroup decomposition proof is coherent as far as I checked. The uniform-in-time discretization result for stable SDEs with degenerate noise (Theorem 12) appears new and is a genuine contribution to the SDE approximation literature; the step-size condition and ergodicity arguments are substantial. The experiments are honest, and the limitation statement is candid.\n\nThe soft spots: aside from Theorem 6, the paper compares upper bounds, not actual generalization errors; even a corrected bound might not imply that momentum is worse in reality. The general non-convex bound is too implicit to compare. The authors acknowledge they cannot explain the phenomenon, which is fine but underscores how fragile the headline is.\n\nWho this is for: researchers in heavy-tailed ML theory, especially those working on stability of optimization algorithms and SDE discretization. The discretization result deserves attention. But the paper needs major revision: either fix Theorem 6 with a correct operator-norm estimate (which would likely involve ||A - A-hat|| instead of ||Y0|| and may not rank the bounds) or reframe the contribution without the strong momentum-is-harmful claim. I would send this to peer review — a serious referee is warranted — but with the expectation of heavy revision. The general non-convex and discretization parts deserve a fair shot.","headline":"Solid technical work on general non-convex stability and degenerate-noise discretization, but the headline claim about momentum harming generalization rests on a flawed bound in Theorem 6 that vanishes for zero initial condition.","tokens_in":51528,"tokens_out":2670,"would_cite":false,"duration_ms":27421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60G51","60G52","60H10","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"For quadratic losses, adding momentum provably makes heavy-tailed SGD's generalization bound worse.","keywords":["algorithmic stability","generalization","stochastic gradient descent","momentum","heavy tails","Wasserstein distance","Lévy-driven SDE","degenerate noise"],"falsifier":"A direct computation of the p-th moment of the difference of the two stochastic integrals in the linear quadratic case, with initial condition $Y_0=0$ and two datasets differing in one point, would settle the bound: if that moment is positive and does not scale as $|\\sigma_1+\\sigma_2|\\,\\|Y_0\\|/n$, the displayed Wasserstein bound cannot be correct.","tokens_in":50544,"feed_emoji":"📉","tokens_out":9512,"duration_ms":91112,"temperature":0.7,"pith_summary":"This paper extends algorithmic-stability analysis of heavy-tailed gradient noise from plain SGD to SGD with momentum (SGDm), modeling the algorithm's continuous-time limit as a Lévy-driven underdamped stochastic differential equation. Its central quantitative claim is that for quadratic losses the p-Wasserstein stability bound—and hence the generalization bound—for SGDm is always at least as large as the corresponding bound for SGD without momentum. The mechanism is algebraic: the smallest singular value of the momentum dynamics' drift matrix is never larger than the smallest singular value of the data-covariance matrix that controls plain SGD. The paper also proves a uniform-in-time discretization bound showing that the discrete recursion inherits the SDE's generalization behavior for suitably small step sizes, and reports experiments on quadratic problems and neural networks that match the predicted ordering. A careful reader should care because momentum is a default optimizer choice; if heavy-tailed noise is present, this analysis identifies a regime where the default choice carries a provable generalization penalty.","feed_headline":"Momentum provably worsens heavy-tailed SGD's generalization bound","feed_subtitle":"For quadratic losses the momentum bound is never tighter than plain SGD's—heavy tails and momentum together can backfire.","key_machinery":"The load-bearing object is the drift matrix $A=\\begin{pmatrix}0&-I\\\\ X^\\top X/n&\\gamma I\\end{pmatrix}$ of the underdamped linear SDE that is the continuous-time limit of SGDm; its block structure couples position and velocity and makes the noise term degenerate, since only the velocity component is driven by the Lévy process. The comparison with SGD reduces to comparing $σ_{\\min}$, the smaller of the smallest singular values of the two momentum drift matrices, with $θ_{\\min}$, the smaller of the smallest singular values of the two empirical covariance matrices $X^\\top X/n$ and $\\hat X^\\top \\hat X/n$. Proposition 8 proves $σ_{\\min}\\le\\u03b8_{\\min}$ by diagonalizing $AA^\\top$ into $2\\times2$ blocks with eigenvalues $μ_{i,\\pm}$ and showing $μ_{i,-}\\le\\kappa_i^2$ for every positive friction parameter $γ$. The Wasserstein bounds themselves are carried by the explicit solution of the linear SDEs together with a small-jump/large-jump split of the stochastic integral, and the general non-quadratic stability bound is carried by a Lyapunov-function contraction estimate plus a Lindeberg-exchange decomposition of the semigroup difference.","core_discovery":"On the paper's own terms, the central discovery is a comparison: for quadratic losses, the algorithmic-stability upper bound of SGD with momentum under heavy-tailed noise is always at least as large as the corresponding upper bound for SGD without momentum. The argument writes both continuous-time limits as linear SDEs driven by the same rotationally symmetric $α$-stable Lévy process; the momentum dynamics have a drift matrix $A$ whose smallest singular value $σ_{\\min}$ is shown in Proposition 8 to be no larger than $θ_{\\min}$, the smallest singular value of the empirical covariance $X^\\top X/n$. Since every factor multiplying these singular values in the Wasserstein bound is monotone decreasing, Corollary 7 extracts the ordering of the generalization bounds. The discrete-time extension, Theorem 12 with Corollaries 13–14, shows that for sufficiently small step sizes the recursion inherits the continuous-time bound up to a uniform $O(\\eta^{1/\\alpha})$ discretization error. The synthetic and neural-network experiments are presented as consistent with the predicted ordering.","pith_inferences":["The comparison is a statement about upper bounds, not about realized generalization error; a direct finite-time comparison of the stochastic integrals would show whether actual generalization gaps follow the same ordering as the bounds.","If the degenerate velocity noise is the mechanism behind the loss, then other momentum variants whose continuous-time limits share the same block-drift structure may face the same comparison, which the paper does not address.","The bounds scale with the initial condition $‖Y_0\\|$, so the momentum penalty should be most visible from nonzero starts; a warm-started run may shrink the apparent penalty, a prediction the paper leaves untested.","For locally quadratic non-convex losses, one could replace $X^\\top X$ with the Hessian at a minimum and test whether the same $σ_{\\min}\\le\\u03b8_{\\min}$ ordering predicts momentum's generalization gap on neural networks beyond the reported experiments."],"forward_implications":["For quadratic losses, no choice of the momentum/friction parameter $γ>0$ can make the SGDm generalization bound tighter than plain SGD's; at best smaller $γ$ shrinks the gap.","The tail index $α$ enters the bound through factors like $(2-α)^{-1/2}$ and $(\\alpha-1)^{-1}$, so the looser momentum bound is amplified as the noise gets heavier.","For general non-convex losses satisfying the dissipativity and pseudo-Lipschitz conditions, the continuous-time stability bound is linear in the dataset distance $ρ(X_n,\\hat X_n)$ up to explicit constants, giving an $O(D/n)$ generalization bound under bounded data.","With step size $η<\\bar\\eta$, the discrete-time SGDm recursion has a unique invariant measure, and its invariant measure is within $O(\\eta^{1/\\alpha})$ in Wasserstein distance of the SDE's, so the discrete dynamics inherit the generalization comparison for small step sizes.","The synthetic quadratic and neural-network experiments show the median generalization gap of SGDm is consistently at least as large as SGD's, and it grows with $γ$."],"supporting_citations":[{"why":"Defines uniform algorithmic stability and provides the theorem that converts stability into a generalization bound, the formal target of the paper.","marker":"Hardt et al. (2016)"},{"why":"Gives the quadratic-loss heavy-tailed SGD setting and the singular-value parameterization adopted for the no-momentum comparison.","marker":"Raj et al. (2023a)"},{"why":"Establishes heavy-tailed SGD stability bounds for general losses and supplies the Lindeberg-exchange proof template extended here to momentum.","marker":"Raj et al. (2023b)"},{"why":"Provides the Wasserstein contraction and Lyapunov-function estimates for the underdamped Lévy-driven SDE that make the stationary stability bounds finite.","marker":"Bao and Wang (2022)"},{"why":"Supplies the Markov-process probability approximation framework used to compare the two semigroups in one-step increments.","marker":"Chen et al. (2023c)"},{"why":"Introduces the recursion for heavy-tailed SGD with momentum whose continuous-time limit is the Lévy-driven underdamped SDE analyzed throughout.","marker":"Simsekli et al. (2020)"},{"why":"Provides the maximal inequality used to control the large-jump part of the stochastic integral in the quadratic-loss Wasserstein bound.","marker":"Zhu et al. (2019)"},{"why":"Gives the Euler–Maruyama invariant-measure approximation for stable SDEs that the paper's uniform-in-time discretization bound extends to degenerate noise.","marker":"Chen et al. (2023a)"}],"fun_headline_variants":["Momentum hurts SGD generalization under heavy-tailed noise","Heavy tails + momentum: worse generalization for SGD","SGD with momentum: weaker bound than plain under heavy tails","Momentum weakens heavy-tailed SGD's stability bound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random-noise part of the difference between the two processes obeys the same bound, with the same initial-condition factor, as the deterministic part, so that the displayed stability bound vanishes when the initial condition is zero.","fun_headline_variants_meta":{"raw":{"variants":["Momentum hurts SGD generalization under heavy-tailed noise","Heavy tails + momentum: worse generalization for SGD","SGD with momentum: weaker bound than plain under heavy tails","Momentum weakens heavy-tailed SGD's stability bound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1446,"prompt_tokens":968,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":584,"tokens_out":478,"duration_ms":5241,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:22:33.875370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct computation of the p-th moment of the difference of the two stochastic integrals in the linear quadratic case, with initial condition $Y_0=0$ and two datasets differing in one point, would settle the bound: if that moment is positive and does not scale as $|\\sigma_1+\\sigma_2|\\,\\|Y_0\\|/n$, the displayed Wasserstein bound cannot be correct.","supporting_citations":[],"review_version":1}