{"id":"fd66e22b-93f6-4976-b8f8-e802c0159688","arxiv_id":"2607.08104","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Vanilla SGD with momentum converges in expectation under p-moment noise for Hölder-smooth f, at rates inferior to clipped/normalized SGD, with the key condition ν+1≤p.","lead":"Vanilla SGD with momentum converges under heavy-tailed gradient noise for strongly convex, convex and nonconvex Hölder-smooth objectives, without any clipping or normalization. The rates are strictly slower than those of clipped or normalized variants, establishing a clean baseline for how much gradient control actually buys.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The paper’s central contribution is a clean, self-contained expectation analysis showing that vanilla SGD with momentum still converges (sub-optimally) under heavy-tailed noise once the Hölder–moment compatibility condition ν+1≤p is imposed. The proofs are fully written, the classical special cases are recovered, the rates are correctly compared with the clipped/normalized literature, and the critical condition is both theoretically necessary (via Lemma 2.4) and empirically confirmed. The bounded-iterate assumption required for the convex analyses is the only technical caveat; it is standard, clearly stated, and does not affect the nonconvex or strongly-convex statements that form the bulk of the strongest claim. Because no load-bearing flaw was located that would overturn the claim, the reader’s ACCEPT / high-confidence verdict stands.","tokens_in":35512,"tokens_out":599,"duration_ms":5292,"concrete_test":"Independently re-derive the nonconvex momentum bound of Theorem 3.6 from Lemmas B.2–B.3 and the Lyapunov Φ_t without invoking the diameter argument of Lemma B.1 (which is used only for the strongly-convex case); confirm that the coefficient of E[∥∇f(x_t)∥²] remains ≥η/4 under the stated step-size restriction η≤min{(1/(4L))^{1/ν}, ((1-β)/β)(1/(4ν L²))^{1/(2ν)}} and that the final rate O(1/(ηT)+η^ν) is recovered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (Assumption 3.1 bounded iterates for the convex/momentum-convex cases, plus the diameter consequence of strong convexity + Hölder for ν<1) is real but standard, explicitly flagged by the authors, and does not undermine the strongest claim. That claim is the first expectation analysis of vanilla momentum SGD (Algorithm 1) under heavy-tailed noise (Assumptions 2.1–2.2) for all three function classes whenever ν+1≤p, with the nonconvex rate O(T^{-(p-1)/2p}) when p is known (Thm 3.6). The proofs (Appendix B) close under the stated conditions via the Hölder descent lemma, von Bahr–Esseen, and the Lyapunov/auxiliary-sequence arguments; the rates correctly recover the classical p=2, ν=1 special cases and are shown to be suboptimal relative to clipped/normalized methods. The synthetic experiments (Figs. 1–2) and public code further corroborate the necessity of ν+1≤p. No internal inconsistency or hidden gap that would falsify the central claim was found.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies vanilla SGD and vanilla SGD with momentum (Algorithm 1, no clipping or normalization) under heavy-tailed stochastic noise (Assumption 2.2: only p-th moments finite for p in (1,2]) and Hölder-continuous gradients (Assumption 2.1 with parameter ν in (0,1]). It supplies the first expectation convergence guarantees for the momentum method on strongly convex (Theorem 3.2), convex (Theorem 3.4) and nonconvex (Theorem 3.6) objectives whenever the key compatibility condition ν+1 ≤ p holds, together with refined rates for plain SGD (Theorems 3.1, 3.3, 3.5). The obtained rates recover the classical p=2, ν=1 special cases and are shown to be strictly slower than the optimal rates of clipped or normalized variants; synthetic heat-maps and trajectories (Figures 1–2) and a Transformer-XL experiment corroborate the necessity of ν+1 ≤ p and the practical convergence of the vanilla methods.","tokens_in":35752,"tokens_out":917,"duration_ms":18437,"significance":"If the claims hold, the work supplies a clean theoretical baseline that was previously missing: vanilla momentum SGD converges in expectation under heavy tails without any gradient control, albeit at suboptimal rates. The Hölder framework plus the elementary moment lemmas (2.1–2.4, 3.1) cleanly extend classical Lyapunov/auxiliary-sequence arguments, recover known rates as special cases, and identify the sharp structural condition ν+1 ≤ p. Full appendix proofs, public code, and controlled synthetic experiments that falsify the condition when violated are genuine strengths that make the contribution usable as a reference point for future algorithmic improvements.","major_comments":[{"comment":"Assumption 3.1 (bounded iterates of both x_t and the auxiliary sequence z_t) is required to close the convex and momentum-convex arguments (Theorems 3.3–3.4 and the associated lemmas in Appendix B.4–B.5). While the authors correctly flag it as standard, the assumption is not derived from the other hypotheses and therefore restricts the theorems to domains that are a priori bounded; a short discussion of when the assumption can be removed (e.g., by projection or by growth conditions) would strengthen the claim that the analysis is “comprehensive.”","section":null},{"comment":"Lemma B.1 (used for the strongly-convex momentum result, Theorem 3.2) relies on the diameter bound that follows from simultaneous µ-strong convexity and Hölder continuity when ν < 1. The paper notes this fact, yet the statement of Theorem 3.2 itself does not make the implicit restriction of the domain explicit; readers may therefore overestimate the scope of the linear-rate claim for ν < 1.","section":null}],"minor_comments":[{"comment":"Table 1: the column “w. Mom.” is slightly ambiguous for the rows that already include momentum; a footnote clarifying that “✓” means the analysis covers the momentum case would improve readability.","section":null},{"comment":"Figure 1 caption: the phrase “Below the red dotted line, the theoretical condition ν+1 ≤ α is satisfied” is correct, but the dotted line itself is not drawn on the heat-maps; adding it (or a shaded triangle) would make the visual claim immediate.","section":null},{"comment":"Page 4, display of E_0: the constant is used repeatedly; defining it once in the main text (as done) is good, but a short parenthetical reminder of its dependence on σ, b, p, ν would help readers who jump between theorems.","section":null},{"comment":"Appendix C: the additional vanilla-SGD heat-maps and trajectories are useful; a one-sentence cross-reference in Section 4 would make them easier to locate.","section":null}],"recommendation":"accept","confidential_remarks":"The manuscript is technically solid and fills a genuine gap. The only soft spot is the standard but strong bounded-iterate assumption for the convex cases; it does not invalidate the central nonconvex-momentum claim (Theorem 3.6). I see no novelty or citation issues that would require editorial intervention. Suitable for acceptance essentially as is."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This paper finally gives the missing baseline: vanilla SGD with momentum (no clip, no norm) converges in expectation under Hölder gradients and p-moments for strongly convex, convex, and nonconvex objectives, provided ν+1 ≤ p. Theorems 3.2, 3.4 and 3.6 are the new pieces; the nonconvex rate O(T^{-(p-1)/2p}) when p is known is the cleanest statement. They also strip the bounded-gradient assumption from the recent Fatkhullin et al. vanilla-SGD results, which is a genuine technical improvement.\n\nThe math is careful. The Hölder descent lemma, von Bahr–Esseen, and the Lyapunov/auxiliary-sequence arguments close under the stated conditions and recover the classical p=2, ν=1 rates as special cases. The authors are honest that their rates sit strictly below the optimal clipped/normalized rates; that comparison is useful rather than defensive. Synthetic heat-maps and trajectories (Figs. 1–2) plus public code make the necessity of ν+1 ≤ p concrete, and the short Transformer-XL run is a sensible sanity check.\n\nSoft spots are real but standard and flagged. Assumption 3.1 (bounded iterates and the auxiliary sequence) is needed for the convex analyses; the diameter consequence of strong convexity + Hölder for ν<1 is used in the strongly-convex momentum proof. Neither is hidden, and neither falsifies the central claim. The paper is theory-first; the real-data experiment is only confirmatory.\n\nThis is for people who work on heavy-tailed SGD theory and need a clean reference point. The proofs are fully written out, the citation pattern is fair, and the contribution is self-contained. I would send it to referees without hesitation; it is the natural baseline the subfield has been missing.","headline":"First clean expectation rates for momentum SGD under p-moments without clipping or normalization; solid baseline, rates correctly suboptimal.","tokens_in":36339,"tokens_out":467,"would_cite":true,"duration_ms":6015,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Vanilla SGD with momentum converges under heavy-tailed noise without clipping or normalization, but at rates strictly slower than clipped or normalized variants.","keywords":["stochastic gradient descent","momentum","heavy-tailed noise","Hölder smoothness","convergence rates","nonconvex optimization","bounded moments"],"falsifier":"Run the same synthetic Hölder objectives with ν+1 > p (for example ν=1 and p=1.5) and check whether the final objective or gradient norm remains bounded after a few thousand steps; systematic divergence would falsify the claimed necessity of the moment condition.","tokens_in":36412,"feed_emoji":"📉","tokens_out":934,"duration_ms":22738,"temperature":0.7,"pith_summary":"This paper asks whether plain stochastic gradient descent, especially the momentum version used everywhere in practice, still converges when gradient noise has only finite p-th moments for p between 1 and 2 (heavy tails) and no artificial clipping or normalization is applied. By replacing classical L-smoothness with Hölder continuous gradients of order ν, the authors prove the first expectation-convergence guarantees for vanilla SGD with momentum on strongly convex, convex, and nonconvex objectives, provided the smoothness and tail indices satisfy ν+1 ≤ p. The obtained rates recover the classical light-tailed rates when ν=1 and p=2, yet are strictly inferior to the optimal rates known for clipped or normalized methods. Synthetic experiments confirm that the condition ν+1 ≤ p is essentially necessary for stable behavior, showing that classical smoothness is incompatible with strictly heavy-tailed noise.","feed_headline":"Vanilla momentum SGD still converges under heavy tails","feed_subtitle":"But its rates lag behind clipped or normalized methods; Hölder smoothness is the key fix.","key_machinery":"The Hölder descent lemma together with a moment-comparison inequality that converts p-moments of the stochastic gradient into (ν+1)-moments precisely when ν+1 ≤ p; this bound, combined with an auxiliary momentum sequence and a Lyapunov function, closes the analysis for all three convexity regimes.","core_discovery":"Vanilla SGD with momentum (the recursion that mixes past mini-batch gradients with a fixed momentum coefficient and takes an ordinary step) converges in expectation for Hölder-smooth objectives under only a bounded p-th moment assumption on the noise, without any gradient clipping or normalization; the nonconvex rate is O(T^{-(p-1)/2p}) when p is known and the step-size is tuned accordingly, and the same framework yields matching results for the pure SGD case under weaker assumptions than earlier work.","pith_inferences":["Because the paper already shows that classical L-smoothness is incompatible with p<2, training pipelines that rely on heavy-tailed gradient noise (transformers, RL) may be silently operating outside the regime where standard convergence proofs apply.","If the bounded-iterate assumption can be removed by a more refined Lyapunov argument, the convex rates would become fully global and the theory would cover unconstrained nonconvex deep learning more convincingly.","The empirical necessity of ν+1 ≤ p on synthetic problems suggests a diagnostic: estimate the Hölder exponent of the loss landscape and the empirical tail index of the gradients; if the inequality fails, one should switch to clipping or normalization."],"forward_implications":["Practitioners can keep using ordinary momentum SGD under heavy-tailed noise and still obtain a theoretical guarantee, albeit a suboptimal one.","Any future algorithm that claims improvement over vanilla momentum under heavy tails must beat the rates O(T^{-(p-1)/2p}) (nonconvex) or O(T^{-(p-1)}) (strongly convex).","Hölder smoothness (or (L0,L1)-smoothness) should replace classical L-smoothness as the default analytic setting whenever noise tails are known to be heavier than Gaussian.","The same technical toolkit extends immediately to other momentum-based methods such as Adam or Muon once their update rules are rewritten with an auxiliary sequence."],"fun_headline_variants":["Vanilla momentum SGD converges under heavy tails without clipping","Momentum SGD survives heavy-tailed noise with no gradient controls","First full rates: vanilla momentum SGD under heavy tails only","Heavy-tailed noise: momentum SGD converges but lags clipped rates","Vanilla SGD plus momentum handles heavy tails, rates suboptimal"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The analysis for convex problems (and the momentum case under strong convexity) needs the iterates and an auxiliary sequence to stay inside a fixed ball; without that bound the descent inequalities do not close.","fun_headline_variants_meta":{"raw":{"variants":["Vanilla momentum SGD converges under heavy tails without clipping","Momentum SGD survives heavy-tailed noise with no gradient controls","First full rates: vanilla momentum SGD under heavy tails only","Heavy-tailed noise: momentum SGD converges but lags clipped rates","Vanilla SGD plus momentum handles heavy tails, rates suboptimal"]},"model":"grok-4.5","effort":"low","cost_usd":0.005718,"raw_usage":{"total_tokens":1423,"prompt_tokens":705,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":57180000,"prompt_tokens_details":{"text_tokens":705,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":637,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":705,"tokens_out":81,"duration_ms":6135,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T12:59:30.355532+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same synthetic Hölder objectives with ν+1 > p (for example ν=1 and p=1.5) and check whether the final objective or gradient norm remains bounded after a few thousand steps; systematic divergence would falsify the claimed necessity of the moment condition.","supporting_citations":[],"review_version":1}