{"id":"ef06abdf-9d1a-4504-9e26-6117b39e4518","arxiv_id":"2607.27383","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Under heavy-tailed noise with bounded p-th moments, vector-form Adam converges to (ρ,ε)-stationary points at rate O(ε^{-(5p/(3p-4)+3/2)}) for p∈(4/3,2]; with known-radius clipping the rate is optimal O(ε^{-(p/(p-1)+3/2)}).","lead":"This paper proves the first convergence rates for the plain vector-form Adam optimizer when stochastic gradients have only a bounded p-th moment ('heavy tails'): Adam reaches approximate stationary points, but at a slower-than-optimal rate. When the domain radius is known and updates are clipped to it, the paper shows Adam matches the optimal heavy-tailed rate.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corollary 6.3's η = D√(1−β1) is algebraically inconsistent with its displayed regret bound; the known-radius optimal rate relies on this step.","rationale":"I read the paper as genuinely analyzing the vector-form FTRL/Adam variant in Algorithm 2, and the heavy-tail conversion framework appears plausible. The most load-bearing problem I found is not the parameter-coupling issue the reader emphasized: Remark 5.3's C(β) is automatically satisfied for any β1≤β2 by taking C(β)=(1−β1)/(1−β2), so 'only β1≤β2' is not an overstatement. The concrete failure is in Corollary 6.3: the stated η makes the simplification to C(R) invalid by a factor of 1−β1 in the first term and (1−β1)² in the second. Since Corollary 6.5's optimal complexity is derived from Corollary 6.3, the proof as written does not establish the optimal known-radius rate. If the intended η is D/√(1−β1), this is a fixable typo; otherwise the result is unsupported. This localized algebra check is more decisive than the scoping concerns about delayed or clipped updates, so I disagree with the reader's choice of weakest assumption while agreeing that the verdict should remain conditional pending correction.","tokens_in":25054,"tokens_out":30483,"duration_ms":358067,"concrete_test":"Recompute the two terms of Eq. (3) with η = D√(1−β1) and with η = D/√(1−β1). Check which choice reproduces the C(R) coefficients printed in Corollary 6.3. Then re-derive the T selection in Corollary 6.5 with the consistent choice: if the printed T is recovered, the issue is a typographical slash; if not, the optimal-rate claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Corollary 6.3 (statement and proof) sets η = D√(1−β1). Substituting this into the regret bound Eq. (3) gives first term D²/(2η(1−β1)) = D/(2(1−β1)^{3/2}) and second term η·2√β2(1−β1)/(β1(1−β2)) = 2D√β2(1−β1)^{3/2}/(β1(1−β2)). Factoring D/√(1−β1), the coefficients would be 1/(2(1−β1)) and 2√β2(1−β1)²/(β1(1−β2)), not the printed 1/2 and 2√β2(1−β1)/(β1(1−β2)). The displayed simplification is valid only with η = D/√(1−β1). With the stated η, the regret bound is O(D/(1−β1)^{3/2}); carrying this into Corollary 6.5, the T-independent contribution becomes C(R)G/√(1−β1), which does not vanish as T grows and is not ≤ ε for the chosen β1. Thus the claimed optimal T in Corollary 6.5 does not follow as written. Since the known-radius optimality result is a main contribution, this is load-bearing and must be corrected to η = D/√(1−β1) (or the C(R) coefficients revised).","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the convergence of vector-form Adam under heavy-tailed stochastic noise, assuming only a bounded p-th central moment (p∈(1,2]). It (i) generalizes the discounted online-to-nonconvex conversion framework to heavy-tailed martingale-difference noise, (ii) proves a discounted regret bound for an FTRL formulation of Adam under a β1≤β2 condition with the additional coupling 1−β1=C(β)(1−β2), and (iii) converts this into (ρ,ε)-stationarity guarantees. For unclipped plain Adam the paper obtains a p-dependent, suboptimal iteration complexity (e.g. O(ε^{-13/2}) at p=2), while a clipped/constrained version with known radius is claimed to match the optimal heavy-tail rate (e.g. O(ε^{-7/2}) at p=2).","tokens_in":25491,"tokens_out":11920,"duration_ms":158524,"significance":"If the results are correct as intended, this is a meaningful advance: it provides the first convergence guarantees for a vector Adam update under heavy-tailed noise without explicit truncation of the gradient, and it generalizes the online-to-nonconvex conversion machinery beyond bounded variance. The proof chain is detailed and, away from the issues noted below, internally consistent: the martingale controls, the β-weighted regret decomposition, and the balancing of β1, D, and T are transparent and lead to the claimed exponents. The paper also makes honest statements about the suboptimality of plain Adam, which is a useful conceptual contribution. However, the scope of the claim 'plain Adam' is weakened by the one-step-delayed FTRL formulation and by the C(β) coupling, and one displayed parameter choice in Corollary 6.3 is algebraically inconsistent as written.","major_comments":[{"comment":"The stated choice η = D√(1−β1) is inconsistent with the displayed regret simplification. Substituting this η into Eq. (3) gives D/(2(1−β1)^{3/2}) as the first term and 2D√β2(1−β1)^{3/2}/(β1(1−β2)) as the second, not the printed D/√(1−β1) times the printed coefficients. The displayed bound is valid only for η = D/√(1−β1). Since Corollary 6.5 uses the √(1−β1) denominator to choose β1 and obtain the claimed optimal heavy-tail exponent, this is load-bearing. The notation must be corrected to η = D/√(1−β1) (or the regret calculation revised), and the proof of Corollary 6.5 re-checked.","section":"§6.3, Corollary 6.3 (statement and proof)"},{"comment":"The abstract and contributions state that Adam is analyzed 'without restrictive parameter coupling' and that 'we only require β1 ≤ β2'. This is an overstatement: Remark 5.3 additionally requires 1−β1 = C(β)(1−β2) for a fixed constant C(β)≥1. Proposition 5.4 and the subsequent regret and stationarity bounds depend on this coupling through the deterministic output bound ∥z_t∥ ≤ η_t√C(β). The paper should state the C(β) coupling in the abstract/contributions and not present β1≤β2 as the sole condition. This is a scope issue, not just a wording issue, because the rates apply to the family of parameter pairs satisfying the coupling.","section":"§1 contributions and §5.2 Remark 5.3"},{"comment":"The algorithm analyzed is not the standard vector-form Adam update as usually implemented. In Algorithm 2, the increment z_t uses gradients v_1,...,v_{t−1} and bias-correction exponents t−1, whereas standard Adam uses v_t and bias-correction exponent t at round t. The paper calls this 'the exact vector-form Adam update' and 'plain Adam', but the one-step delay is an algorithmic modification. The authors disclose the shift in the footnote, but the abstract and contributions should state prominently that the results are for this delayed FTRL form; otherwise the 'plain Adam' claim is misleading and the practical relevance is unclear.","section":"§5, Algorithm 2"}],"minor_comments":[{"comment":"There is an inconsistency between the stated D and the proof. The theorem states D = (1−β1)^{5/4}ε^{1/2}/(C(β)ρ^{1/2}), but the proof's Δ/(DT) term uses √C(β)ρ^{1/2}Δ/((1−β1)^{5/4}ε^{1/2}T), which corresponds to D = (1−β1)^{5/4}ε^{1/2}/(√C(β)ρ^{1/2}). The constants should be aligned.","section":"§6.2, Theorem 6.4"},{"comment":"The definition of C(R) in Corollary 6.3 is a maximum of two branches, but the proof of the clipped case uses only the second branch. The first branch appears to be a vestige of the unclipped proof and should be removed or explained.","section":"§6.3, Corollary 6.3"},{"comment":"Corollary 6.5 states the second T-term with (C(R)G+σ)^{p/(p−1)}, while the abstract and discussion omit C(R). Since C(R) is a constant depending on C(β) and β1, this is not a mathematical issue, but the statements should be harmonized to avoid apparent discrepancies.","section":"§6.3, Corollary 6.5 vs Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central proof structure appears sound, and the heavy-tailed conversion extension is a genuine contribution. The main blocking issue is the η choice in Corollary 6.3: as written, the known-radius optimality claim does not follow. This is fixable by correcting η to D/√(1−β1) and rechecking the subsequent algebra. The C(β) coupling and the one-step-delayed FTRL formulation should also be presented accurately in the claims. I would be willing to look at a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is real: first convergence guarantee for the unmodified vector-form Adam under bounded p-th moments, with a clean suboptimality statement and an optimal rate when the online-learner output is clipped. The proof of the plain-Adam part (Theorem 6.4) is detailed and, as far as I checked, internally consistent. The generalization of the discounted-to-nonconvex conversion to heavy-tailed martingale differences is a useful tool in itself.\n\nThe main soft spot is the one the stress-test note flags. Corollary 6.3 sets eta = D*sqrt(1-beta1), but the displayed simplification and the resulting regret bound require eta = D/sqrt(1-beta1). Substituting the stated value leaves terms that don't simplify to the printed coefficients, and carrying them into Corollary 6.5 would break the optimal rate. This is load-bearing because the known-radius optimality is a headline claim. It's fixable—just correct the statement to eta = D/sqrt(1-beta1)—but as written, the proof doesn't support the conclusion.\n\nSecond, the contributions bullet says the analysis only requires beta1 <= beta2. That's misleading. The deterministic bound on ||z_t|| uses the ratio (1-beta1)/(1-beta2) and requires it to be bounded by a fixed constant C(beta); the theorems then choose beta2 via 1-beta1 = C(beta)(1-beta2). That's a mild coupling, not the absence of coupling. The paper's own Remark 5.3 says this, but the bullet oversells. Not a fatal issue, but it should be corrected before publication.\n\nAlso worth saying: the analyzed algorithm is the vector-form FTRL representation of Adam, not the standard coordinate-wise Adam most implementations use. The paper acknowledges this and points to an extension, but the extension isn't worked out. Fine as scoping, less fine if the title and abstract imply the exact practical algorithm.\n\nThe citation pattern looks appropriate, no data fitting, no invented entities. I checked the stress-test note algebra; it holds up.\n\nWho is this for? Optimization theorists working on adaptive methods and heavy-tailed noise. The plain-Adam suboptimality result is likely correct and worth knowing. The optimal-rate corollary is plausible once the eta typo is fixed. I'd send it to peer review, with the expectation that the author fixes the eta and rewords the parameter-coupling claim. I'd cite the heavy-tailed conversion and plain-Adam result; I'd hold off citing Corollary 6.5 until the fix is in.","headline":"Genuine plain-Adam heavy-tail result, but the known-radius optimal-rate claim has a fixable algebra slip in Corollary 6.3, and the 'no parameter coupling' bullet oversells.","tokens_in":25879,"tokens_out":5856,"would_cite":true,"duration_ms":71633,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C26","90C15","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, with suboptimal but explicit iteration complexity and an optimal rate when the update is clipped to a known doma","keywords":["Adam optimizer","heavy-tailed noise","nonconvex optimization","bounded p-th moment","discounted regret","online-to-nonconvex conversion","stationarity guarantees","iteration complexity"],"falsifier":"Take the vector-form Adam regret bound with beta1=0.9, beta2=0.99 and cost vectors with a bounded p-th moment for p=1.5 that saturate the moment bound; compute expected discounted regret over many seeds. If the regret exceeds the predicted O((1-beta1)^(-3/4)) scaling, or if for any constructed noise sequence the deterministic inequality ||z_t|| <= eta_t sqrt(C(beta)) fails, the central theorem collapses. A cheaper check: verify that for beta1=0.9, beta2=0.99 the weight ratio max_s w1/w2 stays bounded by 10=C(beta) for all t.","tokens_in":24918,"feed_emoji":"📉","tokens_out":5512,"duration_ms":81914,"temperature":0.7,"pith_summary":"The paper asks what happens to Adam when stochastic gradient noise has only a bounded p-th central moment, p in (1,2], instead of a bounded variance. It answers by extending the discounted-regret route from online learning to nonconvex optimization so that heavy-tailed martingale-difference noise is allowed. On that basis it proves that the plain vector-form Adam update converges in expectation to a (rho,epsilon)-stationary point, with a rate that depends on p and is suboptimal: the dominant term is T=O(epsilon^-(5p/(3p-4)+3/2)) for p>4/3, which is epsilon^-13/2 for p=2. When the radius of the domain is known and the online learner's output is clipped to that ball, the paper shows the rate matches the known optimal exponent p/(p-1)+3/2 for every p in (1,2]. The significance is that the exact Adam update, not a modified or clipped variant, has rigorous convergence guarantees in a regime relevant to deep learning, and the gap to optimality is identified precisely.","feed_headline":"Adam provably converges under heavy-tailed noise, with suboptimal rate","feed_subtitle":"New proof gives ε^-13/2 for plain Adam; clipping to known radius restores the optimal ε^-7/2.","key_machinery":"The argument is carried by two pieces. First, a generalized discounted-regret-to-nonconvex-stationarity conversion that uses a martingale-difference moment inequality of von Bahr-Esseen type to control heavy-tailed noise; this reduces the optimization guarantee to bounding a discounted regret plus the mean squared norm of the online learner's increments. Second, an exact vector-form Adam viewed as Follow-the-Regularized-Leader with exponentially discounted costs and a quadratic regularizer, including bias-correction terms. The key technical lemmas are a deterministic norm bound for Adam's increment, ||z_t|| <= eta_t sqrt(C(beta)), obtained from the 1-beta1 = C(beta)(1-beta2) relation, and mo","core_discovery":"The central claim is Theorem 6.4. Under a bounded-p-th-moment noise model and the mild coupling 1-beta1 = C(beta)(1-beta2) with fixed C(beta) >= 1, the vector-form Adam update, run inside the discounted conversion procedure, satisfies E[||grad F(x_tilde)||^[rho]] <= (8 + C(R) + 2C(R)C(beta)^{1/p-1/2}) epsilon after T iterations, with an explicit T. The same machinery, with the output clipped to a known domain radius, yields the optimal heavy-tail rate for all p in (1,2]. The paper thus establishes both a positive result -- plain Adam converges under heavy tails -- and a precise measurement of its suboptimality.","pith_inferences":["A direct consequence the paper does not spell out: if the C(beta) coupling between beta1 and beta2 is violated by an implementation, the deterministic increment bound can fail, which may explain empirically observed instability of Adam in heavy-tailed settings; the paper's summary statement 'only beta1 <= beta2' is stronger than what the proof supports.","Since the known-radius result clips only the online-learner output, not the gradients, a testable prediction is that equivalent norm control via weight decay or normalization, as in AdamW, could also close the optimality gap without needing the domain radius D.","The rates suggest a practical schedule for heavy-tailed problems: choose beta1 close to 1 according to epsilon, keep C(beta) small and bounded, and use output clipping when the domain radius is known; the paper does not test this empirically."],"forward_implications":["If the main theorem is right, practitioners can trust that plain Adam will not diverge under heavy-tailed noise; its worst-case iteration count grows like epsilon^-13/2 for bounded variance, a concrete price for not modifying the update.","Because the known-radius clipped version matches the optimal complexity, the result suggests explicit norm control of the update is sufficient to restore optimality; the paper leaves open whether it is necessary.","The conversion framework now covers heavy-tailed martingale-difference noise, so the same discounted-regret technique can be applied to other adaptive optimizers.","The p>4/3 restriction for unclipped Adam and p>1 for clipped Adam determines exactly which heavy-tail regimes are covered.","The numerical stabilizer in Adam's denominator is allowed to be a fixed small constant independent of G and sigma, so the analysis covers the way Adam is usually coded.","The rates imply that for p=2, plain Adam needs epsilon^-13/2 iterations while the clipped version needs only epsilon^-7/2, quantifying the cost of not using domain information."],"fun_headline_variants":["Adam provably converges under heavy-tailed noise, but suboptimally","Plain Adam converges under heavy tails at ε^-13/2","Adam's heavy-tail convergence proven; rate suboptimal","Heavy-tail Adam: proven convergence, suboptimal rate","Adam's heavy-tail rate ε^-13/2; with radius, ε^-7/2"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"All theorems rely on the relation 1-beta1 = C(beta)(1-beta2) for a fixed constant C(beta) >= 1, together with beta1 <= beta2; if the momentum decays are not coupled this way, the deterministic norm bound on Adam's update and every rate that depends on it are not proved.","fun_headline_variants_meta":{"raw":{"variants":["Adam provably converges under heavy-tailed noise, but suboptimally","Plain Adam converges under heavy tails at ε^-13/2","Adam's heavy-tail convergence proven; rate suboptimal","Heavy-tail Adam: proven convergence, suboptimal rate","Adam's heavy-tail rate ε^-13/2; with radius, ε^-7/2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001008,"raw_usage":{"total_tokens":4207,"prompt_tokens":961,"completion_tokens":3246,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":3165}},"tokens_in":705,"tokens_out":3246,"duration_ms":17936,"temperature":1.0,"reasoning_tokens":3165,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T03:18:55.152772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the vector-form Adam regret bound with beta1=0.9, beta2=0.99 and cost vectors with a bounded p-th moment for p=1.5 that saturate the moment bound; compute expected discounted regret over many seeds. If the regret exceeds the predicted O((1-beta1)^(-3/4)) scaling, or if for any constructed noise sequence the deterministic inequality ||z_t|| <= eta_t sqrt(C(beta)) fails, the central theorem collapses. A cheaper check: verify that for beta1=0.9, beta2=0.99 the weight ratio max_s w1/w2 stays bounded by 10=C(beta) for all t.","supporting_citations":[],"review_version":2}