{"id":"e5d268be-0428-48ee-b5f5-24ea4504c1b7","arxiv_id":"2504.19426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Adam can achieve the accelerated momentum convergence rate locally on smooth strongly convex problems when its momentum and step size are tuned to the condition number, while RMSprop is shown to converge at the slower gradient descent rate.","lead":"This paper proves that, with hyperparameters tuned to the problem, the Adam optimizer can converge locally as fast as accelerated momentum methods on smooth deterministic problems, while RMSprop converges at the plain gradient descent rate. The result supplies sharp rate guarantees that clarify how adaptivity and momentum interact in optimizer design.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1.2's proof instantiates Corollary 4.11 with the wrong momentum coefficient: α must be R^2, not R, so the central rate does not follow as written.","rationale":"I read the paper's central contribution as the existence statement in Theorem 1.2(iv): some hyperparameter choice for the Adam update (4.45) yields local linear convergence at the Polyak momentum rate. The internal scaffolding for this claim is credible: Proposition 4.9 gives a self-contained contraction argument for the extended (Θ,m,M) system near the minimizer, Lemma 3.2 supplies the spectral-radius calculation, and no external result is needed for Corollary 4.11. The most concrete place where the argument as written fails is the instantiation of Corollary 4.11 in the proof of Theorem 1.2: the substituted α is the desired rate R rather than the momentum coefficient R^2 required by the corollary. The same error appears in item (ii). This is a small, easily corrected slip, not a refutation of the mathematical claim, so it does not move the verdict. I also agree with the reader's weaker-assumption observation: the rate holds only for tuned (α,γ), not for default Adam hyperparameters, and the abstract overstates this; however that is a scope/communication issue rather than a gap in the formal existence theorem.","tokens_in":26847,"tokens_out":21568,"duration_ms":223333,"concrete_test":"Re-run the substitution: in Corollary 4.11 set κ=r, K=rχ, γ=ε/(r√χ), and α=(√χ−1)^2/(√χ+1)^2. Verify that γ ε^{-1}κ=1/√χ, so the corollary's hypothesis α=((1−γ ε^{-1}κ)/(1+γ ε^{-1}κ))^2 holds, and that its conclusion (4.46) becomes ‖Θ_n‖≤c((√χ−1)/(√χ+1)+δ)^n. Then repeat the check with the paper's α=(√χ−1)/(√χ+1): the hypothesis fails and the conclusion would only give rate ((√χ−1)/(√χ+1))^{1/2}. This settles whether the proof of Theorem 1.2(iv) closes and exactly what the corrected hyperparameter choice must be.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 1.2(iv) says to apply Corollary 4.11 with κ↦r, K↦rχ, γ↦r^{-1/2}(rχ)^{-1/2}ε, and α↦(√χ−1)/(√χ+1) = R. But Corollary 4.11 presupposes α = ((1−γ ε^{-1}κ)/(1+γ ε^{-1}κ))^2. With the substituted γ, γ ε^{-1}κ = r^{-1/2}(rχ)^{-1/2}·r = 1/√χ, so the right-hand side is ((1−1/√χ)/(1+1/√χ))^2 = R^2. Hence the paper's α=R does not satisfy the corollary's hypothesis. If one nevertheless runs the conclusion with α=R, the proven exponential rate is √R, which is slower than R and does not establish convergence with χ-order R (for ρ∈(R,√R) the defining supremum need not be finite). The same slip occurs in item (ii) via Corollary 4.10. The gap is easily repaired by setting α=R^2, but as written the direct proof of the paper's headline rate does not close. The reader's separate concern about hyperparameter tuning is also real: the corrected proof only covers Adam with momentum coefficient R^2 and step size ε/(r√χ), both depending on the Hessian bounds; the abstract's unqualified \"Adam converges with the faster rate\" is not a statement about default hyperparameters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies local exponential convergence rates of RMSprop and Adam for smooth deterministic objectives near a strict local minimizer. The technical core is a set of perturbation results for linear iterations whose matrices converge (Lemmas 2.3–2.5), used to reduce the local dynamics of adaptive optimizers to the spectral radius of a limiting linearization. For a condition number χ, the authors prove that RMSprop attains the same rate as gradient descent, (χ−1)/(χ+1), and that Adam attains the accelerated momentum rate (√χ−1)/(√χ+1). The latter is obtained by combining a local contraction argument for a class of momentum iterations with convergent adaptive step sizes (Proposition 4.9) with the observation that Adam's bias-corrected prefactors converge to 1. The paper also contains a global convergence result for smooth strongly convex objectives with small step sizes and a comparison theorem showing that Adam asymptotically beats GD.","tokens_in":27149,"tokens_out":12824,"duration_ms":126292,"significance":"If the main result stands, this is a significant contribution: it gives the first explicit sharp local convergence rate for Adam matching the optimal momentum rate, with a clean and largely self-contained proof. The rates are derived from the spectral radius of the limiting linearization, not fitted to simulations, and no constants are tuned to data. The authors are honest that the GD and momentum parts are classical, and the external input from [2] is confined to a preparatory gradient-vanishing lemma that is not needed for the sharp local-rate proof. The main caveat is that the rate is an existence statement: it requires the momentum coefficient and step size to be chosen using the Hessian bounds, so the unqualified abstract sentence overstates what is proven for default hyperparameters.","major_comments":[{"comment":"The proof of Theorem 1.2(ii) and (iv) applies Corollary 4.10 and Corollary 4.11 with the wrong momentum coefficient. Corollary 4.10 requires α = ((1−γκ)/(1+γκ))^2, and Corollary 4.11 requires α = ((1−γε^{-1}κ)/(1+γε^{-1}κ))^2. In Theorem 1.2(ii), the substitution γ = r^{-1/2}(rχ)^{-1/2} gives γκ = 1/√χ, so the corollary requires α = R^2 with R = (√χ−1)/(√χ+1), whereas the proof sets α = R. With α = R, Corollary 4.10 only yields the rate √R, which is slower than the claimed rate R. The identical mismatch occurs in item (iv): substituting γ = r^{-1/2}(rχ)^{-1/2}ε gives γε^{-1}κ = 1/√χ, so Corollary 4.11 requires α = R^2, not α = R. Thus the cited implications, as written, do not prove the headline rate. The repair is immediate: set α ↦ R^2 in both applications; since α is existential in Theorem 1.2, the statement of the theorem remains valid after this correction.","section":"Theorem 1.2 (Section 1.2) and Corollaries 4.10–4.11"}],"minor_comments":[{"comment":"The text states 'Lemma 3.2 therefore demonstrates that ρ(A) ≤ α'; this should be ρ(A) = √α (or ≤ √α). The subsequent conclusion uses the exponent √α + δ, so the typo does not affect the result, but the displayed inequality is incorrect.","section":"Section 3.4, proof of Proposition 3.4"},{"comment":"The notation in Theorem 4.6 is confusing: γ is used both for the uniform upper bound γ ∈ (0, 1/(4K)) and for a step-size variable γ ∈ [0, γ], and the expression Θ^{α,β,ε}_γ_n in the conclusion is not defined. The proof uses η = εγ, so the statement should be aligned with the proof.","section":"Section 4.5, Theorem 4.6"},{"comment":"The abstract's sentence 'Adam also converges with the strictly faster rate' is unqualified. The theorem itself is an existence result: the accelerated rate requires the momentum coefficient α = R^2 and a step size γ of order ε/√(κK), both depending on the Hessian bounds. The abstract should indicate this qualification.","section":"Abstract and Section 1.2"},{"comment":"In the chain of inequalities, the denominator 'κ + 2√χ + 1' should read 'χ + 2√χ + 1'.","section":"Equation (1.3)"},{"comment":"In the display after (4.40), the perturbation term is written as h_{Γ_θ_n}(Θ_θ_n, m_θ_n); it should be evaluated at the previous iterate, h_{Γ_θ_n}(Θ_θ_{n−1}, m_θ_{n−1}), as the preceding line and the bound require.","section":"Section 4.6, proof of Proposition 4.9"}],"recommendation":"major_revision","confidential_remarks":"The main issue is a localized but load-bearing substitution error in the proof of Theorem 1.2: α must be R^2 rather than R when applying Corollaries 4.10 and 4.11. The fix is straightforward and the central claim appears correct after it. If the authors correct this, qualify the abstract, and clean up the noted notation issues, I would support acceptance. I see no concern about novelty disclosure or citation practice; the reliance on [2] is clearly stated and confined to a preparatory lemma."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper is worth reading—the main result is real and important—but the proof of Theorem 1.2 as written has a parameter slip that needs a small fix before the headline rate is actually established. The slip is easy to repair, so this is a conditional accept, not a reject.\n\nWhat's new: they give explicit local exponential rates for RMSprop and Adam. The Adam result, matching the momentum rate (√χ−1)/(√χ+1), was not in the cited predecessors. The mechanism is clean: near a strict minimizer the adaptive scaling factor in Adam converges to a constant, so the analysis reduces to a momentum iteration with converging step sizes; RMSprop similarly reduces to gradient descent. The spectral-radius arguments in Propositions 2.8, 3.3, 3.4, 4.9 are coherent and the rates follow. They also correctly credit Polyak for the GD and momentum items.\n\nNow the soft spots, in proportion. The stress-test is right: Theorem 1.2's proof applies Corollary 4.11 with α set to the desired rate R=(√χ−1)/(√χ+1), but Corollary 4.11's hypothesis is α = ((1−γε^{-1}κ)/(1+γε^{-1}κ))^2, and with their choice of γ the right side is R^2. The corollary's conclusion is a rate √α, so plugging in α=R gives only √R, not R. The same mistake appears in item (ii). The fix is simply to set α=R^2. This is a genuine gap in the write-up, but it is localized and I would not call the underlying claim into question.\n\nTwo more caveats. The abstract says RMSprop 'only' converges with the GD rate, but the theorem is an upper bound on the rate; there is no lower bound here, so 'only' is too strong. And the Adam rate requires α and γ tuned to the Hessian bounds at the minimizer; the abstract does not say that. Neither changes the mathematics, but they matter for how the result is advertised. Also, the gradient-convergence lemma 4.2 is imported from Barakat–Bianchi; that is legitimate, but the local-rate proof is not fully self-contained.\n\nThe paper is for optimization theorists working on adaptive methods or local convergence. It deserves a serious referee. My recommendation: send it out, but ask the authors to fix the α↦α^2 slip and soften the abstract.","headline":"Important result on Adam's local rate, but the proof of Theorem 1.2 instantiates the wrong momentum coefficient—easy fix, deserves refereeing.","tokens_in":27708,"tokens_out":6484,"would_cite":true,"duration_ms":55774,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68W40","90C25"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adam with Hessian-tuned hyperparameters provably converges at the same rate as momentum, while RMSprop stays at gradient-descent speed.","keywords":["Adam optimizer","RMSprop","momentum method","convergence rate","condition number","adaptive learning rate","spectral radius","local minimizer"],"falsifier":"Simulate the exact quadratic $L(\\theta)=\\frac12\\sum_i\\lambda_i(\\theta_i-\\psi_i)^2$ with known $\\kappa$ and $K$, initialize near $\\psi$, and run Adam with the paper's $\\alpha=((1-(\\gamma/\\varepsilon)\\kappa)/(1+(\\gamma/\\varepsilon)\\kappa))^2$ and $\\gamma=\\varepsilon/\\sqrt{\\kappa K}$, with any $\\beta\\in(0,1)$. If the empirical contraction rate of $\\|\\Theta_n-\\psi\\|$ is not bounded by $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)+\\delta$ for all sufficiently small $\\delta>0$, the theorem's rate is false. Running the same experiment with default $\\alpha=0.9$ would not falsify the theorem, because that setting lies outside its assumptions.","tokens_in":26612,"feed_emoji":"📉","tokens_out":9885,"duration_ms":92567,"temperature":0.7,"pith_summary":"The paper proves a sharp local convergence-rate theorem for the Adam optimizer: if the momentum coefficient and step size are set using the extreme eigenvalues of the Hessian at a local minimizer, then Adam converges in a neighborhood of that minimizer with rate $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)$, where $\\chi$ is the Hessian's condition number. This is the same rate as the momentum method and strictly faster than gradient descent's classical rate $(\\chi-1)/(\\chi+1)$. The paper also proves that RMSprop, Adam's adaptive-rate cousin without momentum, reaches only the slower gradient-descent rate. The result applies to deterministic $C^2$ objectives whose Hessian at the minimizer lies between $rI$ and $r\\chi I$.","feed_headline":"Tuned Adam matches momentum's optimal convergence rate","feed_subtitle":"Proof: with step size and momentum chosen from Hessian bounds, Adam keeps the faster momentum rate.","key_machinery":"The proof treats Adam as a momentum iteration with adaptive per-coordinate step sizes. The load-bearing object is the linear iteration in the joint state $(\\Theta_n,m_n)$ whose coefficient matrices $A_n$ converge, as the moment estimates settle, to the constant block matrix of the momentum method; its spectral radius is $\\sqrt{\\alpha}$. A generalized Gelfand spectral-radius lemma, together with a perturbation lemma for linear iterations, transfers that limiting spectral radius to the time-varying system, and a local-stability argument shows that the second-moment estimate and the bias-correction factors converge so that the adaptive step-size schedule settles at the fixed step $\\gamma/\\varepsilon$. The same machinery yields the RMSprop rate by viewing RMSprop as plain gradient descent with learning rates converging to $2/(\\kappa+K)$.","core_discovery":"On its own terms, the central discovery is Theorem 1.2(iv) via Corollary 4.11: for every dimension, every condition number $\\chi\\in(1,\\infty)$, and every $C^2$ objective $L$ with $(\\nabla L)(\\psi)=0$ and $rI\\preceq(\\mathrm{Hess}\\,L)(\\psi)\\preceq r\\chi I$, there exists a momentum coefficient $\\alpha\\in(0,1)$ such that the $\\alpha$-$\\beta$-$\\varepsilon$-Adam optimizer converges to $\\psi$ with rate $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)$. The theorem exhibits the specific tuning $\\alpha=((1-(\\gamma/\\varepsilon)\\kappa)/(1+(\\gamma/\\varepsilon)\\kappa))^2$ with $\\gamma\\le\\varepsilon/\\sqrt{\\kappa K}$, where $\\kappa=r$ and $K=r\\chi$ are the smallest and largest Hessian eigenvalues. The companion claim is that every RMSprop optimizer converges with the slower rate $(\\chi-1)/(\\chi+1)$ (Corollary 2.9). Concretely, the rate means that for every $\\rho$ larger than the stated rate, the distance $\\|\\Theta_n-\\psi\\|$ after $n$ steps stays bounded by a constant times $\\rho^n$; in short, Adam's adaptivity costs nothing in local speed when the hyperparameters know the local curvature.","pith_inferences":["An immediate consequence the authors do not state: if a practitioner keeps a fixed momentum such as $\\alpha=0.9$ instead of the Hessian-tuned value, the theorem's accelerated rate is not in force; nothing in the proof bounds that case.","Because the mechanism only uses that Adam's step-size factors converge to $1$ and the momentum block's spectral radius is $\\sqrt{\\alpha}$, similar local rate statements should hold for debiased Adam variants and for weight-decoupled Adam, as long as the same tuning is used.","A stochastic version of the local rate should appear before noise dominates: with small-variance gradient noise, the deterministic contraction should dominate early iterations, so finite-time bounds should show a window where the accelerated exponent is visible.","The RMSprop/Adam separation predicts that the acceleration in Adam comes entirely from momentum; a minimal experiment would compare Adam with momentum coefficient $\\alpha=0$ against the theorem's tuned $\\alpha$ on a quadratic."],"forward_implications":["For every deterministic $C^2$ objective whose Hessian at the minimizer lies between $rI$ and $r\\chi I$, Adam with the tuned parameters converges locally with rate $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)$: for every $\\rho$ above that rate, $\\sup_n \\rho^{-n}\\|\\Theta_n-\\psi\\|$ is finite.","RMSprop, with any $\\beta\\in[0,1)$ and $\\varepsilon>0$, is shown to converge with the slower rate $(\\chi-1)/(\\chi+1)$, exactly the gradient-descent rate.","Because $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)<(\\chi-1)/(\\chi+1)$ for every $\\chi>1$, Adam is strictly faster than gradient descent in every sufficiently ill-conditioned quadratic neighborhood.","The weak optimization error $|L(\\Theta_n)-L(\\psi)|$ inherits the squared rates $(\\sqrt{\\chi}-1)^2(\\sqrt{\\chi}+1)^{-2}$ for Adam and $(\\chi-1)^2(\\chi+1)^{-2}$ for RMSprop."],"supporting_citations":[{"why":"Supplies the classical momentum convergence rate and the perturbation lemma that the paper generalizes to time-varying adaptive iterations.","marker":"[36]"},{"why":"Provides the momentum-algorithm convergence theorem and Proposition 14 used in Lemma 4.2 to prove Adam's gradients converge to zero under Lipschitz smoothness.","marker":"[2]"},{"why":"Cited for the lower bound showing that the accelerated rate cannot be improved within the class of first-order methods.","marker":"[34]"},{"why":"Source of the generalized Gelfand spectral-radius lemma and the matrix-norm perturbation that converts limiting spectral radii into finite-step rates.","marker":"[25]"},{"why":"Standard reference for the gradient-descent and momentum convergence facts and for the matrix lemma used to compute the momentum block spectrum.","marker":"[27]"}],"fun_headline_variants":["Tuned Adam matches momentum's optimal rate","Adam can match momentum if tuned","Adam's rate equals momentum's when tuned","RMSprop slower, Adam matches momentum","Adam matches momentum's rate with tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accelerated rate holds only when the optimizer's momentum coefficient and step size are set from the smallest and largest Hessian eigenvalues at the minimizer; if those numbers are unknown and a fixed default momentum such as 0.9 is used, the theorem's optimality claim is not asserted.","fun_headline_variants_meta":{"raw":{"variants":["Tuned Adam matches momentum's optimal rate","Adam can match momentum if tuned","Adam's rate equals momentum's when tuned","RMSprop slower, Adam matches momentum","Adam matches momentum's rate with tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1518,"prompt_tokens":1076,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":692,"tokens_out":442,"duration_ms":4599,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:55:08.794259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the exact quadratic $L(\\theta)=\\frac12\\sum_i\\lambda_i(\\theta_i-\\psi_i)^2$ with known $\\kappa$ and $K$, initialize near $\\psi$, and run Adam with the paper's $\\alpha=((1-(\\gamma/\\varepsilon)\\kappa)/(1+(\\gamma/\\varepsilon)\\kappa))^2$ and $\\gamma=\\varepsilon/\\sqrt{\\kappa K}$, with any $\\beta\\in(0,1)$. If the empirical contraction rate of $\\|\\Theta_n-\\psi\\|$ is not bounded by $(\\sqrt{\\chi}-1)/(\\sqrt{\\chi}+1)+\\delta$ for all sufficiently small $\\delta>0$, the theorem's rate is false. Running the same experiment with default $\\alpha=0.9$ would not falsify the theorem, because that setting lies outside its assumptions.","supporting_citations":[{"cited_title":"Some methods of speeding up the convergence of iteration met hods","cited_arxiv_id":null,"evidence_quote":"Supplies the classical momentum convergence rate and the perturbation lemma that the paper generalizes to time-varying adaptive iterations."},{"cited_title":"Introductory lectures on convex optimization , vol","cited_arxiv_id":null,"evidence_quote":"Cited for the lower bound showing that the accelerated rate cannot be improved within the class of first-order methods."},{"cited_title":"A., and Johnson, C","cited_arxiv_id":null,"evidence_quote":"Source of the generalized Gelfand spectral-radius lemma and the matrix-norm perturbation that converts limiting spectral radii into finite-step rates."}],"review_version":1}