Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

In a prototypical convex quadratic problem, Adam converges to the minimizer if and only if the data distribution is symmetric; asymmetric two-point data provably stalls at a spurious fixed point.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:13 UTC pith:ZMGFG6CH

load-bearing objection A genuinely new Adam non-convergence characterization, conditional on an unproved companion-paper bridge that a referee must verify. the 2 major comments →

arxiv 2511.06675 v2 pith:ZMGFG6CH submitted 2025-11-10 math.OC cs.LG

Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

Steffen Dereich , Thang Do , Arnulf Jentzen , Philippe von Wurstemberger This is my paper
classification math.OC cs.LG MSC 90C1562L2065K10
keywords Adam optimizerstochastic gradient descentstrongly convex optimizationconvergence ratesnon-convergencesymmetry theoremAdam vector fieldmini-batch size
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to give a complete convergence characterization for the Adam optimizer on the simplest nontrivial strongly convex stochastic problem: minimize E[(θ-X)^2] over θ, where X comes from a two-point distribution. It proves that Adam converges to the unique minimizer—in the strong L^p sense and at the standard rate sqrt(γ_n)—if and only if the data distribution is symmetric around the minimizer. For every asymmetric two-point distribution, Adam provably does not converge to the minimizer; instead it approaches a spurious fixed point of the 'Adam vector field', an expected normalized-update field governing Adam's large-time behavior. The paper also proves quantitative error bounds for strongly convex stochastic optimization, showing an additional unavoidable error term of order M^{-1}(1-β_2) alongside the standard sqrt(γ_n) term. If these theorems are right, Adam has a fundamental bias on convex quadratics that no number of training steps removes—only larger mini-batches or β_2 closer to 1 shrink it.

Core claim

Stated on the paper's own terms, the Adam symmetry theorem says: for the loss L(θ,x)=||θ-x||^2 with i.i.d. two-point data taking values v and w, any β_1∈(0,1), β_2∈(β_1^2,1), and any decreasing learning rates satisfying a standard regularity condition, the following are equivalent: (i) the p-th moment of the Adam error from the minimizer ϑ is bounded by c sqrt(γ_n); (ii) for every mini-batch size M, the iterates converge to ϑ in probability; and (iii) the data X_{1,1} is symmetrically distributed. The heart of the proof is the Adam vector field: the expected value of Adam's normalized update evaluated at a frozen point. The paper invokes a companion result, quoted as Theorem 3.2, that any li

What carries the argument

The central object is the Adam vector field, defined for a frozen θ as f_i^M(θ) = (1-α) E[ Σ_{n≥0} α^n X_{n,i}^M(θ) / (ε + sqrt((1-β) Σ_{n≥0} β^n |X_{n,i}^M(θ)|^2)) ], where X_{n,i}^M is the negative mini-batch averaged gradient component and α, β are the momentum parameters. This field is what the Adam recursion tracks in the large-time limit, and the paper's central mechanism is the bridge 'limit points of Adam = zeros of this field', imported from a companion article. For symmetric two-point data the field is strictly monotone and vanishes exactly at the minimizer, so convergence rates follow from a local-monotonicity framework. For asymmetric data the field is nonzero at the minimizer an

Load-bearing premise

The non-convergence half of the Adam symmetry theorem depends on the companion-paper result (quoted here as Theorem 3.2) that every limit point of Adam must be a zero of the Adam vector field; this bridge is not proved in this manuscript, and if it breaks, Adam could converge to the minimizer even for asymmetric two-point data.

What would settle it

Fix d=1, β_1=0.9, β_2=0.999, ε=1e-8, γ_n=n^{-0.7}, and let X take values -1 and 0.1 with probabilities 1/11 and 10/11, so the minimizer is 0. Run Adam with M=1 from θ_0=0 for about 10^6 steps and estimate E[min{1,|θ_n|}] as well as E[θ_n]. The theorem predicts these do not go to 0; the limiting value should match the zero of the Adam vector field, located roughly at distance 0.001 from 0. Observing convergence to 0 would refute the characterization; observing a plateau at the predicted nonzero root would confirm it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In the quadratic two-point test problem, Adam with any hyperparameters satisfying β_1^2 < β_2 fails to converge to the minimizer for every asymmetric two-point data distribution, while the standard SGD method converges in the same setting.
  • The strong-convexity error bound for Adam contains an additive M^{-1}(1-β_2) term that the symmetry theorem shows cannot be removed: the bias persists for all n and is governed by a spurious zero of the Adam vector field.
  • When the data is symmetric around the minimizer, Adam attains the standard sqrt(γ_n) rate, so the extra bias is not a general defect of Adam but specifically a symmetry effect.
  • The deterministic case v=w is automatically symmetric, so Adam's non-convergence to the minimizer does not occur when stochasticity is absent.
  • The paper's numerical experiments indicate that the rates sqrt(γ_n), M^{-1}, and 1-β_2 are sharp, with the plateau appearing exactly at the predicted spurious root.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: the load-bearing structural condition exposed by the proof is symmetry of the stochastic gradient noise at the minimizer, so for a general strongly convex loss the analogous non-convergence should be triggered by asymmetric gradient noise, not necessarily by asymmetric raw data; this extension is not in the paper.
  • My inference: since the bias is of order M^{-1}(1-β_2), a practical remedy suggested by the theorem—increasing batch size or annealing β_2 toward 1—should remove the plateau; the paper treats fixed β_2, so this remedy is my extrapolation, not its claim.
  • My inference: the characterization supplies a cheap falsification probe for practitioners: if a convex quadratic training run stalls, checking whether the plateau location moves linearly with 1-β_2 and 1/M can distinguish an Adam spurious fixed point from a learning-rate effect.
  • My inference: replacing the two-point distribution by continuous asymmetric noise and checking whether the same spurious zero persists at the same order is a natural stress test of the mechanism beyond the theorem's hypotheses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper analyzes Adam on strongly convex stochastic optimization problems (SOPs). The first main result, the “Adam symmetry theorem” (Theorem 1.1 / Theorem 4.12), considers quadratic losses L(θ,x)=‖θ−x‖² with i.i.d. two-point data X∈{v,w}, under standard Adam hyperparameter conditions and learning-rate assumptions. It claims equivalence between (i) an L^p error bound of order √γ_n, (ii) convergence in probability of Adam to the unique minimizer, and (iii) symmetric distribution of the data. The second main result (Theorem 1.2 / Theorem 2.10) gives strong error bounds for strongly convex SOPs under an explicit boundedness assumption, including a non-learning-rate term of order M^{-1}(1−β₂). The proof framework splits into a rate-analysis part built on the authors' earlier work [13] and a non-convergence part built on a limit-point characterization imported from the companion paper [14].

Significance. If the symmetry theorem is correct, it is a striking and nontrivial result: Adam provably fails to converge to the unique minimizer for non-symmetric two-point data even in a prototypical strongly convex quadratic problem, and the failure is governed by a spurious fixed point of the Adam vector field at distance O(M^{-1}(1−β₂)). The rate bound with the explicit M^{-1}(1−β₂) term is also a useful new contribution, and the paper gives detailed proofs for the self-contained parts. The numerical simulations support the sharpness of the claimed rates and the predicted spurious fixed point. The main caveat is that the non-convergence direction — the most surprising half of the symmetry theorem — rests on an external theorem that is not proved in this manuscript; the rate results and the symmetric-data direction are independent of that bridge and are much more firmly established.

major comments (2)
  1. [§3.1, Theorem 3.2 (Eq. (234))] Theorem 3.2 is the load-bearing bridge for the “only if” direction of the symmetry theorem. It states that if Adam converges in probability to a random variable ϑ, then f(ϑ)=0 P-a.s., where f is the Adam vector field. The proof is a one-line citation to [14, Corollary 1.10]. Proposition 3.5 and Corollary 3.6 then use f(0)<0 to conclude non-convergence for non-symmetric two-point data. Thus the central claim that Adam fails to converge to the minimizer is not independently established in this manuscript: if [14, Cor. 1.10] fails in this two-point quadratic regime, the non-convergence claim collapses. The symmetric-data direction and the rate theorems would survive. I recommend either proving the needed variant of the limit-point characterization in this paper, or explicitly stating it as an assumption and clearly marking the non-convergence claim as conditional on it.
  2. [§1.2 / Theorem 1.2] The announced convergence rates for “strongly convex SOPs” are conditional on the assumption inf_{N,m} P(‖Θ^{0,N,β}_m‖≤p)=1 for a fixed p. This boundedness assumption is not derived in the paper for the general setting; the paper notes that it is satisfied in the quadratic case by a companion result [12]. Since the theorem is stated with this assumption, the claim is formally correct. Still, the abstract and introduction tend to state the rate result more broadly, and the reader could be misled. Please make the conditional nature prominent in the abstract and in the introduction.
minor comments (4)
  1. [§2.5, Lemma 2.14 proof] In the proof of Lemma 2.14, the text says “the fact that (γ_n) is increasing” for γ_n=n^{-r}, which is decreasing. Also in Eq. (361), the telescoping sum contains an index typo: the summand should be (γ_k−γ_{k+1}), not (γ_n−γ_{n−1}). The conclusion is correct, but the typos should be fixed.
  2. [§1.2 / Theorem 1.2 statement] Theorem 1.2 states p∈N, while the abstract advertises strong L^p rates for p∈[1,∞). State the precise p-range in the abstract or adjust the theorem statement. The condition β₂∈[p^{-1},1) is empty for p=1, which also deserves a comment.
  3. [§4.4, Theorem 4.12] The equivalence in Theorem 4.12 is stated for all p∈(0,∞), but the proof of (iii)⇒(ii) invokes Proposition 4.10, which is stated for integer p. A short interpolation or monotonicity argument is needed to cover non-integer p; otherwise the theorem should be restricted to integer p.
  4. [§5, Figure 4 caption] The right panel of Figure 4 uses P(X=−1)=w/(1+w) as the x-axis, but this expression is not defined in the caption text or in Setting 5.1. Please add a one-line explanation.

Circularity Check

1 steps flagged

Only-if direction rests on a load-bearing self-citation to companion paper [14]; no constructional circularity in the core derivation.

specific steps
  1. self citation load bearing [Section 3.1, Theorem 3.2 (proof); used in Corollary 3.6 and in the proof of Theorem 4.12, item (i)⇒(iii)]
    "Proof of Theorem 3.2. Note that [14, Corollary 1.10] implies that E[|f(ϑ)|]=0. The proof of Theorem 3.2 is thus complete."

    The only-if half of the symmetry theorem is contrapositive: for non-symmetric two-point data Proposition 3.5 shows f(0)<0 at the centered minimizer, so convergence to the minimizer would contradict Theorem 3.2's conclusion P(f(ϑ)=0)=1. Theorem 3.2 is the sole bridge from stochastic-process convergence to an algebraic zero of the Adam vector field, and in this manuscript its proof is a one-line citation to the authors' companion paper [14], posted the same week and not proved here. Thus the central non-convergence claim reduces to a load-bearing self-citation rather than to a derivation contained in this paper. This is a dependency, not a normalization/fit identity, so it does not make the result definitionally circular.

full rationale

The paper's core derivation is not circular by construction. The Adam vector field f is an independently defined object; the key computation f(0)<0 for asymmetric two-point data in Proposition 3.5 is a genuine analytic derivation, and the M^{-1}(1−β2) offset in the error bounds comes from the explicit bound ∥θ_M−ϑ∥ ≤ ρℳ M^{-1} in Lemma 2.7/Proposition 2.9, not from fitting data or renaming a fitted parameter. The symmetric-data direction and the convergence-rate theorems are self-contained given the cited prior framework [13]. The one genuinely load-bearing external input is Theorem 3.2, imported from [14, Corollary 1.10], which underpins the non-convergence claim for non-symmetric data. That theorem is not proved in this paper, and its proof is a one-line self-citation to a companion preprint posted the same week; if it fails in this two-point quadratic regime, the only-if direction collapses. This is a transparency and verification concern, but it is not an equation-level equivalence between the paper's inputs and conclusions. Accordingly, the overall circularity score is low: 2.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 1 invented entities

No free parameters are fitted: every constant (c_0, c_beta, delta, M, varrho, chi) is an existential bound from a proof, and no number is tuned to make the theorems work. The claims rest on standard inequalities (BDG, CLT, dominated convergence), explicit domain assumptions (bounded Adam path for the rate theorem; two-point symmetric noise for the characterization; slow-decay learning rates), and one external same-week companion theorem ([14, Cor. 1.10]) that the authors do not prove here.

axioms (7)
  • standard math Strong convexity of theta |-> E[L(theta, X_{1,1})] (Definition 2.1) and the implied coercivity inequality (11)-(12).
    Basis for converting vector-field/local-monotonicity control into distance-to-minimizer bounds throughout Section 2 (cf. (12) used in the literature review and Proposition 2.9).
  • standard math Burkholder-Davis-Gundy inequality with universal constant B for the third moment.
    Invoked in Lemma 2.7 and Proposition 2.9 to bound third-moment terms of mini-batch gradient noise.
  • standard math Central limit theorem for square-integrable i.i.d. summands.
    Lemma 2.5 uses the CLT to show the mini-batch gradient at the minimizer is small with controlled probability for large M; the bound feeds Lemma 2.7.
  • domain assumption Boundedness of the Adam path: inf_{N,m} P(||Theta^{0,N,beta}_m|| <= p) = 1.
    Load-bearing assumption of Theorem 1.2 / Corollary 2.11; for general strongly convex SOPs it is assumed, not proved (the body refers to the authors' [12] for quadratic cases). The abstract omits it, making the advertised rate result stronger than the theorem.
  • domain assumption Learning-rate regime: lim sup ((gamma_n)^{-2}(gamma_n - gamma_{n+1})) = 0 and sum (gamma_n)^p < infinity.
    Forces gamma_n to decay strictly slower than 1/n (Lemma 2.14); standard for this group's Adam analyses [13] but stronger than what classical SGD analyses need.
  • domain assumption Theorem 3.2 = [14, Corollary 1.10]: any in-probability limit of Adam is P-a.s. a zero of the vector field f.
    External, same-week, overlapping-author companion result; it carries the 'only if' direction of the symmetry theorem. Not proved in this manuscript (stated as Theorem 3.2 with proof deferred to arXiv:2511.04622).
  • domain assumption Two-point / bounded i.i.d. data model for the characterization (Theorem 1.1).
    Defines the scope of the iff statement; the theorem claims nothing outside this class, so this is a naming of scope rather than a hidden assumption.
invented entities (1)
  • Adam vector field f_M (Setting 2.2, Eq. (14); introduced in the authors' [13]) independent evidence
    purpose: Expectation of the infinite-horizon normalized momentum; its zeros are the only possible Adam limit points (Theorem 3.2), and the distance from its zeros to the minimizer generates the M^{-1}(1-beta_2) error term.
    Not a physical entity; a mathematical construction inherited from the authors' prior work. It does carry a falsifiable handle: the zero of f_M predicts where Adam actually ends up, which any independent run of Adam on the two-point quadratic problem can test.

pith-pipeline@v1.3.0-alltime-deepseek · 72915 in / 27129 out tokens · 241441 ms · 2026-08-03T23:13:59.009262+00:00 · methodology

0 comments
read the original abstract

Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training of deep neural networks in artificial intelligence (AI) systems. Despite the popularity and the success of Adam it remains an \emph{open research problem} to provide a rigorous convergence analysis for Adam even for the class of strongly convex SOPs. In one of the main results of this work we establish convergence rates for Adam in terms of the number of gradient steps (convergence rate \nicefrac{1}{2} w.r.t. the size of the learning rate), the size of the mini-batches (convergence rate 1 w.r.t. the size of the mini-batches), and the size of the second moment parameter of Adam (convergence rate 1 w.r.t. the distance of the second moment parameter to 1) for the class of strongly convex SOPs. In a further main result of this work, which we refer to as \emph{Adam symmetry theorem}, we illustrate the optimality of the established convergence rates by proving for a special class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity to the solution of the SOP (the unique minimizer of the strongly convex objective function) if and \emph{only} if the random variables in the SOP (the data in the SOP) are \emph{symmetrically distributed}. In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we \emph{disprove} that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity. We also complement the conclusions of our convergence analysis and the Adam symmetry theorem by several numerical simulations that indicate the sharpness of the established convergence rates and that illustrate the practical appearance of the phenomena revealed in the \emph{Adam symmetry theorem}.

Figures

Figures reproduced from arXiv: 2511.06675 by Arnulf Jentzen, Philippe von Wurstemberger, Steffen Dereich, Thang Do.

Figure 1
Figure 1. Figure 1: Approximative plot of E [PITH_FULL_IMAGE:figures/full_fig_p060_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Approximative plot of E [PITH_FULL_IMAGE:figures/full_fig_p061_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Approximative plot of E [PITH_FULL_IMAGE:figures/full_fig_p062_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Approximative plot of E [PITH_FULL_IMAGE:figures/full_fig_p062_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks

    math.OC 2026-07 accept novelty 7.0

    Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...

  2. On the Convergence of Adam, Revisited

    cs.LG 2026-07 accept novelty 7.0

    Projected Adam and listed variants have lim RT/T > 0 for all β1, β2 ∈ [0,1) on a 3-periodic linear online problem with slopes near 2, −1, −1.

  3. Central limit theorem for the averaged Adam optimizer

    math.PR 2026-06 unverdicted novelty 5.0

    Establishes a central limit theorem for averaged Adam with n^{-1/2} convergence rate to an attracting zero and covariance determined by the algorithm at the attractor.

Reference graph

Works this paper leans on

7 extracted references · 6 linked inside Pith · cited by 3 Pith papers

  1. [1]

    (373)) againstw∈(∪ 7 i=−8{2i}) whereM= 1,β 1 = 0.9,β 2 = 0.9,ε= 10 −8, andv=−1 (cf

    1.0 0.8 0.6 0.4 0.2 0.0 Symmetric distribution Minimum point of optimization problem Expected Adam end point Zero of Adam vector field Figure 4: Approximative plot ofE Θ0 400000 and a zero of the Adam vector fieldf(cf. (373)) againstw∈(∪ 7 i=−8{2i}) whereM= 1,β 1 = 0.9,β 2 = 0.9,ε= 10 −8, andv=−1 (cf. Setting 5.1).Left:We measure the asymmetry of the dist...

    Pith/arXiv arXiv 2018

  2. [10]

    arXiv:1807.06766(2018)

    [10]De, S., Mukherjee, A., and Ullah, E.Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration. arXiv:1807.06766(2018). [11]Dereich, S., Graeber, R., and Jentzen, A.Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rate...

    Pith/arXiv arXiv 2018

  3. [16]

    arXiv:2006.06657(2020)

    [39]Ji, Z., and Telgarsky, M.Directional convergence and alignment in deep learning. arXiv:2006.06657(2020). [40]Kingma, D. P., and Ba, J.Adam: A Method for Stochastic Optimization. arXiv:1412.6980(2014). 64 [41]Klenke, A.Probability Theory, 2nd ed. Springer-Verlag London Ltd., (2014), xii+638. [42]Kovachki, N. B., and Stuart, A. M.Continuous time analysi...

    Pith/arXiv arXiv 2006

  4. [40]

    arXiv:2505.09572(2025)

    [43]Kranz, J., Gallon, D., Dereich, S., and Jentzen, A.SAD Neural Net- works: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures. arXiv:2505.09572(2025). [44]Kuckuck, B.Some useful latex commands.https://latex.bennokuckuck.de [Accessed 22- August-2025](2025). [45]Li, H., Qian, J., Tian, Y., Rakhlin, A., and Jadbabaie, A.Convex and...

    arXiv 2025

  5. [47]

    [50]Liu, Y., Gao, Y., and Yin, W.An Improved Analysis of Stochastic Gradient Descent with Momentum.arXiv:2007.07989(2020)

    [49]Li, X., and Orabona, F.On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes.arXiv:1805.08114(2018). [50]Liu, Y., Gao, Y., and Yin, W.An Improved Analysis of Stochastic Gradient Descent with Momentum.arXiv:2007.07989(2020). [51]Lu, L., Shin, Y., Su, Y., and Karniadakis, G. E.Dying ReLU and initialization: theory and numerical examp...

    Pith/arXiv arXiv 2018

  6. [2024]

    [3]Barakat, A., and Bianchi, P.Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization.SIAM J

    [2]Barakat, A., and Bianchi, P.Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for Non Convex Optimization.arXiv:1911.07596(2019). [3]Barakat, A., and Bianchi, P.Convergence and dynamical behavior of the Adam algorithm for nonconvex stochastic optimization.SIAM J. Optim. 31, 1 (2021), 244–274. [4]Bock, S., Goppold, J., and Weiß, M.An ...

    Pith/arXiv arXiv 1911

  7. [2025]

    [61]Reddi, S

    https://pytorch.org/docs/stable/generated/torch.optim.Adam.html(accessed July 9, 2025). [61]Reddi, S. J., Kale, S., and Kumar, S.On the Convergence of Adam and Beyond. arXiv:1904.09237(2019). [62]Ruder, S.An overview of gradient descent optimization algorithms.arXiv:1609.04747 (2016). [63]Shen, L., Chen, C., Zou, F., Jie, Z., Sun, J., and Liu, W.A Unified...

    Pith/arXiv arXiv 2025