REVIEW 3 major objections 4 minor 19 references
Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Adam's convergence rate depends on epsilon through a 1/epsilon^2 term, and replacing epsilon with a softplus-calibrated step size provably removes that penalty.
desk verdict Softplus calibration is a plausible empirical trick, but the advertised convergence theory rests on a monotonicity condition the algorithms do not satisfy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the adaptive learning rate itself, written as 1/($\sqrt$(v_t)+epsilon) in Adam and 1/softplus($\sqrt$(v_t)) in Sadam, where v_t is the exponential moving average of squared gradients. The proof machinery is a bounded-A-LR lemma: under L-smoothness, bounded gradients, and bounded variance, every coordinate of the A-LR lies in [mu_l, mu_u], with mu_l = 1/($\sqrt$($sigma^{2}$+$G^{2}$)+epsilon) and mu_u = 1/epsilon for Adam, and analogous bounds (mu_3, mu_4) involving $\beta$ for Sadam. These bounds are then fed into a telescoping sum over an auxiliary sequence z_t, which turns the per-iteration decrease of f(z_t) into a bound on the average squared gradient norm. The telescoping step uses the inequality (a-b)^2 <= $a^{2}$ - $b^{2}$, which is valid only when the coordinate-wise second moment is nondecreasing, i.e., v_t >= v_{t-1}. That monotonicity is stated in the appendix as a standing condition for all the analyses, even though the Adam and Sadam updates as defined in Algorithms 1 and 2 do not enforce it.
What would settle it
Run Adam or Sadam exactly as defined in Algorithms 1 and 2 on a small smooth problem whose coordinate-wise v_t is strictly decreasing at some iterations, then check whether the claimed gradient-norm bound of Theorem 2 or Theorem 9 holds; a single instance where the bound is violated while all assumptions of Assumption 1 hold would show the theorems are not valid without the monotonicity condition. More directly, compute the term sum_j [(1/($\sqrt$(v_{t-1,j})+epsilon))^2 - (1/($\sqrt$(v_{t,j})+epsilon))^2]; whenever v_t < v_{t-1} this difference is negative, and the paper's Lemma 23 step (a-b)^2 <= $a^{2}$ - $b^{2}$ no longer applies.
Extended reading notes
Core claim
The central claim is that Adam's convergence behavior is governed by the bounds of its adaptive learning rate, and that those bounds are set by epsilon: Adam has 1/($\sqrt$(v_{t,j}) + epsilon) between mu_1 = 1/($\sqrt$($sigma^{2}$+$G^{2}$)+epsilon) and mu_2 = 1/epsilon, so all rate constants scale as 1/$epsilon^{2}$, 1/epsilon, and d/$epsilon^{2}$. The paper further claims that calibrating the adaptive learning rate with softplus, which is lower bounded by (1/$\beta$) log 2 and grows linearly for large arguments, replaces the 1/epsilon factors by $\beta$ factors and yields strictly better dependence under nonconvex, non-strongly convex, and Polyak-Lojasiewicz settings. Concretely, Sadam's nonconvex rate is O($beta^{2}$/$\sqrt$(T) + d $\beta$/T + d $beta^{2}$/(T $\sqrt$(T))), versus Adam's O(1/($epsilon^{2}$ $\sqrt$(T)) + d/(epsilon T) + d/($epsilon^{2}$ T $\sqrt$(T))). With the recommended $\beta$=50 and standard epsilon=1e-8, the difference is six orders of magnitude in the leading constant. The authors' experiments on MNIST, CIFAR-10, CIFAR-100, and LSTM language models are offered as evidence that Sadam and SAMSGrad close the generalization gap between Adam and S-Momentum.
Load-bearing premise
Every convergence proof in the paper assumes the coordinate-wise second moment never decreases, v_t >= v_{t-1}, a condition stated only in the appendix and not enforced by Adam or Sadam as defined in Algorithms 1 and 2.
Editorial extensions
If this is right
- Adam's convergence guarantees are incomplete without epsilon: the rate degrades as 1/epsilon^2, so the common choice epsilon = 1e-8 carries a huge hidden constant.
- The softplus calibration gives Sadam and SAMSGrad rates that replace 1/epsilon with beta, and with beta=50 they are provably faster than Adam under the same assumptions.
- When beta is small, Sadam recovers the SGD convergence rate O(1/sqrt(T)) in nonconvex and non-strongly convex settings, matching S-Momentum in T-dependence.
- Under the Polyak-Lojasiewicz condition, both Adam and Sadam converge linearly at rate (1 - 2 lambda mu/T^2)^T, with mu depending on epsilon or beta respectively.
- Because the calibration applies to any adaptive gradient method, combining softplus with AMSGrad (SAMSGrad) yields the same improved rates and better empirical test accuracy.
Reading between the lines
- The monotonicity gap suggests a clean testable repair: if Adam and Sadam were run with the AMSGrad-style max operation v_t = max(v_{t-1}, tilde v_t), the theorems would hold as stated; the paper's SAMSGrad already does this, which may be why its theory is cleanest.
- One could extend the calibration to other monotone activation functions; the bounded-A-LR lemma only needs a positive monotone map, so sigmoid, ELU, or tanh variants would likely enjoy the same rate form with different beta-like constants.
- The anisotropic A-LR plots hint that the per-coordinate spread, not just its max, may predict generalization; a direct test would be measuring the interquartile range of 1/softplus(sqrt v_t) against final test accuracy across beta values.
- If epsilon-dependence transfers to other adaptive methods such as Yogi or AdaBound, then their reported improvements may be partly explained by how they constrain the A-LR range rather than by their specific update rules.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper revisits adaptive gradient methods from the perspective of the adaptive learning rate (A-LR), argues that the A-LR is anisotropic, and proposes to calibrate it with a softplus function, giving the Sadam and SAMSGrad algorithms. The main theoretical claims are that Adam's convergence rate depends on epsilon as O(1/(epsilon^2 sqrt(T)) + d/(epsilon T) + d/(epsilon^2 T sqrt(T))) and that Sadam achieves O(beta^2/sqrt(T) + d beta/T + d beta^2/(T sqrt(T))), with similar improvements in non-strongly convex and Polyak-Lojasiewicz settings. Empirical comparisons on MNIST, CIFAR, and LSTM benchmarks report that the proposed methods improve test accuracy over Adam, AMSGrad, and several other optimizers.
Significance. The empirical study is broad and the code is released, and the observation that Adam's A-LR ranges widely across coordinates (Section 3.1) is clearly illustrated. If the claimed epsilon/beta dependence were rigorously established, the paper would make a useful contribution. However, the theoretical comparison is not currently supportable: the proofs rely on a coordinate-wise monotonicity condition that is false for the algorithms as defined, and another key inequality is invalid as written. The advertised advantage of Sadam over Adam therefore rests on unproven rates; the paper's empirical results do not compensate for this gap because the central claim is the convergence comparison.
major comments (3)
- [Appendix 11.2, Lemma 23; Algorithms 1-2] The appendix begins the Adam and Sadam nonconvex proofs with the statement that all analyses hold under the condition vt >= vt-1, and Lemma 23's third inequality uses (a-b)^2 <= a^2 - b^2, which requires a >= b, i.e., v_t >= v_{t-1}. This condition is not stated in Theorems 2, 3, 4, 9, 12, or 14, and it is false for Algorithm 1: with v_t = beta_2 v_{t-1} + (1-beta_2) g_t^2, the sequence can decrease whenever recent squared gradients are small. Consequently, the telescoping sums over 1/sqrt(v_t) and 1/v_t do not go through, and the claimed rates O(1/(epsilon^2 sqrt T) + d/(epsilon T) + d/(epsilon^2 T sqrt T)) for Adam and O(beta^2/sqrt T + d beta/T + d beta^2/(T sqrt T)) for Sadam are unsupported. Only SAMSGrad (Algorithm 2, which uses the max update) satisfies the monotonicity condition. This is the central derivation gap and invalidates the abstract and Section 5 claims.
- [Appendix 11.2 and 11.3, lower-bound step for the inner product] The proof lower-bounds E[<grad f(x_t), (1/(sqrt v_t + epsilon)) g_t>] by splitting coordinates according to the sign of grad_i g_i and then replacing grad_i g_i by grad_i^2. This is not valid pointwise: on the event grad_i > 0 and g_i > 0, one has grad_i g_i < grad_i^2 whenever g_i < grad_i, which is not ruled out by Assumption 1; the analogous failure occurs on the negative-sign branch. The resulting lower bound eta mu_1 ||grad f(x_t)||^2 is therefore not established. The same defect appears in the Sadam proof. A repair would require an additional conditional independence or moment assumption that is not stated.
- [Section 5, Remark 10; Corollary 11] The claimed 'better convergence speed' of Sadam over Adam is obtained by comparing beta = 50 with epsilon = 1e-8 inside loose upper bounds; this is a comparison of two hyperparameter values, not a parameter-free property of the algorithms. With epsilon = 1e-3, as used in the Adam and Yogi rows of Table 1, the constants O(1/epsilon^2) and O(1/epsilon) would shrink by orders of magnitude, eliminating the stated advantage. Thus, even setting aside the proof gap in the first major comment, the comparison does not support the paper's central claim of superior convergence.
minor comments (4)
- [Appendix 11.4, non-strongly convex proof] The sentence 'The second inequality is based on the fact that, when iteration t reaches the maximum number T, xt is the optimal solution, zT = x*' is false; the inequality is valid by simply dropping the nonpositive term -E[||z_T - x*||^2], so the proof should be reworded.
- [Theorems 4 and 14 vs. Theorem 30] The main text states eta = O(1/T^2) for the P-L condition, while Theorem 30 in the appendix states eta = O(1/T) and the proof then sets eta = 1/T^2; these should be aligned.
- [Introduction and Section 3.1] The contribution statement claims 'theoretical guarantees of both optimization and generalization error analysis,' but no generalization bound is proved anywhere in the manuscript; either provide such a result or remove the claim.
- [Throughout] Typos and inconsistent notation remain: 'funciton' (Section 4.1), 'diffentiable' (Lemma 29), 'perfoms' (Remark 28), duplicate '(d)' in the Figure 1 caption, and 'exsiting' (Appendix 9).
Circularity Check
No significant circularity: the derivation is self-contained and the theoretical concerns are correctness/scope issues, not circular reductions.
full rationale
I walked the claimed derivation chain. The convergence rates in the main text are derived in the appendix from Lemma 21 (bounded A-LR) and Lemma 23, and the appendix explicitly states: “All the analyses hold true under the condition: vt>=vt-1.” This condition is not satisfied by Adam or Sadam as defined in Algorithms 1 and 2, because vt is an exponential moving average that can decrease; the proof of Lemma 23 uses (a-b)^2 <= a^2-b^2 exactly under that condition. This is a serious correctness and scope problem for Theorems 2, 3, 4, 9, 12, and 14, but it is not circularity: the theorems do not define their conclusions into their assumptions, and the failure is a false or unstated premise rather than an equation reducing to its own input. The claimed Sadam-vs-Adam acceleration in Remark 10 compares the upper-bound constants O(beta^2) vs O(1/epsilon^2) using the recommended beta=50 and customary epsilon=1e-8. That comparison is weak and hyperparameter-dependent, but the beta-dependence follows algebraically from the softplus definition and the bounded-A-LR lemma; it is not a fitted parameter being renamed as a prediction, and it is not supported by a load-bearing self-citation. The empirical generalization claims rest on standard benchmark experiments. I therefore find no circular step under the specified definitions.
Assumptions & free parameters
free parameters (1)
- beta (softplus temperature) =
recommended 50, grid searched over {10, 50, 100} in the main text and up to 1000 in the appendix
assumptions (5)
- domain assumption Assumption 1: L-smoothness, bounded gradients (||grad f_i|| <= G), and bounded stochastic gradient variance (E||g_t - grad f(x_t)||^2 <= sigma^2)
- ad hoc to paper Coordinate-wise monotonicity vt >= vt-1 for all t
- domain assumption Bounded iterates: E||xt-x*|| <= D and E||xm-xn|| <= D_infinity (used in convex analysis)
- standard math Existence of a global minimum and a minimizer x*
- domain assumption Polyak-Lojasiewicz condition with parameter lambda (for the P-L results)
Cite this review
Pith. "Pith review of Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM." pith.science (2026). https://pith.science/paper/POR5YBSL
@misc{pith2026190800700,
author = {Pith},
title = {Pith review of: Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM},
year = {2026},
howpublished = {\url{https://pith.science/paper/POR5YBSL}},
note = {Machine review of arXiv:1908.00700}
}
abstract
Adaptive gradient methods (AGMs) have become popular in optimizing the nonconvex problems in deep learning area. We revisit AGMs and identify that the adaptive learning rate (A-LR) used by AGMs varies significantly across the dimensions of the problem over epochs (i.e., anisotropic scale), which may lead to issues in convergence and generalization. All existing modified AGMs actually represent efforts in revising the A-LR. Theoretically, we provide a new way to analyze the convergence of AGMs and prove that the convergence rate of \textsc{Adam} also depends on its hyper-parameter $\epsilon$, which has been overlooked previously. Based on these two facts, we propose a new AGM by calibrating the A-LR with an activation ({\em softplus}) function, resulting in the \textsc{Sadam} and \textsc{SAMSGrad} methods \footnote{Code is available at https://github.com/neilliang90/Sadam.git.}. We further prove that these algorithms enjoy better convergence speed under nonconvex, non-strongly convex, and Polyak-{\L}ojasiewicz conditions compared with \textsc{Adam}. Empirical studies support our observation of the anisotropic A-LR and show that the proposed methods outperform existing AGMs and generalize even better than S-Momentum in multiple deep learning tasks.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
+TLηµ 2 2 µ1 (σ2 +G2) Divided by 1 T , 1 T T∑ t=1 [‖∇f(xt)‖2]≤ 1 ηµ1TE[f(z1)−f∗] + β1d (1−β1)µ1TG √ σ2 +G2(µ2−µ1) + 1 2µ1 L2ηµ2 2( β1 1−β1 )2(σ2 +G2) + 1 2µ1 ηµ2 2(σ2 +G2) + Lηβ 2 1d(σ2 +G2) (1−β1)2µ1T (µ2 2−µ2
-
[3]
+Lηµ2 4T (σ2 +G2) µ3 37 Divided by 1 T , 1 T T∑ t=1 [‖∇f(xt)‖2]≤ 1 ηµ3TE[f(z1)−f∗] + β1d (1−β1)µ3TG √ σ2 +G2(µ4−µ3) + L2ηµ2 4 2µ3 ( β1 1−β1 )2(σ2 +G2) +ηµ2 4 2µ3 (σ2 +G2) + Lηβ 2 1d(σ2 +G2) (1−β1)2µ3T (µ2 4−µ2
-
[6]
+Lηµ2 2 µ1 (σ2 +G2) ≤ 1 ηµ1TE[f(z1)−f∗] + ( β1d (1−β1)µ1T (µ2−µ1) + 1 2µ1 L2ηµ2 2( β1 1−β1 )2 + ηµ2 2 2µ1 + Lηβ 2 1d(µ2 2−µ2 1) (1−β1)2µ1T + Lηµ2 2 µ1 )(σ2 +G2) The second inequality holds because G √ σ2 +G2≤σ2 +G2. Setting η = 1√ T , let x0 =x1, then z1 =x1, f(z1) =f(x1) we derive the final result: min t=1,...,T E[‖∇f(xt)‖2]≤ 1 µ1 √ T E[f(x1)−f∗] + ( β1d ...
work page 2013
-
[8]
Architecture Used in Our Experiments Here we mainly introduce the MNIST architecture with Pytorch used in our empirical study, ResNets and DenseNets are well-known architectures used in many works and we do not include details here. layer layer setting F.relu(self.conv1(x)) self.conv1 = nn.Conv2d(1, 6, 5) F.max pool2d(x, 2, 2) F.relu(self.conv2(x)) self.c...
-
[9]
More Empirical Results In this section, we perform multiply experiments to study the property of anisotropic A-LR exsinting in AGMs and the performance of softplus function working on A-LR. We first show the A-LR range of popular Adam-type methods, then present how the parameterβ in Sadam and SAMSGrad reduce the range of A-LR and improve both training and ...
-
[10]
Besides the figures in main text, we have repeated experiments and show results as follows
CIF AR100 Two popular CNN architectures are tested on CIFAR-100 dataset to compare different algorithms: VGGNet (Simonyan and Zisserman, 2014) and ResNets18 (He et al., 2016). Besides the figures in main text, we have repeated experiments and show results as follows. Our proposed methods again perform slightly better than S-Momentum in terms of test accurac...
work page 2014
-
[11]
Theoretical Analysis Details We analyze the convergence rate of Adam and Sadam under different cases, and derive competitive results of our methods. The following table gives an overview of stochastic gradient methods convergence rate under various conditions, in our work we provide a different way of proof compared with previous works and also associate th...
-
[12]
+Lηµ2 4(σ2 +G2) µ3 ≤ 1 ηµ3TE[f(z1)−f∗] + ( β1d (1−β1)µ3T (µ4−µ3) + L2ηµ2 4 2µ3 ( β1 1−β1 )2 + ηµ2 4 2µ3 + Lηβ 2 1d (1−β1)2µ3T (µ2 4−µ2
Show all 19 references
-
[13]
With fixed L,σ,G,β 1, we have C1 =O(β2), C2 =O(dβ), C3 =O(dβ2)
+Lηµ2 4 µ3 )(σ2 +G2) Setting η = 1√ T , let x0 =x1, then z1 =x1, f(z1) = f(x1) we derive the final result for Sadam method: min t=1,...,T E[‖∇f(xt)‖2]≤ 1 µ3 √ T E[f(x1)−f∗] + ( β1d (1−β1)µ3T (µ4−µ3) + L2µ2 4 2µ3 √ T ( β1 1−β1 )2 + µ2 4 2µ3 √ T + Lβ2 1d(µ2 4−µ2 3) (1−β1)2µ3T √ T...
-
[14]
By Jensen’s inequality, 1 T T∑ t=1 (f(xt)−f∗)≥f(¯xt)−f∗, where ¯xt = 1 T ∑T t=1xt
+ηµ2 2T µ1 (σ2 +G2) +β1d(σ2 +G2) 2ηµ1(1−β1) (µ2 2−µ2 1) + ηβ1D2T µ1(1−β1) + ηβ 3 1D2 ∞T µ1(1−β1)3 + ηβ1µ2 2T (1−β1)µ1 (σ2 +G2) The second inequality is based on the fact that, when iteration t reaches the maximum numberT , xt is the optimal solution, zT =x∗. By Jensen’s inequa...
-
[15]
f(¯xt)−f∗≤ D2 2µ1 √ T + β2 1d(σ2 +G2) (1−β1)2µ1T √ T (µ2 2−µ2
+ηµ2 2 µ1 (σ2 +G2) + β1d(σ2 +G2) 2ηµ1(1−β1)T (µ2 2−µ2 1) + ηβ1D2 µ1(1−β1) + ηβ 3 1D2 ∞ µ1(1−β1)3 + ηβ1µ2 2 (1−β1)µ1 (σ2 +G2) By plugging the stepsize η = O( 1√ T ), we complete the proof of Adam in non-strongly convex case. f(¯xt)−f∗≤ D2 2µ1 √ T + β2 1d(σ2 +G2) (1−β1)2µ1T √ T (µ2 2−µ2
-
[16]
42 Remark 27 The leading item of convergence order of Adam should be O( ˜C√ T ), where ˜C = D2 2µ1 + µ2 2 µ1 (σ2 +G2) + β1d(σ2+G2) 2µ1(1−β1) (µ2 2−µ2
+ µ2 2 µ1 √ T (σ2 +G2) + β1d(σ2 +G2) 2µ1(1−β1) √ T (µ2 2−µ2 1) + β1D2 µ1(1−β1) √ T + β3 1D2 ∞ µ1(1−β1)3√ T + β1µ2 2 (1−β1)µ1 √ T (σ2 +G2) =O( 1√ T ) +O( 1 T √ T ) =O( 1√ T ). 42 Remark 27 The leading item of convergence order of Adam should be O( ˜C√ T ), where ˜C = D2 2µ1 + µ...
-
[17]
+ β1D2 µ1(1−β1) + β3 1D2 ∞ µ1(1−β1)3 + β1µ2 2 (1−β1)µ1 (σ2 +G2). With fixed L,σ,G,β 1,D,D∞, ˜C = O( d ϵ2 ), which also contains ϵ as well as dimension d, here with bigger ϵ, the order should be better, this also supports the discussion in our main paper. The analysis of Sadam i...
-
[18]
f(¯xt)−f∗≤ D2 2µ3 √ T + β2 1d(σ2 +G2) (1−β1)2µ3T √ T (µ2 4−µ2
+ηµ2 4 µ3 (σ2 +G2) + β1d(σ2 +G2) 2ηµ3(1−β1)T (µ2 4−µ2 3) + ηβ1D2 µ3(1−β1) + ηβ 3 1D2 ∞ µ3(1−β1)3 + ηβ1µ2 4 (1−β1)µ3 (σ2 +G2) By plugging the stepsize η = O( 1√ T ), we get the convergence rate of Sadam in non- strongly convex case. f(¯xt)−f∗≤ D2 2µ3 √ T + β2 1d(σ2 +G2) (1−β1)2...
-
[19]
For brevity, f(¯xt)−f∗ =O( 1√ T )
+ µ2 4 µ3 √ T (σ2 +G2) + β1d(σ2 +G2) 2µ3(1−β1) √ T (µ2 4−µ2 3) + β1D2 µ3(1−β1) √ T + β3 1D2 ∞ µ3(1−β1)3√ T + β1µ2 4 (1−β1)µ3 √ T (σ2 +G2) =O( 1√ T ) +O( 1 T √ T ) =O( 1√ T ). For brevity, f(¯xt)−f∗ =O( 1√ T ). Remark 28 The leading item of convergence order of Sadam should be ...
-
[20]
+ β1D2 µ3(1−β1) + β3 1D2 ∞ µ3(1−β1)3 + β1µ2 4 (1−β1)µ3 (σ2 +G2). With fixed L,σ,G,β 1,D,D∞, ˜C = O(dβlog(1 +eβ)) = O(dβ2), with small β, the Sadam will be similar to SGD convergence rate, and β is a much smaller number compared with 1/ϵ, proving that Sadam method perfoms better...
-
[21]
+Lη2µ2 2T )(σ2 +G2) =θT Φ1 +O(ηT ) +O(η2T ) +O(η) +O(η2) From the above inequality, η should be set less than O( 1 T ) to ensure all items in the RHS small enough. Set η = 1 T 2 , then θ = 1− 2ληµ1 = 1− 2λµ1 T 2 ΦT +1 =θT Φ1 +O( 1 T ) +O( 1 T 3 ) +O( 1 T 2 ) +O( 1 T 4 ) =θT Φ1...
-
[22]
Set η =O( 1 T 2 ), E[f(xT +1)−f∗]≤ (1− 2λµ3 T 2 )TE[f(x1)−f∗] +O( 1 T )
+Lη2µ2 4T )(σ2 +G2) =θT Φ1 +O(ηT ) +O(η2T ) +O(η) +O(η2) By setting appropriate η, we can also prove the Sadam converges under PL condition (and strongly convex). Set η =O( 1 T 2 ), E[f(xT +1)−f∗]≤ (1− 2λµ3 T 2 )TE[f(x1)−f∗] +O( 1 T ). Overall, we have proved Adam algorithm an...
-
[2013]
38 Remark 26 The leading item from the above convergence is C1/ √ T , β plays an essential role in the complexity, and a more accurate convergence should be O(βlog (1+eβ )√ T )
in nonconvex optimization in terms of T . 38 Remark 26 The leading item from the above convergence is C1/ √ T , β plays an essential role in the complexity, and a more accurate convergence should be O(βlog (1+eβ )√ T ). When β is chosen big, this will become O( β2 √ T ), someh...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.