Pith. sign in

REVIEW 5 major objections 6 minor 35 references

Adaptive Inertial Method

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adaptive momentum rule yields O(1/k) for any inertial term

desk verdict Genuinely new adaptive momentum rule, but the headline O(1/k) theorem is false as stated and needs a repairable extra assumption; worth refereeing after major revision. read the letter →

arxiv 2505.15114 v1 pith:I32UDNKI submitted 2025-05-21 math.OC

Han Long , Bingsheng He , Yinyu Ye , Jiheng Zhang This is my paper · ORCID
classification math.OC MSC 90C2590C3090C5365K10
keywords adaptiveinertialmethodmomentumaccelerationconvexoptimizationfirst-ordermethodsquasi-NewtonregularizedNewtonparameterselectionO(1/k)convergencerate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the Adaptive Inertial Method (AIM), a first-order optimization scheme in which the momentum gain is chosen adaptively rather than by hand. The central claim is that, for any nonzero inertial direction, AIM converges globally at rate O(1/k) under only convexity and local Lipschitz differentiability, with no strong-convexity assumption. The same framework, with a particular inertial term, is shown to coincide with the regularized Newton method and to inherit its O(1/$k^{2}$) rate, while still avoiding Hessian inversions. A positive result here would give practitioners a principled way to pick momentum without tuning.

What carries the argument

The central object is the matrix $M_k = I_n + (\mu_k/(1-\mu_k)) \Pi_k$, built from the projection $\Pi_k$ onto the inertial vector $m_k$. Rewriting AIM as a proximal-point-like update $x_{k+1} = x_k - \beta_k M_k^{-1} \nabla f(x_k)$, the adaptive gain $\gamma_k = \mu_k (m_k^\top \nabla f(x_k)) / (\|m_k\|^2 \beta_k)$ is exactly the choice that makes the update a metric projection step with respect to the $M_k$-norm. The convergence argument telescopes the per-iteration inequality (2.15) over iterations to obtain the $O(1/k)$ bound; the same $M_k$ structure is used to derive the quasi-Newton and Hessian-gradient inertial terms that connect AIM to regularized Newton.

What would settle it

Run AIM on a convex quadratic with a deliberately chosen inertial direction that makes $\|M_k\|$ grow unboundedly (for example, $m_k$ alternates in a way that aligns with the projection), and check whether the bound $f(x_k)-f(x^*)$ actually follows $O(1/k)$ or whether the proof's inequality breaks; a failure of the bound would falsify the claimed rate.

Watch

Extended reading notes

Core claim

The core discovery is a parameter selection rule that guarantees descent regardless of which nonzero inertial direction is used. Writing the iteration as $x_{k+1} = x_k - \beta_k M_k^{-1}\nabla f(x_k)$, with $M_k^{-1} = I - \mu_k \Pi_k$ where $\Pi_k$ projects onto the inertial direction $m_k$, the choice $\gamma_k = \mu_k (m_k^\top \nabla f(x_k)) / (\|m_k\|^2 \beta_k)$ makes the objective strictly decrease each step. The proof then establishes an $O(1/k)$ convergence bound by telescoping an inequality measured in the $M_k$-norm. A second result shows that when the inertial direction is the product of the Hessian and gradient, and the Hessian satisfies a low-degree polynomial condition, AIM becomes algebraically equivalent to the regularized Newton method and achieves $O(1/k^2)$ without computing matrix inverses.

Load-bearing premise

The convergence proof telescopes the $M_k$-norm over iterations as if $M_k$ were a constant matrix, but $M_k$ changes every iteration because it depends on the inertial term $m_k$, and the paper supplies no explicit uniform bound on $M_k$.

Editorial extensions

If this is right

  • Any nonzero inertial direction, such as velocity, acceleration, Hessian-gradient product, or a custom mapping, can be plugged into AIM and still yields the $O(1/k)$ rate.
  • No manual tuning of the momentum gain is required: $\gamma_k$ is fully determined by $m_k$, $\nabla f(x_k)$, and $\beta_k$.
  • With the Hessian-gradient product as the inertial term and a rank-one (or low-polynomial) Hessian condition, AIM reproduces the regularized Newton update and achieves $O(1/k^2)$ with only one extra gradient evaluation per step.
  • The method retains the simple per-iteration cost of a first-order method while the adaptive gain can mimic second-order behavior in favorable regimes.
  • The step-size condition (2.1b) is satisfied by a backtracking rule, so the method is implementable without knowing the Lipschitz constant in advance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $O(1/k)$ proof appears to require the $M_k$-norm to telescope over iterations; since $M_k$ depends on $m_k$ and changes each step, a fully rigorous statement would need a uniform bound on $\|M_k\|$ or a corrected inequality. This is our reading, not the paper's claim.
  • If that gap is real, the guaranteed rate may degrade for inertial directions that make $M_k$ grow, even though the method may still perform well numerically.
  • The blockwise diagonal variant sketched in Remark 2.5 could be tested as a memory-efficient extension for large-scale problems.
  • The Hessian-gradient inertial term, which approximates the Hessian by finite differences, suggests a natural stochastic variant where the same curvature estimate is computed on minibatches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes the Adaptive Inertial Method (AIM), an inertial first-order scheme of the form x_{k+1}=x_k-β_k∇f(x_k)+γ_k m_k with the momentum gain γ_k chosen adaptively as γ_k=μ_k (m_k^T∇f(x_k))/||m_k||^2 β_k, μ_k∈[0,1), so that each step is guaranteed to decrease the objective. The main theoretical contribution is a claimed global O(1/k) convergence rate for any nonzero inertial term m_k under only convexity and local Lipschitz differentiability of the objective, together with a derivation of particular inertial terms that turn AIM into a quasi-Newton method and, under special structural conditions, into a regularized Newton method with an O(1/k^2) rate. The paper also reports numerical comparisons on logistic regression and L2-Lp minimization problems.

Significance. The adaptive rule for γ_k is a novel and attractive feature, and the possibility of unifying many momentum methods by a single framework is appealing. The numerical experiments are extensive, include reproducible code, and show AIM variants (especially with Hessian-gradient inertia) performing well against several baselines. However, the central convergence-rate theorem is false as stated, and the quasi-Newton/regularized-Newton connections rely on restrictive or tautological constructions. The core idea is worth pursuing, and the rate claim appears repairable by adding a uniform bound on μ_k, but the paper as written does not deliver the advertised guarantees.

major comments (5)
  1. [§2.2, Theorem 2.10] The theorem is false as stated. Let f(x)=x^2/2, x_0=1, β_k=0.1, η=0.4, m_k=x_k, and μ_k=1-1/(k+2). These choices satisfy all hypotheses (convexity, local Lipschitz differentiability, (2.1b), μ_k∈[0,1), m_k≠0), but the update (2.1) gives x_{k+1}=(1-β/(k+2))x_k, so f(x_k)=Θ(k^{-0.2}), contradicting the claimed O(1/k) rate in (2.22). A uniform bound μ_k≤μ_max<1 (or equivalently liminf(1-μ_k)>0) would exclude this counterexample and is needed for the theorem.
  2. [§2.2, Lemma 2.9] The summation leading to (2.18) is not a valid telescoping. Inequality (2.15) is stated in the M_j-norm, and M_k defined in (2.4) depends on m_k and therefore changes with j. Summing (2.15) over j leaves cross-terms of the form ∥x_{j+1}-x^*∥_{M_j} and ∥x_{j+1}-x^*∥_{M_{j+1}}, and the final bound (2.18) states ∥x_0-x^*∥_{M_k} with the terminal index k, which does not follow. The proof requires either a fixed norm or a uniform norm-equivalence bound on the family {M_j}, neither of which is stated.
  3. [§2.2, Theorem 2.10] Even if (2.18) were valid, (2.22) would not imply O(1/k) because the numerator contains ∥x_0-x^*∥^2_{M_k}, which grows with k when μ_k→1 (e.g., as (1-μ_k)^{-1}). Also, the constant C in (2.22) is defined with a sum starting at j=1, but the negative j=0 term (1-2η)∥x_0-x_1∥^2_{M_0} from (2.18) is not included, so the inequality does not follow from (2.18) for η>0.5.
  4. [§2.2] Theorem 2.10 omits the constant-step-size assumption β_k≡β that is introduced immediately before Lemma 2.9. The proof of (2.18)-(2.22) uses a single β outside the sums; the stated theorem allows variable β_k, so the proof does not apply to the stated result.
  5. [§3.1-3.3 and abstract] The O(1/k^2) claim is not a general property of AIM. Theorem 3.1 requires the restrictive condition H_k^2=a_k H_k, and Theorem 3.3 forces equality with regularized Newton by constructing m_k, r_k, and β_k from the characteristic polynomial of H_k. That construction is a parameter-matching device, not an independent convergence result, and computing q_k(H_k)g_k for a degree-(n-1) polynomial is not a practical "no matrix inversion" procedure. The abstract's phrasing should be qualified accordingly.
minor comments (6)
  1. [Page 1 header] The title contains a typo: "ADAPTIVE INER TIAL METHOD" should be "ADAPTIVE INERTIAL METHOD".
  2. [§2, Remark 2.4] The stated implicit form x_{k+1}=x_k-β_k M_k^{-1}∇f(x_{k+1}) is not equivalent to the explicit update (2.6), which uses ∇f(x_k); the proximal-point interpretation is therefore incorrect as written.
  3. [§2.2, around (2.15)] The notation should make clear that the M_k-norm is iteration-dependent; otherwise equations such as (2.16)-(2.17) appear to treat M_k as constant.
  4. [§3.2, equation (3.3)] The condition ∥s∥^2<α s^T y is a curvature condition on the iterates, not a statement that f is locally strongly convex; the sentence after (3.3) overstates the implication.
  5. [§3.3] The quantity r_k=(g_k^T H_k^2 g_k)/(g_k^T H_k g_k) is undefined when g_k^T H_k g_k=0; the nonzero-inertial-term requirement should be stated explicitly.
  6. [Algorithm A.1, line 16] The backtracking check uses ∥x_k-x_{k+1}∥^2_{M_k} as in Remark 2.7, but the theorems in Section 2 are stated under condition (2.1b); the relationship between the implemented condition and the theoretical condition should be clarified.

Circularity Check

1 steps flagged · score 5.0 of 10

The regularized-Newton 'coincidence' is imposed by construction; the central O(1/k) claim is not circular, though its proof appears invalid as written.

  1. self definitional [Section 3.3, Theorem 3.3 and its proof (equations (3.12)–(3.14))]
    "If we choose the inertial term m_k = q_k(H_k)g_k and set r_k = ..., with coefficients defined by the recursive relationships: b_{k,n-1}=1, b_{k,n-2}=a_{k,n-1}-θ_k b_{k,n-1}, ..., b_{k,0}=a_{k,1}-θ_k b_{k,1}+r_k/µ_k, and step size parameter: β_k=1/(θ_k+µ_k/r_k(a_{k,0}-θ_k b_{k,0})), then the Adaptive Inertial Method coincides with the regularized Newton method, provided both methods are initialized at the same starting point."

    The unknowns of the theorem (the coefficients of q_k, the scalar r_k, and β_k) are recursively defined precisely from the requirement that the AIM update equal the regularized Newton update. The proof then enforces this equality by setting c_{k,i} proportional to the characteristic-polynomial coefficients a_{k,i} (equation (3.14): c_{k,i} = (r_k/µ_k)a_{k,i}). Thus the 'coincidence' with regularized Newton is not an independent finding; it is a parameter choice manufactured to satisfy the target identity. The advertised inherited O(1/k^2) rate is therefore a restatement of the known regularized-Newton rate for a method that has been algebraically forced to be regularized Newton.

full rationale

The central O(1/k) argument in Section 2.2 is not circular: the momentum gain γ_k is chosen so that the current step is a descent step, but the O(1/k) rate is not encoded in that choice, and the adaptive rule is an algorithm design rather than a fitted prediction. The serious problem there is a mathematical flaw, not circularity: Lemma 2.9 sums inequalities with the time-varying M_k norm as though M_k were fixed, and the skeptic's 1D quadratic example suggests the claimed rate is false as stated. That is a correctness risk, not a reduction-by-construction, so it does not by itself raise the circularity score. The genuine circular step is in the regularized-Newton claim of Theorem 3.3, where the coefficients are recursively defined so that the AIM update algebraically equals (θ_k I + H_k)^{-1}g_k; the equivalence is true by construction. The paper's self-citations (He–Liao step-size strategy in Algorithm A.1, DRSOM as a numerical baseline) are not load-bearing for the central convergence claims. Overall, the paper has one headline secondary result that reduces to its own construction, while the main O(1/k) claim retains independent mathematical content, giving a partial-circularity score of 5.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The core O(1/k) proof relies on convexity, local Lipschitz differentiability, and an Armijo-type condition; these are mild. The accelerated O(1/k^2) claim imports an idempotent-Hessian assumption or a characteristic-polynomial construction, which are restrictive and ad hoc. Fixed user parameters mu, eta, epsilon, and beta_0 supply the 'adaptive' mechanism's manual constants.

free parameters (5)
  • mu_k (momentum gain) = 0.75 in experiments
    User-chosen in Algorithm A.1; the theory only requires mu_k in [0,1). Despite the 'without manual tuning' claim, this value is fixed by hand.
  • eta (line-search parameter) = 0.9
    Condition (2.1b) and the step-size update depend on eta; set to 0.9 in Algorithm A.1.
  • epsilon (finite-difference step) = 1e-3
    Used to approximate the Hessian-gradient product in the Hessian-gradient inertia; fixed by hand in Algorithm A.1.
  • beta_0 (initial step size) = 1.0
    Initial step size in Algorithm A.1; subsequent beta_k are adapted by line search.
  • alpha_k (quasi-Newton scale) = not specified
    In (3.2), m_k = alpha_k y_{k-1} - s_{k-1}; only a lower bound (3.4) is given, with no selection rule supplied for the AIM QN experiments.
assumptions (6)
  • domain assumption Convexity and differentiability of f on R^n
    Assumed in problem (1.1) and used throughout Section 2; needed for inequalities (2.8) through (2.10).
  • domain assumption Local Lipschitz differentiability of grad f
    Used to justify the Armijo-type condition (2.1b) and Remark 2.2; the line search is assumed to terminate.
  • standard math Existence of an optimal solution x*
    Used in Lemmas 2.8 and 2.9 and Theorem 2.10; for coercive functions this follows, but the paper does not state a coercivity assumption.
  • domain assumption Secant positivity condition (3.3) for the quasi-Newton variant
    Requires (s_{k-1})^T y_{k-1} > 0, i.e., local strong convexity, to keep M_k positive definite; not needed for the general O(1/k) result.
  • ad hoc to paper Hessian condition H_k^2 = a_k H_k in Theorem 3.1
    Assumed to obtain algebraic equivalence with regularized Newton; restricts the result to Hessians proportional to idempotent matrices, e.g., rank-one Hessians.
  • domain assumption Hessian is 2L-Lipschitz for the regularized Newton rate
    Imported from Mishchenko [24]; required for the O(1/k^2) rate that AIM inherits only under the constructed equivalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Inertial Method." pith.science (2026). https://pith.science/paper/I32UDNKI

@misc{pith2026250515114,
  author       = {Pith},
  title        = {Pith review of: Adaptive Inertial Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I32UDNKI}},
  note         = {Machine review of arXiv:2505.15114}
}
read the original abstract

In this paper, we introduce the Adaptive Inertial Method (AIM), a novel framework for accelerated first-order methods through a customizable inertial term. We provide a rigorous convergence analysis establishing a global convergence rate of O(1/k) under mild conditions, requiring only convexity and local Lipschitz differentiability of the objective function. Our method enables adaptive parameter selection for the inertial term without manual tuning. Furthermore, we derive the particular form of the inertial term that transforms AIM into a new Quasi-Newton method. Notably, under specific circumstances, AIM coincides with the regularized Newton method, achieving an accelerated rate of O(1/k^2) without Hessian inversions. Through extensive numerical experiments, we demonstrate that AIM exhibits superior performance across diverse optimization problems, highlighting its practical effectiveness.

Figures

Figures reproduced from arXiv: 2505.15114 by the authors.

Figure 1
Figure 1. Choose mk = mk v and µk = 0.5 mk a ∇f(x k−1 ) ∇f(x k ) x k x k−1 x k+1 [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 3
Figure 3. Convergence behavior of AIM and selected benchmark algorithms for logistic [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Convergence behavior of AIM and selected benchmark algorithms for [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 28 canonical work pages

  1. [1]

    Attouch and J

    H. Attouch and J. Fadili , From the ravine method to the nesterov method and vice versa: a dynamical system perspective , SIAM Journal on Optimization, 32 (2022), pp. 2074--2101

    work page 2022

  2. [2]

    Attouch, X

    H. Attouch, X. Goudou, and P. Redont , The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system , Communications in Contemporary Mathematics, 2 (2000), pp. 1--34

    work page 2000

  3. [3]

    Aujol, C

    J.-F. Aujol, C. Dossal, and A. Rondepierre , Convergence rates of the heavy ball method for quasi-strongly convex optimization , SIAM Journal on Optimization, 32 (2022), pp. 1817--1842

    work page 2022

  4. [4]

    Aujol, C

    J.-F. Aujol, C. Dossal, and A. Rondepierre , Convergence rates of the heavy-ball method under the ojasiewicz property , Mathematical Programming, 198 (2023), pp. 195--254

    work page 2023

  5. [5]

    C. Bao, L. Chen, J. Li, and Z. Shen , Accelerated gradient methods with gradient restart: Global linear convergence , arXiv preprint arXiv:2401.07672, (2024)

    arXiv 2024

  6. [6]

    Beck and M

    A. Beck and M. Teboulle , A fast iterative shrinkage-thresholding algorithm for linear inverse problems , SIAM journal on imaging sciences, 2 (2009), pp. 183--202

    work page 2009

  7. [7]

    Betancourt, M

    M. Betancourt, M. I. Jordan, and A. C. Wilson , On symplectic optimization , arXiv preprint arXiv:1802.03653, (2018)

    arXiv 2018

  8. [8]

    C. G. Broyden , Quasi-newton methods and their application to function minimisation , Mathematics of Computation, 21 (1967), pp. 368--381

    work page 1967

Show all 35 references
  1. [9]

    Bubeck, Y

    S. Bubeck, Y. T. Lee, and M. Singh , A geometric alternative to nesterov's accelerated gradient descent , arXiv preprint arXiv:1506.08187, (2015)

    2015 arXiv

  2. [10]

    Carmon, J

    Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford , Accelerated methods for nonconvex optimization , SIAM Journal on Optimization, 28 (2018), pp. 1751--1772

    2018

  3. [11]

    Chang and C.-J

    C.-C. Chang and C.-J. Lin , Libsvm: a library for support vector machines , ACM transactions on intelligent systems and technology (TIST), 2 (2011), pp. 1--27

    2011

  4. [12]

    Chen , Smoothing methods for nonsmooth, nonconvex minimization , Mathematical programming, 134 (2012), pp

    X. Chen , Smoothing methods for nonsmooth, nonconvex minimization , Mathematical programming, 134 (2012), pp. 71--99

    2012

  5. [13]

    Drusvyatskiy, M

    D. Drusvyatskiy, M. Fazel, and S. Roy , An optimal first order method based on optimal quadratic averaging , SIAM Journal on Optimization, 28 (2018), pp. 251--271

    2018

  6. [14]

    Duchi, E

    J. Duchi, E. Hazan, and Y. Singer , Adaptive subgradient methods for online learning and stochastic optimization. , Journal of machine learning research, 12 (2011)

    2011

  7. [15]

    Farazmand , Multiscale analysis of accelerated gradient methods , SIAM Journal on Optimization, 30 (2020), pp

    M. Farazmand , Multiscale analysis of accelerated gradient methods , SIAM Journal on Optimization, 30 (2020), pp. 2337--2354

    2020

  8. [16]

    Fletcher , Practical methods of optimization , John Wiley & Sons, 2000

    R. Fletcher , Practical methods of optimization , John Wiley & Sons, 2000

    2000

  9. [17]

    Fran c a, J

    G. Fran c a, J. Sulam, D. Robinson, and R. Vidal , Conformal symplectic and relativistic optimization , Advances in Neural Information Processing Systems, 33 (2020), pp. 16916--16926

    2020

  10. [18]

    Ghadimi, H

    E. Ghadimi, H. R. Feyzmahdavian, and M. Johansson , Global convergence of the heavy-ball method for convex optimization , in 2015 European control conference (ECC), IEEE, 2015, pp. 310--315

    2015

  11. [19]

    He and L.-Z

    B. He and L.-Z. Liao , Improvements of some projection methods for monotone nonlinear variational inequalities , Journal of Optimization Theory and applications, 112 (2002), pp. 111--128

    2002

  12. [20]

    Jin and Q

    Q. Jin and Q. Huang , An adaptive heavy ball method for ill-posed inverse problems , SIAM Journal on Imaging Sciences, 17 (2024), pp. 2212--2241

    2024

  13. [21]

    C. Josz, L. Lai, and X. Li , Convergence of the momentum method for semialgebraic functions with locally lipschitz gradients , SIAM Journal on Optimization, 33 (2023), pp. 3012--3037

    2023

  14. [22]

    Kim and J

    D. Kim and J. A. Fessler , Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions , Journal of optimization theory and applications, 188 (2021), pp. 192--219

    2021

  15. [23]

    D. P. Kingma , Adam: A method for stochastic optimization , arXiv preprint arXiv:1412.6980, (2014)

    2014 arXiv

  16. [24]

    Mishchenko , Regularized newton method with global convergence , SIAM Journal on Optimization, 33 (2023), pp

    K. Mishchenko , Regularized newton method with global convergence , SIAM Journal on Optimization, 33 (2023), pp. 1440--1462

    2023

  17. [25]

    Nesterov , A method for solving the convex programming problem with convergence rate O(1/k^2) , in Dokl akad nauk Sssr, vol

    Y. Nesterov , A method for solving the convex programming problem with convergence rate O(1/k^2) , in Dokl akad nauk Sssr, vol. 269, 1983, p. 543

    1983

  18. [26]

    Nesterov , Introductory lectures on convex optimization: A basic course , vol

    Y. Nesterov , Introductory lectures on convex optimization: A basic course , vol. 87, Springer Science & Business Media, 2013

    2013

  19. [27]

    B. T. Polyak , Some methods of speeding up the convergence of iteration methods , Ussr computational mathematics and mathematical physics, 4 (1964), pp. 1--17

    1964

  20. [28]

    Saab Jr, S

    S. Saab Jr, S. Phoha, M. Zhu, and A. Ray , An adaptive polyak heavy-ball method , Machine Learning, 111 (2022), pp. 3245--3277

    2022

  21. [29]

    T. Sun, P. Yin, D. Li, C. Huang, L. Guan, and H. Jiang , Non-ergodic convergence analysis of heavy-ball algorithms , in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 5033--5040

    2019

  22. [30]

    Wibisono, A

    A. Wibisono, A. C. Wilson, and M. I. Jordan , A variational perspective on accelerated methods in optimization , proceedings of the National Academy of Sciences, 113 (2016), pp. E7351--E7358

    2016

  23. [31]

    X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan , Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models , IEEE Transactions on Pattern Analysis and Machine Intelligence, (2024)

    2024

  24. [32]

    Z. Xie, X. Wang, H. Zhang, I. Sato, and M. Sugiyama , Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum , in International conference on machine learning, PMLR, 2022, pp. 24430--24459

    2022

  25. [33]

    Zhang, D

    C. Zhang, D. Ge, C. He, B. Jiang, Y. Jiang, and Y. Ye , Drsom: A dimension reduced second-order method , arXiv preprint arXiv:2208.00208, (2022)

    2022 arXiv

  26. [34]

    Zhang, C

    Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. P. Kingma, Y. Ye, Z.-Q. Luo, and R. Sun , Adam-mini: Use fewer learning rates to gain more , arXiv preprint arXiv:2406.16793, (2024)

    2024 arXiv

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTION or pop #1 'skip if FUNCTION new.block.check...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.