Pith. sign in

REVIEW 3 major objections 5 minor 69 references

Complexities of Armijo-like algorithms in Deep Learning context

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a memory-backtracking Armijo algorithm reaches optimal worst-case rates on deep-learning-relevant smoothness classes, and that an explicit-implicit variant reaches the optimal $\varepsilon^{-1}$ complexity on…

desk verdict Main complexity theorem for memory Armijo rests on a false step-size lower bound; Section 3 leans on an unpublished preprint, but the ideas are promising. read the letter →

arxiv 2412.14637 v1 pith:J2SVTPZP submitted 2024-12-19 math.OC

Bensaid Bilel (IMB) This is my paper · ORCID
classification math.OC MSC 65K1090C2665Y2026E05
keywords Armijobacktrackinggeneralizedsmoothness(L0L1)-smoothnessanalyticfunctionsLojasiewiczinequalitynonconvexoptimizationcomplexityparameter-freedeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Armijo-style line-search algorithms, which require no learning-rate tuning, achieve optimal worst-case complexity under assumptions tailored to deep-learning losses. For an $(L_0,L_1)$-smooth objective, meaning the Hessian norm is bounded by $L_0$ plus $L_1$ times the gradient norm, the memory-backtracking Armijo variant reaches a gradient norm below $\varepsilon$ in at most $f_1 L_1 \Delta/(\lambda \varepsilon \ln((L_0+L_1(2-\lambda)\varepsilon)/(L_0+L_1\varepsilon)))$ iterations, which behaves like $8L_0\Delta/\varepsilon^2$ at the practical parameter values $f_1=2$, $\lambda=1/2$. That matches the acceleration of clipping gradient descent while removing the need to tune the step size and clipping threshold. For analytic radially unbounded objectives, the explicit-implicit Armijo algorithm reaches $\varepsilon$-stationarity in $O(\varepsilon^{-1})$ iterations, the optimal $\varepsilon$-dependence, and its complexity changes from linear in the initial gap $\Delta$ to roughly $\sqrt{\Delta}$ once $\Delta$ is small relative to the gaps between critical values. The practical upshot is that a parameter-free first-order method can theoretically match the best tuned optimizers on nonconvex problems relevant to deep learning.

What carries the argument

The argument runs through two mechanisms. For generalized smoothness, Lemma 2.2 replaces the standard quadratic upper model of smooth optimization with an exponential bound coming from a Gronwall estimate on the gradient, and Lemma 2.3 solves the resulting Lambert-type inequality to give an explicit step-size formula $\tilde\eta_n = \ln((L_0+L_1(2-\lambda)\|\nabla R(\theta_n)\|)/(L_0+L_1\|\nabla R(\theta_n)\|))/(L_1\|\nabla R(\theta_n)\|)$. The backtracking must accept a step at least $\tilde\eta_n/f_1$, so each iteration decreases the function by at least $\lambda \tilde\eta_n \|\nabla R(\theta_n)\|^2/f_1$, and monotonicity of the logarithm allows replacing the per-iteration gradient norm by $\varepsilon$ and telescoping the gaps. For analytic functions, the explicit-implicit Armijo condition $R(\theta_{n+1})-R(\theta_n) \le -\lambda \eta_n \|\nabla R(\theta_n)\|\,\|\nabla R(\theta_{n+1})\|$ couples the step to the next gradient; a global Lojasiewicz inequality (Lemma 3.2) built from the finitely many critical values of the analytic function provides $\varphi'(\tilde R(\theta))\|\nabla R(\theta)\| \ge 1$ on compact sets, and the interval $[R(\theta_n), R(\theta_0)]$ is cut at critical values and their midpoints so that iterate-length sums telescope on each critical-value-free piece. The lower bound $\eta_* = 2(1-\lambda)/(f_1(1+2\lambda)L)$, with $L$ the local Lipschitz constant of $\nabla R$, converts the path-length bound into an iteration count.

What would settle it

Run Algorithm 1 with a large initial step $\eta_{\mathrm{init}}$ on a one-dimensional $(L_0,L_1)$-smooth function with $L_1>0$ and record, for each iteration, whether the accepted step is below $\tilde\eta_n/f_1$ when the first trial is already admissible. If such an iteration occurs, the per-iteration decrease can fall below $\lambda \tilde\eta_n \|\nabla R(\theta_n)\|^2/f_1$, breaking the telescoping sum that produces Theorem 2.1; the claim would then be settled by exhibiting the corresponding iteration count exceeding $8L_0\Delta/\varepsilon^2$.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 2.1: for a twice-differentiable, lower-bounded function satisfying $\|\nabla^2R(\theta)\| \le L_0 + L_1\|\nabla R(\theta)\|$, the memory-backtracking Armijo algorithm guarantees $\min_{0\le k\le n} \|\nabla R(\theta_k)\| \le \varepsilon$ as soon as $n \ge f_1 L_1 \Delta/(\lambda \varepsilon \ln((L_0+L_1(2-\lambda)\varepsilon)/(L_0+L_1\varepsilon)))$, with asymptotic form $f_1 L_0 \Delta/(\lambda(1-\lambda)\varepsilon^2)$ as $\varepsilon\to 0$ and $8L_0\Delta/\varepsilon^2$ for $f_1=2$, $\lambda=1/2$. Unlike plain gradient descent under this assumption, the bound contains no dependence on $M=\sup_S\|\nabla R\|$, so the algorithm has the same form of acceleration as clipping gradient descent, but without any tuned step size or clipping threshold. The second central claim, Corollary 3.1, says that for analytic radially unbounded functions, the explicit-implicit Armijo algorithm reaches $\min \|\nabla R\| \le \varepsilon$ in $O(\varepsilon^{-1})$ iterations, which is the optimal $\varepsilon$-dependence for first-order methods and matches the best known rate for smooth convex optimization despite the objective being nonconvex. The proof also produces a path-length bound that makes the dependence on the initial gap $\Delta$ conditional on $\Delta/(s+1)$, where $s$ is the number of critical values, yielding a phase transition: for small gaps the complexity depends on $\Delta^\alpha$ with $\alpha\le 1/2$ rather than on $\Delta$.

Load-bearing premise

The load-bearing premise is that every accepted backtracking step is no smaller than a fixed fraction $1/f_1$ of the explicit admissible step $\tilde\eta_n$ derived from the smoothness constants; the memory variant can exit its inner loop on the first trial step, before any division, so that inequality is not guaranteed, and the Section 3 lower bound $\eta_*$ has the same vulnerability.

Editorial extensions

If this is right

  • On $(L_0,L_1)$-smooth objectives, the memory Armijo algorithm reaches $\varepsilon$-stationarity in at most $8L_0\Delta/\varepsilon^2$ iterations at the standard parameters, independent of the global gradient bound $M$ and without tuning.
  • Counting backtracking function evaluations, the total gradient evaluations for Algorithm 1 stay within a factor of the iteration bound stated in the paper, roughly $4[1+\log(f_2)/\log(f_1)]L_0\Delta/\varepsilon^2 + 8L_0\Delta/\varepsilon^2$, so the improvement over clipping gradient descent is not erased by the line search.
  • For analytic radially unbounded functions, the explicit-implicit Armijo algorithm attains $O(\varepsilon^{-1})$ iteration complexity, matching the optimal first-order rate for smooth convex functions despite nonconvexity.
  • The same analytic-function analysis gives $\min_{0\le k\le n-1}\|\nabla R(\theta_k)\| = o(n^{-1})$, the same asymptotic speed as analyses of o-minimal or definable functions, but without requiring any tuned step size.
  • The complexity exhibits a bifurcation in the initial gap $\Delta$: when $\Delta/(s+1)$ is below a threshold set by the critical-value gaps, the dependence is $\Delta^\alpha$ with $\alpha\le 1/2$, which becomes $\sqrt{\Delta}$ for Morse functions, instead of linear $\Delta$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the $\Delta$-bifurcation is real, worst-case complexity for analytic nonconvex problems can improve by a square root simply because the initial gap is small relative to critical-value spacing; that is a qualitative prediction about optimization landscape geometry that could be tested on synthetic analytic functions with known critical values.
  • The explicit-implicit condition is a semi-implicit discretization of the Lyapunov inequality for gradient flow; the same stabilization idea could be applied to stochastic or Adam-like updates, and whether the $O(\varepsilon^{-1})$ rate survives noise is a natural open extension the paper does not address.
  • A practical testable extension is to compare Algorithm 1 against tuned clipping gradient descent and Adam on a small neural-network training task in terms of wall-clock gradient evaluations; the paper's bounds predict the Armijo variant should match clipping's acceleration per gradient evaluation.
  • The reliance on finite critical values and a global Lojasiewicz function suggests the analytic-rate result may extend to any function whose critical values are finite on compact sets and whose gradients satisfy a global Lojasiewicz inequality, even outside analyticity; the paper notes this structural possibility but does not pursue it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript studies two Armijo-type line-search methods in the deep-learning-relevant settings of (L0,L1)-smoothness and analyticity. Section 2 analyses a 'memory' backtracking Armijo variant (Algorithm 1) and claims in Theorem 2.1 that, after n ≥ f1 L1 Δ / (λ ε ln((L0+L1(2-λ)ε)/(L0+L1 ε))) iterations, the iterates satisfy min_{0≤k≤n} ||∇R(θ_k)|| ≤ ε; this is asymptotic to 8 L0 Δ / ε² for the practical choice f1=2, λ=1/2, matching clipping gradient descent without hyperparameter tuning. Section 3 introduces an Explicit-Implicit Armijo (EIA) method, derives a global Łojasiewicz inequality, and claims O(ε^{-1}) complexity for analytic radially unbounded functions (Corollary 3.1) together with o(n^{-1}) decay of the minimal gradient (Corollary 3.2). The paper also compares total evaluation counts with clipping GD, using estimates from the author's own thesis and preprints.

Significance. If the results were correct, they would be significant: they would show that Armijo-like line search can retain the parameter-free character of classical backtracking while achieving clipping-GD-like acceleration under generalized smoothness and an optimal ε-dependence for analytic functions. The paper has some strengths: explicit constants in the main bound, a Lambert-type sufficient step-size argument, and an attempt to account for function evaluations. However, the central theorem rests on a false invariant about accepted step sizes, and Section 3 inherits the same structural problem while also relying on load-bearing results from an unpublished preprint. The advertised claims are therefore not established in the present form.

major comments (3)
  1. [§2.3, Theorem 2.1] The proof uses the assertion 'At iteration n, the time step found by the backtracking satisfies η_n ≥ \tilde η_n / f1'. In Algorithm 1 the trial step η is carried over from the previous outer iteration and is multiplied by f2 only after an accepted step (line 15). If the carried η already satisfies the Armijo condition, the repeat-until loop exits immediately without any division by f1, so η_n can be arbitrarily small compared with \tilde η_n. In particular, the lower bound η_n ≥ \tilde η_n / f1 is not valid without an additional assumption such as η_init ≥ \tilde η_0 / f1 or a reset of the trial step at each iteration. Choosing η_init sufficiently small (for example, exponentially small in 1/ε on a strongly convex quadratic) can keep the step size tiny for many more than O(1/ε²) iterations, so the gradient norm need not reach ε within the claimed bound. Thus Theorem 2.1 is false as stated, and the telescoping argument in §2.3 does not establish the advertised clipping-GD-like acceleration.
  2. [§3.2] The lower bound η_n ≥ η* = 2(1-λ)/(f1(1+2λ)L) is asserted by combining Lemma 3.1 with Proposition 1 of [11]. Lemma 3.1 provides only a sufficient upper bound on η for inequality (8); it does not imply that the step produced by Algorithm 1's memory backtracking is bounded below. A carried-over η that already satisfies (2) and (8) is accepted without division, so the same flaw as in §2.3 occurs. Moreover, the convergence of EIA and the step lower bound are imported from Proposition 1 and Theorem 6 of the author's preprint [11] without statement or proof. Since these results are load-bearing for Corollary 3.1, the paper is not self-contained; the authors should either prove these results or formulate them explicitly in the manuscript.
  3. [§3.6, Remark 3.4] The claim that EIA achieves the optimal O(ε^{-1}) complexity on analytic functions is not established. The bound in Corollary 3.1 involves constants \tilde s, L, M, and φ that depend on the whole trajectory K0 and on the Łojasiewicz data of the specific function; it is not a uniform complexity over a class of functions with fixed smoothness and initial gap. Nesterov's lower bound cited in Remark 3.4 is for smooth convex functions and does not directly yield a lower bound for the class of general nonconvex analytic functions considered here. A precise statement of the function class and a matching lower bound are needed before concluding optimality.
minor comments (5)
  1. [§2.2, Lemma 2.2] The displayed inequality in Lemma 2.2 has an erroneous coefficient: the term multiplying ||y2-y1|| should be (L0 + L1||∇R(y1)||)/L1, not (L0 + L1||∇R(y1)||). The integral in the proof and the subsequent use in §2.3 are consistent with the corrected version, so this appears to be a typo, but it should be fixed.
  2. [§2.3, Theorem 2.1] The statement writes 'min_{0≤k≤n} ∇R(θ_k) ≤ ε' where clearly the norm ||∇R(θ_k)|| is intended; there is also a typo 'tyhat' for 'that'. Please correct these.
  3. [§3.4, Lemma 3.3] In the proof of Lemma 3.3, case 1, equation (14) states \tilde R(θ_k) = R_i - R(θ_k), but for an interval of the form ](R_i+R_{i+1})/2, R_{i+1}[ the correct identity is \tilde R(θ_k) = R_{i+1} - R(θ_k). The argument goes through with the corrected index, but the displayed equation should be amended.
  4. [§3.4, Lemma 3.4] The definitions of r0 and r1 in Parts 2 and 3 appear to have an inverted factor: the expressions (1-d)/(2 λ η_min ε²) should likely be (1-d) λ η_min ε² / 2. Although the proof only needs some positive radius, the displayed formulas should be corrected for clarity.
  5. [§2.3, evaluation count] The upper bound on function evaluations per iteration is quoted from the author's own thesis [10] and preprint [12] with no statement or proof. Since this bound is used to compare the total cost with clipping GD, it should be stated precisely and either proved or clearly identified as an external heuristic.

Circularity Check

2 steps flagged · score 4.0 of 10

Section 3's O(ε^{-1}) complexity is made to depend on the author's own preprint [11] for the step-size lower bound and convergence, and the Section 2 evaluation-count comparison rests on self-cited counts from [10,12]; the main Section 2 rate also contains a non-circular proof gap.

  1. self citation load bearing [Section 3.2, paragraph following Lemma 3.1 (p. 12)]
    "Combining the previous inequality with proposition 1 of [11] and since 1+2λ ≥ 1, the admissible time steps are lower bounded by: ∀n ∈ N, η n ≥ 2(1 − λ)/f1(1 + 2λ) 1/L := η∗. Using theorem 6 of [11], the sequence ( θn)n∈N generated by EIA converges to a critical point of R."

    Corollary 3.1 converts the path-length bound of Theorem 3.1 into an O(ε^{-1}) iteration bound only by replacing min_k η_k with the global lower bound η∗. The only support given for η_k ≥ η∗ is 'proposition 1 of [11]', an arXiv preprint whose first author is the present author and whose proposition is not stated or proved in this paper. The convergence assertion also uses 'theorem 6 of [11]'. Since the step-size lower bound is the device that turns a sum of step lengths into an iteration count, the central Section 3 complexity result is not self-contained: it reduces to a self-cited, unverified result rather than to an in-paper derivation.

  2. self citation load bearing [Section 2.3, paragraph after Theorem 2.1 (p. 10)]
    "In [12, 10] (chapter2, p55-59), an upper bound on the number of evaluations per iteration is given. It is stated that for neural networks, the number of function evaluations per iteration is equivalent to: 1/2 [1 + log(f2)/log(f1)] gradient evaluations, for algorithm 1. Therefore, taking into account the backtracking (that is to say the function evaluations), the real number of gradient evaluations for these algorithms, for f1 = 2, is given by: 4 [1 + log(f2)/log2] L0Δ/ε2 + 8L0Δ/ε2. Compared with theorem 1.3, the multiplicative constant in our bound is better for small ε for f2 ≤ 24.5 ≈ 2^3."

    The conclusion advertises that the method 'improves the complexity of clipping GD even in terms of gradient/function evaluations'. That improvement is computed from a per-iteration evaluation count that is imported from [12] and from the author's own thesis [10] (chapter 2, pp. 55–59), with no proof or independent check in the present paper. Thus the quantitative comparison with Theorem 1.3 is inherited from self-cited estimates; the comparative claim is load-bearing and not established independently here.

full rationale

The core derivation chain in Section 2 (Lemmas 2.1–2.3 and Theorem 2.1) is carried out in the paper: the generalized-smoothness descent inequality and the Lambert-type step threshold are derived and the iteration bound follows from them. The Section 2 iteration-complexity theorem therefore has independent in-paper content and is not circular. However, two claims that the paper presents as contributions are supported only by the author's own prior work. In Section 3.2, the lower bound η∗ used by Corollary 3.1 and the convergence of EIA are imported from 'proposition 1' and 'theorem 6' of the same-author preprint [11]; absent an in-paper proof or independent external verification, the O(ε^{-1}) result leans on a self-citation chain. In Section 2.3, the function-evaluation count that yields the smaller constant than clipping GD comes from [12] and from the author's thesis [10], making the comparison claim another load-bearing self-citation. Separately, and not counted as circularity, the proof of Theorem 2.1 contains a formal gap: the sentence 'At iteration n, the time step found by the backtracking satisfies ηn ≥ ηtilde_n/f1' is not guaranteed by Algorithm 1, whose memory step can already satisfy the Armijo condition and be accepted without division by f1, so the accepted step may be arbitrarily small and the telescoping lower bound is not justified. This is a correctness issue rather than a reduction of a prediction to its inputs. Overall score 4: some load-bearing self-citation, but the central Section 2 rate and the analytic Lojasiewicz machinery still have independent content.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No physical entities are invented. The only free quantities are the algorithm hyperparameters lambda, f1, f2 and the analytic-Lojasiewicz constants inherited from the function; none are fitted to data. The main hidden assumption is the step-size lower bound, listed as ad hoc.

free parameters (3)
  • lambda = typically 1/2
    Armijo parameter in (0,1); the complexity bound's constant depends on it, but it is an algorithm hyperparameter, not fitted to data.
  • f1 = typically 2
    Backtracking division factor; the bound scales as f1. It is an algorithm hyperparameter, not fitted to data.
  • f2 = user-specified, >1
    Memory growth factor; enters the evaluation-count constant. It is an algorithm hyperparameter, not fitted to data.
assumptions (5)
  • domain assumption R is (L0,L1)-smooth and C^2
    Definition 1.1; the entire Section 2 assumes this condition, replacing classical Lipschitz smoothness.
  • domain assumption R is analytic and radially unbounded
    Section 3.2 states this assumption; it ensures finite critical values on compacts via Morse-Sard and local Lojasiewicz inequalities.
  • ad hoc to paper Accepted backtracks satisfy eta_n >= eta_tilde_n / f1 (and eta_n >= eta* in Section 3)
    Asserted in Section 2.3; not implied by Algorithm 1 when the trial step is already admissible, so the proof needs an extra initialization or warm-up assumption.
  • ad hoc to paper Proposition 1 and Theorem 6 of [11] on convergence and admissible step lower bounds for EIA
    Section 3.2 imports convergence and the step-size lower bound from the author's own preprint [11]; these are not proved or machine-checked here.
  • standard math Morse-Sard theorem for analytic functions and local Lojasiewicz estimates from [14]
    Used in Lemma 3.2 to construct the global desingularization function phi on the compact set K.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Complexities of Armijo-like algorithms in Deep Learning context." pith.science (2026). https://pith.science/paper/J2SVTPZP

@misc{pith2026241214637,
  author       = {Pith},
  title        = {Pith review of: Complexities of Armijo-like algorithms in Deep Learning context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2SVTPZP}},
  note         = {Machine review of arXiv:2412.14637}
}
read the original abstract

The classical Armijo backtracking algorithm achieves the optimal complexity for smooth functions like gradient descent but without any hyperparameter tuning. However, the smoothness assumption is not suitable for Deep Learning optimization. In this work, we show that some variants of the Armijo optimizer achieves acceleration and optimal complexities under assumptions more suited for Deep Learning: the (L 0 , L 1 ) smoothness condition and analyticity. New dependences on the smoothness constants and the initial gap are established. The results theoretically highlight the powerful efficiency of Armijo-like conditions for highly non-convex problems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 50 canonical work pages

  1. [11]

    Arxiv (2024)

    Bensaid, B., Poette, G., Turpault, R.: An Abstract Lyapu nov Control Optimizer: Local Stabilization and Global Convergence. Arxiv (2024)

    work page 2024

  2. [10]

    Theses, Universit´ e de Bordeaux (2024)

    Bensaid, B.: Analyse et D´ eveloppement de Nouveaux Opti miseurs en Machine Learning. Theses, Universit´ e de Bordeaux (2024). URL https://theses.hal.science/tel-04808839

    work page 2024

  3. [1]

    Stochastic Polyak Step-Size for SGD: An Adaptive Learnin g Rate for Fast Convergence (2020)

    work page 2020

  4. [2]

    SIAM Journal on Optimization 16(2), 531–547 (2005)

    Absil, P., Mahony, R., Andrews, B.: Convergence of the Ite rates of Descent Methods for Analytic Cost Functions. SIAM Journal on Optimization 16(2), 531–547 (2005). DOI 10.1137/040605266. URL https://doi.org/10.1137/040605266

    doi:10.1137/040605266 2005

  5. [3]

    DOI https://doi.org/10.1007/s10898-022-01164 -w

    Apidopoulos, V., Ginatta, N., Villa, S.: Convergence Rat es for the Heavy-Ball Continu- ous Dynamics for Non-Convex Optimization, under Polyak–/suppress Lojasiewicz Condition 84, 563–589. DOI https://doi.org/10.1007/s10898-022-01164 -w

    work page doi:10.1007/s10898-022-01164

  6. [4]

    Pacific Journal of Mathematics 16(1), 1 – 3 (1966)

    Armijo, L.: Minimization of Functions Having Lipschitz Continuous First Partial Deriva- tives. Pacific Journal of Mathematics 16(1), 1 – 3 (1966)

    work page 1966

  7. [5]

    Mathematics of Operations Research 35(2), 438–457 (2010)

    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proxim al Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka- /suppress Lojasiewicz Inequality. Mathematics of Operations Research 35(2), 438–457 (2010). DOI 10.1287/moor.1100.0449. URL https://doi.org/10.1287/moor.1100.0449

    arXiv 2010

  8. [6]

    Mathematical Programming 137, 36 (2011)

    Attouch, H., Bolte, J., Svaiter, B.: Convergence of Desce nt Methods for Semi-Algebraic and Tame Problems: Proximal Algorithms, Forward-Backward Splitting, and Regu- larized Gauss-Seidel Methods. Mathematical Programming 137, 36 (2011). DOI 10.1007/s10107-011-0484-9

    doi:10.1007/s10107-011-0484-9 2011

Show all 69 references
  1. [7]

    SIAM Journal on Optimization 26 (2015)

    Attouch, H., Peypouquet, J.: The Rate of Convergence of Ne sterov´ s Accelerated Forward-Backward Method is actually o(k− 2). SIAM Journal on Optimization 26 (2015). DOI 10.1137/15M1046095

    2015 doi

  2. [8]

    In: Advances in Neural In- formation Processing Systems, vol

    Auer, P., Herbster, M., Warmuth, M.: Exponentially Many L o- cal Minima for Single Neurons. In: Advances in Neural In- formation Processing Systems, vol. 8. MIT Press (1996). URL https://proceedings.neurips.cc/paper/1995/file/3806734b256c27e41ec2c6bffa26d9e7-Paper.pdf

    1996

  3. [9]

    Mathematical Programming198(1), 195–254 (2022)

    Aujol, J., Dossal, C., Rondepierre, A.: Convergence Rate s of the Heavy-Ball Method un- der the Lojasiewicz Property. Mathematical Programming198(1), 195–254 (2022). DOI 10.1007/s10107-022-01770-2. URL https://doi.org/10.1007/s10107-022-01770-2

    2022 doi

  4. [12]

    In: Arxiv (2024)

    Bensaid, B., Poette, G., Turpault, R.: Convergence of th e Iterates for Momentum and RMSProp for Local Lipshitz Functions: Adaptation is the Key . In: Arxiv (2024)

    2024

  5. [13]

    Society for Industrial and Applied Mathematics 17, 1205–1223 (2007)

    Bolte, J., Daniilidis, A., Lewis, A.: The /suppress Lojasiewicz Inequality for Nonsmooth Sub- analytic Functions with Applications to Subgradient Dynam ical Systems. Society for Industrial and Applied Mathematics 17, 1205–1223 (2007). DOI 10.1137/050644641

    2007 doi

  6. [14]

    SIAM Journal on Optimization 18(2), 556–572 (2007)

    Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradients of stratifiable func- tions. SIAM Journal on Optimization 18(2), 556–572 (2007). DOI 10.1137/060670080. URL https://doi.org/10.1137/060670080

    2007 doi

  7. [15]

    lecture no tes of EE392o, Stanford Uni- versity, Autumn Quarter 2004 (2003)

    Boyd, S., Mutapcic, A.: Subgradient Methods. lecture no tes of EE392o, Stanford Uni- versity, Autumn Quarter 2004 (2003)

    2003

  8. [16]

    DOI https://doi.org/10.1007/s10107-019 -01406-y

    Carmon, Y., Duchi, J., Hinder, O., al.: Lower Bounds for F inding Stationary Points I 184, 71–120 (2020). DOI https://doi.org/10.1007/s10107-019 -01406-y

    2020 doi

  9. [17]

    In: In- ternational Conference on Machine Learning (ICML) (2017)

    Carmon, Y., Duchi, J., Hinder, O., Sidford, A.: ”Convex U ntil Proven Guilty”: Dimension-Free Acceleration of Gradient Descent on Non-Co nvex Functions. In: In- ternational Conference on Machine Learning (ICML) (2017)

    2017

  10. [18]

    SIAM Journal on Optimization 28(2), 1751–1772 (2018)

    Carmon, Y., Duchi, J., Hinder, O., Sidford, A.: Accelera ted Methods for NonConvex Optimization. SIAM Journal on Optimization 28(2), 1751–1772 (2018). DOI 10.1137/ 17M1114296

    2018

  11. [19]

    : Robustness to Un- bounded Smoothness of Generalized SignSGD

    Crawshaw, M., Liu, M., Orabona, F., Zhang, W., Zhuang, Z. : Robustness to Un- bounded Smoothness of Generalized SignSGD. In: Neural Info rmation Processing Sys- tems (NeurIPS) (2022)

    2022

  12. [20]

    In: International Conference on Learning Represen tations (2024)

    E.Gorbunov, N.Tupitsa, S.Choudhury, Aliev, A., P.Rich t´ arik, S.Horv´ ath, M.Tak´ aˇ c: Methods for convex $(l 0,l 1)$-smooth optimization: Clipping, acceleration, and adap- tivity. In: International Conference on Learning Represen tations (2024). URL https://openreview.net/f...

    2024

  13. [21]

    In: Annual Conference on L earning Theory, vol

    Faw, M., Root, L., Caramanis, C., Shakkottai, S.: Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD. In: Annual Conference on L earning Theory, vol. 195, pp. 1–72 (2023)

    2023

  14. [22]

    In: Proceedings of the 34th International Conference on Machine Learning, ICML’17, vol

    Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin , Y.: Convolutional Sequence to Sequence Learning. In: Proceedings of the 34th International Conference on Machine Learning, ICML’17, vol. 70, p. 1243–1252. JMLR.org (2017)

    2017

  15. [23]

    Herrera, C., Krach, F., Teichmann, J.: Local Lipschitz Bounds of Deep Neural Networks (2023)

    2023

  16. [24]

    In: International Conference on Neural Information Processin g Systems, NIPS ’23

    H.Li, A.Rakhlin, A.Jadbabaie: Convergence of adam unde r relaxed assumptions. In: International Conference on Neural Information Processin g Systems, NIPS ’23. Curran Associates Inc., Red Hook, NY, USA (2024)

    2024

  17. [25]

    In: International Conference on Neural Information Pro- cessing Systems, NIPS ’23

    H.Li, J.Qian, Y.Tian, A.Rakhlin, A.Jadbabaie: Convex a nd non-convex optimization under generalized smoothness. In: International Conference on Neural Information Pro- cessing Systems, NIPS ’23. Curran Associates Inc., Red Hook , NY, USA (2024)

    2024

  18. [26]

    Zhang and T

    J. Zhang and T. He and S. Sra and A. Jadbabaie: Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. In: I nternational Conference on Learning Representations (2020). URLhttps://openreview.net/forum?id=BJgnXpVYwS

    2020

  19. [27]

    J ournal of Inequalities and Applications 2019, 16 (2019)

    Jia, Z., Wu, Z., Dong, X.: An Inexact Proximal Gradient Al gorithm with Extrapolation for a Class of Nonconvex Nonsmooth Optimization Problems. J ournal of Inequalities and Applications 2019, 16 (2019). DOI 10.1186/s13660-019-2078-7

    2019 doi

  20. [28]

    Mathematical Programming 202, 385 (2023)

    Josz, C.: Global convergence of the gradient method for f unctions definable in o- minimal structures. Mathematical Programming 202, 385 (2023). DOI 10.1007/ s10107-023-01972-2. URL https://doi.org/10.1007/s10107-023-01972-2

    2023 doi

  21. [29]

    SIAM Journal on Optimization 33(4), 3012–3037 (2023)

    Josz, C., Lai, L., Li, X.: Convergence of the Momentum Method for Semialgebraic Func- tions with Locally Lipschitz Gradients. SIAM Journal on Optimization 33(4), 3012–3037 (2023). DOI 10.1137/23M1545720. URL https://doi.org/10.1137/23M1545720

    2023 doi

  22. [30]

    In: International Conference on Learning Representations (ICLR) (2014)

    Kingma, D., Ba, J.: Adam: A Method for Stochastic Optimiz ation. In: International Conference on Learning Representations (ICLR) (2014). DOI 10.48550/ARXIV.1412. 6980

    2014 doi

  23. [31]

    Physics of Plasmas 27(5), 052,707 (2020)

    Kluth, G., Humbird, K., Spears, B., Peterson, J., Scott, H., Patel, M., Koning, J., Marinak, M., Divol, L., Young, C.: Deep Learning for NLTE Spectral Opacities. Physics of Plasmas 27(5), 052,707 (2020). DOI 10.1063/5.0006784

    2020 doi

  24. [32]

    An- nales de l’Institut Fourier 48(3), 769–783 (1998)

    Kurdyka, K.: On Gradients of Functions Definable in o-Min imal Structures. An- nales de l’Institut Fourier 48(3), 769–783 (1998). DOI 10.5802/aif.1638. URL https://aif.centre-mersenne.org/articles/10.5802/aif.1638/

    1998 doi

  25. [33]

    Lamy, C., Dubroca, B., Nicola¨ ı, P., Tikhonchuk, V., Feu geas, J.: Modeling of Electron Nonlocal Transport in Plasmas using Artificial N eural Networks. Phys. Rev. E 105, 055,201 (2022). DOI 10.1103/PhysRevE.105.055201. URL https://link.aps.org/doi/10.1103/PhysRevE.105.055201

    2022 doi

  26. [34]

    In: Proceedings of the 39th International Conference on Machine Learning, Proceedings of Ma- chine Learning Research , vol

    Li, H., Lin, Z.: Restarted Nonconvex Accelerated Gradie nt Descent: No More Polylogarithmic Factor in the O(ε− 7/ 4) Complexity. In: Proceedings of the 39th International Conference on Machine Learning, Proceedings of Ma- chine Learning Research , vol. 162, pp. 12,901–12,916. ...

    2022

  27. [35]

    In: Thirty-Seventh Conference on Neural Information Processing Systems (2023)

    Li, H., Rakhlin, A., Jadbabaie, A.: Convergence of Adam U nder Relaxed Assumptions. In: Thirty-Seventh Conference on Neural Information Processing Systems (2023). URL https://openreview.net/forum?id=yEewbkBNzi

    2023

  28. [36]

    Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Han, J.: On the Variance of the Adaptive Learning Rate and Beyond. p. 13. ICLR (2020). DOI 10 .48550/ARXIV.1908. 03265

    2020

  29. [37]

    In: Int ernational Conference on Artificial Intelligence and Statistics, pp

    Loizou, N., Vaswani, S., Laradji, I., Lacoste-Julien, S .: Stochastic Polyak Step-Size for SGD: An Adaptive Learning Rate for Fast Convergence. In: Int ernational Conference on Artificial Intelligence and Statistics, pp. 1306–1314. P MLR (2021)

    2021

  30. [38]

    /suppress Lojasiewicz, S.: A Topological Property of Real Analytic Subsets (1963)

    1963

  31. [39]

    Seminari di Geometria

    /suppress Lojasiewicz, S.: Trajectories of the Gradient of an Analytic Function. Seminari di Geometria. Universit´ a degli Studi di Bologna1982/1983, 115–117 (1984)

    1984

  32. [40]

    Annales de l’Institut Fourier 43(5), 1575–1595 (1993)

    /suppress Lojasiewicz, S.: On Semi- and Subanalytic Geometry. Annales de l’Institut Fourier 43(5), 1575–1595 (1993). DOI https://doi.org/10.1007/s109 15-023-02215-4 28 Bensaid Bilel

    1993 doi

  33. [41]

    Inter- national journal of information and computing science V olume 6, 566–568 (2019)

    Lydia, A., Francis, S.: Adagrad - An Optimizer for Stocha stic Gradient Descent. Inter- national journal of information and computing science V olume 6, 566–568 (2019)

    2019

  34. [42]

    Marumo, N., Takeda, A.: Parameter-Free Accelerated Gra dient Descent for Nonconvex Minimization (2024)

    2024

  35. [43]

    Marumo, N., Takeda, A.: Universal Heavy-Ball Method for Nonconvex Optimization under H¨ older Continuous Hessians (2024)

    2024

  36. [44]

    Merity, S., Keskar, N., Socher, R.: Regularizing and Optimizing LSTM Language Models

  37. [45]

    Journal of Physics: Conference Series 1237, 022,026 (2019)

    Mo, W., Luo, X., Zhong, Y., Jiang, W.: Image Recognition U sing Convolutional Neural Network Combined with Ensemble Learning Algorithm. Journal of Physics: Conference Series 1237, 022,026 (2019). DOI 10.1088/1742-6596/1237/2/022026

    2019 doi

  38. [46]

    DOI https://doi.org/10.1007/BF0259294 8

    Murty, K., Kabadi, S.: Some NP-Complete Problems in Quad ratic and Nonlinear Pro- gramming 39, 117–129 (1987). DOI https://doi.org/10.1007/BF0259294 8

    1987 doi

  39. [47]

    Proceedings of the USSR Academy of Sciences 269, 543–547 (1983)

    Nesterov, Y.: A method for Solving the Convex Programmin g Problem with Conver- gence Rate O(1/k 2). Proceedings of the USSR Academy of Sciences 269, 543–547 (1983)

    1983

  40. [48]

    DOI https://doi

    Nesterov, Y.: Introductory Lectures on Convex Optimiza tion (2003). DOI https://doi. org/10.1007/978-1-4419-8853-9

    2003 doi

  41. [49]

    Sprin ger New York (2006)

    Nocedal, J., Wright, S.: Numerical Optimization. Sprin ger New York (2006). DOI https://doi.org/10.1007/978-0-387-40065-5

    2006 doi

  42. [50]

    Springer Proceedings in Mathematics and Statis- tics 50 (2013)

    Noll, D., Rondepierre, A.: Convergence of Linesearch an d Trust-Region Methods using the Kurdyka-/suppress Lojasiewicz Inequality. Springer Proceedings in Mathematics and Statis- tics 50 (2013). DOI 10.1007/978-1-4614-7621-427

    2013 doi

  43. [51]

    Journal of Optimization Theory and Applicatio ns 177(1), 153–180 (2018)

    Ochs, P.: Local Convergence of the Heavy-Ball Method and ipiano for Non-Convex Op- timization. Journal of Optimization Theory and Applicatio ns 177(1), 153–180 (2018). DOI https://doi.org/10.1007/s10957-018-1272-y

    2018 doi

  44. [52]

    In: International Conference on International Conf erence on Machine Learning (ICML) (2013)

    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of t raining recurrent neural net- works. In: International Conference on International Conf erence on Machine Learning (ICML) (2013)

    2013

  45. [53]

    In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Ling uistics: Human Language Technologies, pp

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Christo pher, C., al.: Deep Contex- tualized Word Representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Ling uistics: Human Language Technologies, pp. 2227–223...

    2018 doi

  46. [54]

    USSR Computational Mathematics and Mathematical Physics 4(5), 1–17 (1964)

    Polyak, B.: Some Methods of Speeding Up the Convergence of Iteration Methods. USSR Computational Mathematics and Mathematical Physics 4(5), 1–17 (1964). DOI https: //doi.org/10.1016/0041-5553(64)90137-5

    1964 doi

  47. [55]

    Qian, J., Wu, Y., Zhuang, B., Wang, S., Xiao, J.: Understa nding Gradient Clipping In Incremental Gradient Methods. In: A. Banerjee, K. Fukumi zu (eds.) Proceedings of The 24th International Conference on Artificial Intellig ence and Statistics, Proceed- ings of Machine Learnin...

    2021

  48. [56]

    In: Interna- tional Conference on Learning Representations (ICLR) (201 8)

    Reddi, S., Kale, S., Kumar, S.: On the Convergence of Adam and Beyond. In: Interna- tional Conference on Learning Representations (ICLR) (201 8). DOI 10.48550/ARXIV. 1904.09237

    1904 doi arXiv

  49. [57]

    IEEE Signal Processing Letters 22(10), 1671–1675 (2015)

    Richardson, F., Reynolds, D., Dehak, N.: Deep Neural Net work Approaches to Speaker and Language Recognition. IEEE Signal Processing Letters 22(10), 1671–1675 (2015). DOI 10.1109/LSP.2015.2420092

    2015

  50. [58]

    In: 2022 3rd URSI Atlantic and Asia Pacific Radio Science Meeting (AT-AP-RASC), pp

    Ripoll, J., Kluth, G., Has, S., Fischer, A., Mougeot, M., Camporeale, E.: Exploring Pitch-Angle Diffusion during High Speed Streams with Neural Networks. In: 2022 3rd URSI Atlantic and Asia Pacific Radio Science Meeting (AT-AP-RASC), pp. 1–4 (2022). DOI 10.23919/AT-AP-RASC54737....

    2022

  51. [59]

    In: Conference on Decision and Control (CDC) (2 022)

    Romero, O., Benosman, M., Pappas, G.: ODE discretizatio n schemes as optimization algorithms. In: Conference on Decision and Control (CDC) (2 022). DOI 10.1109/ CDC51059.2022.9992691

    2022

  52. [60]

    Com- mentationes Mathematicae Universitatis Carolinae 013(1), 45–51 (1972)

    Souˇ cek, J., Souˇ cek, V.: Morse-sard theorem for real-a nalytic functions. Com- mentationes Mathematicae Universitatis Carolinae 013(1), 45–51 (1972). URL http://eudml.org/doc/16473

    1972

  53. [61]

    Sun, R.: Optimization for Deep Learning: Theory and Algo rithms (2019) Complexities of Armijo-like algorithms in Deep Learning co ntext 29

    2019

  54. [62]

    COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)

    Tieleman, T., Hinton, G., al: Lecture 6.5-rmsprop: Divi de the Gradient by a Running Average of its Recent Magnitude. COURSERA: Neural networks for machine learning 4(2), 26–31 (2012)

    2012

  55. [63]

    In: Annua l Conference on Learning Theory, vol

    Wang, B., Zhang, H., Ma, Z., Chen, W.: Convergence of AdaG rad for Non-convex Ob- jectives: Simple Proofs and Relaxed Assumptions. In: Annua l Conference on Learning Theory, vol. 195, pp. 1–30 (2023)

    2023

  56. [64]

    In: Neural Information Pro cessing Systems (NeurIPS) (2019)

    Wilson, A., Mackey, L., Wibisono, A.: Accelerating Rescaled Gradient Descent: Fast Op- timization of Smooth Functions. In: Neural Information Pro cessing Systems (NeurIPS) (2019)

    2019

  57. [65]

    S IAM Review 11(2), 226–235 (1969)

    Wolfe, P.: Convergence Conditions for Ascent Methods. S IAM Review 11(2), 226–235 (1969). DOI 10.1137/1011036. URL https://doi.org/10.1137/1011036

    1969 doi

  58. [66]

    I I: Some Correc- tions

    Wolfe, P.: Convergence Conditions for Ascent Methods. I I: Some Correc- tions. SIAM Review 13(2), 185–188 (1971). DOI 10.1137/1013035. URL https://doi.org/10.1137/1013035

    1971 doi

  59. [67]

    Yue, P., Fang, C., Lin, Z.: On the Lower Bound of Minimizin g Polyak-/suppress Lojasiewicz Functions. In: G. Neu, L. Rosasco (eds.) Annual Conference on Learning Theory (2023)

    2023

  60. [68]

    DOI 10.48550/ ARXIV.1212.5701

    Zeiler, M.: ADADELTA: An adaptive learning rate method 1212 (2012). DOI 10.48550/ ARXIV.1212.5701

    2012 doi arXiv

  61. [69]

    In: Neural Information Processing Systems (NeurIPS) (2020)

    Zhang, B., Jin, J., Fang, C., Wang, L.: Improved Analysis of Clipping Algorithms for Non-convex Optimization. In: Neural Information Processing Systems (NeurIPS) (2020)

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.