Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For a broad class of convex functions with generalized smoothness, gradient descent converges linearly at first, then sublinearly, and the same two-phase behavior holds for normalized, clipped, and coordinate-descent variants.

desk verdict Full-gradient phase-split results are solid and worth knowing, but the RCD/OrderRCD theorems are not yet supported by the written proof. read the letter →

arxiv 2412.17050 v2 pith:3R3BOGUB submitted 2024-12-22 math.OC

Aleksandr Lobanov , Alexander Gasnikov , Eduard Gorbunov , Martin Takáč This is my paper · ORCID
classification math.OC MSC 90C2590C3065K05
keywords convexoptimizationgeneralizedsmoothness(L0L1)-smoothnessgradientdescentlinearconvergenceclippingcoordinateorderoracle
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proves that, for convex functions satisfying the $(L_0,L_1)$-smoothness condition—gradients change at a rate bounded by $L_0 + L_1$ times the current gradient norm—gradient descent with step size $1/(L_0+L_1\|\nabla f(x_k)\|)$ converges linearly as long as $\|\nabla f(x_k)\|\ge L_0/L_1$, then switches to the usual sublinear rate after the gradient falls below that threshold. The same two-phase behavior is proved for normalized gradient descent, clipped gradient descent, random coordinate descent, and a coordinate method that only compares function values. If correct, the results give faster linear rates during the early phase of optimization, replacing sublinear-only guarantees in the regime where $L_0$ is small relative to $L_1$, as in logistic regression. The paper also extends the GD analysis to strongly convex objectives, where three distinct rates appear.

What carries the argument

The load-bearing objects are the $(L_0,L_1)$-smoothness inequality $\|\nabla f(y)-\nabla f(x)\|\le (L_0+L_1\|\nabla f(x)\|)\|y-x\|$ for nearby points and the adaptive step size $\eta_k=(L_0+L_1\|\nabla f(x_k)\|)^{-1}$. The proof converts convexity into the bound $f(x_k)-f_*\le \|\nabla f(x_k)\|R$ via the imported distance estimate $\|x_k-x_*\|=\|x_0-x_*\|=R$, then splits the analysis at $\|\nabla f(x_k)\|=L_0/L_1$: above the threshold the step is roughly $1/(L_1\|\nabla f(x_k)\|)$, producing a contraction $\eta_k\|\nabla f(x_k)\|^2\gtrsim (f(x_k)-f_*)/(4L_1R)$; below the threshold the descent lemma gives the usual $1/N$ recurrence. A separate monotonicity lemma shows the gradient norm never increases along the iterates, which is what makes the phase description a clean before/after story rather than an interleaving of rates.

What would settle it

Run $(L_0,L_1)$-GD on a convex, $(L_0,L_1)$-smooth function (for example the logistic loss with a small strongly convex regularizer) from a point with $\|\nabla f(x_0)\|>L_0/L_1$, recording $\|x_k-x_*\|$ and $\|\nabla f(x_k)\|$ at every step; if either quantity ever increases, the monotonicity and distance lemmas behind Theorem 3.1 fail, breaking the claimed rate. Absence of the predicted linear segment while the gradient norm stays above $L_0/L_1$ would also directly contradict the theorem.

Watch

Extended reading notes

Core claim

The central claim is that the threshold $\|\nabla f(x)\|=L_0/L_1$ divides the behavior of the methods. For $(L_0,L_1)$-GD, Theorem 3.1 proves $f(x_N)-f_*\le \min\{4L_0R^2/(N-T), (1-1/(4L_1R))^T F_0\}$, where $T$ is the first index with $\|\nabla f(x_T)\|<L_0/L_1$; the linear factor holds while the gradient norm is at or above the threshold and the sublinear factor afterwards. Because the gradient norm is shown to be non-increasing along the iterates, the switch between phases happens at most once. The same linear-then-sublinear split is established for normalized and clipped GD, with the clip radius $c$ determining the effective linear rate, and for random coordinate descent and its order-oracle version under coordinate-wise $(L_0,L_1)$-smoothness, with rates $(1-\rho/(dR))^N F_0$ and $dL_0R^2/N$. In the strongly convex case the GD rate becomes a product of three geometric factors, each active in a different regime.

Load-bearing premise

The proof assumes the iterates never move farther from a minimizer than the starting point, $\|x_k-x_*\|\le\|x_0-x_*\|=R$ for every $k$, which is the step where convexity is converted into the gradient-norm bound on suboptimality.

Editorial extensions

If this is right

  • Under convexity and $(L_0,L_1)$-smoothness, $(L_0,L_1)$-GD, NGD, and Clip-GD all converge linearly while $\|\nabla f(x_k)\|$ stays above the threshold, with a rate of the form $O((1-\rho/R)^T F_0)$ for a method-dependent $\rho$.
  • Once the gradient norm drops below the threshold, the methods fall back to the standard sublinear $O(L_0R^2/(N-T))$ rate, matching previously known worst-case bounds.
  • Random coordinate descent and OrderRCD inherit the same phase split under coordinate-wise $(L_0,L_1)$-smoothness, with linear rate $(1-1/(4\sqrt{2}dL_1R))^N$ and sublinear factor $dL_0R^2/N$; OrderRCD needs only $O(\log(1/\epsilon))$ function-comparison oracle calls per iteration.
  • In the strongly convex case, GD exhibits three regimes: rate $1-1/(4L_1R)$ while the gradient is large and $F_k\ge 1$, rate $1-\max\{\sqrt{\mu}/(2\sqrt{2L_1}),1/(4L_1R)\}$ while the gradient is large, and rate $1-\mu/(2L_0)$ once the gradient is small.
  • When $L_0=0$ (the strong-growth smoothness condition), all analyzed methods converge linearly to arbitrarily high accuracy, e.g., $O(L_1R\log(F_0/\epsilon))$ iterations for GD and $O(dL_1R\log(F_0/\epsilon))$ for RCD.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the threshold $L_0/L_1$ is exactly where the adaptive step's $L_1$ term starts to dominate the $L_0$ term, so the same phase split should appear for other adaptive first-order methods—stochastic gradient descent with clipping, AdaGrad-like steps, and parameter-free methods—on objectives whose smoothness constant grows with the gradient norm.
  • The paper leaves open whether the distance bound $\|x_k-x_*\|\le R$ can be replaced by a sublevel-set localization; if it can, the linear phase would extend to functions with unbounded minimizer sets, such as the logistic loss itself, where the paper's Appendix F uses a finite accuracy radius $R_s$ instead.
  • A testable extension: on a logistic-regression objective (satisfying $L_0=0$), the analysis predicts geometric decay $O((1-1/(2L_1R_s))^N)$ rather than a $1/N$ tail; running GD with $\eta_k=1/(L_1\|\nabla f(x_k)\|)$ and measuring the ratio of successive function gaps would verify the predicted transition.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies convex (L0,L1)-smooth optimization and claims that gradient descent variants enjoy a two-phase convergence behavior: linear convergence while the gradient norm stays above L0/L1, and standard sublinear convergence afterward. The authors prove such rates for (L0,L1)-GD, Normalized GD, and Clipped GD, and extend the same phase-split claim to Random Coordinate Descent and to a coordinate descent method using only an order oracle. A strongly convex extension for (L0,L1)-GD is also provided. The full-gradient proofs are standard descent-plus-convexity arguments, while the coordinate-descent section relies on a subsequence argument that is written informally and whose stated rates do not match the proof.

Significance. If established, the full-gradient results would improve on prior (L0,L1)-smooth convex rates by making the linear initial phase explicit, and the RCD/OrderRCD results would be the first in this setting. The paper is clearly written and the qualitative phase-split claim is a crisp, falsifiable prediction. The step-size choice for (L0,L1)-GD is parameter-free and the comparison table with prior work is useful. However, the advertised coordinate-descent results are not currently supported by the written proof: the theorem states a stronger linear exponent than the proof delivers, and the random-set conditioning in Appendix D.1 is not rigorous. These issues are substantial because RCD/OrderRCD form a central claimed contribution.

major comments (3)
  1. [Theorem 4.1 and Appendix D.1] The theorem states E[f(x_N)] - f* = O(max{(1 - 1/(4*sqrt(2)*d*L1*R))^N F0, d*L0*R^2/N}), but the proof concludes with max{(1 - 1/(4*sqrt(2)*d*L1*R))^{N/2} F0, 16*d*L0*R^2/N}. Since (1-a)^{N/2} cannot be bounded by a constant multiple of (1-a)^N as N grows, the stated linear rate is strictly stronger than what is proven. The theorem and proof must be reconciled, for example by stating the exponent N/2 or by supplying a sharper argument. In addition, the theorem claims the bound for every step size eta_k satisfying the inequality, while the proof uses the maximal step size; a smaller step size weakens descent inequality (38)-(40), so the statement should be restricted to the exact step size or a proof for all smaller steps should be provided.
  2. [Appendix D.1, set K and subsequence telescoping] The set K = {k : sum_{i in I_k} |grad_i f(x_k)|^2 > sum_{i not in I_k} |grad_i f(x_k)|^2} is random because it depends on the random iterates, so r=|K| and the indices k_j, t_j are random objects. The proof applies the linear contraction (43) along the K-subsequence and telescopes the complementary subsequence as though the indices were fixed, and the notation E_{i in K} is not a valid conditional expectation over a random stopping set. A rigorous treatment via stopping times, or a deterministic case analysis over all subsets of the iteration index set, is needed. This gap affects Theorem 4.1 and, through the identical argument referenced in Appendix D.2, Theorem 4.3.
  3. [Theorem 3.5 and Table 1] The theorem states the linear contraction factor as (1 - rho/R)^T with rho = c/max{L0, L1*c}, which is (1 - 1/(L1*R))^T when c >= L0/L1 and (1 - c/(L0*R))^T when c < L0/L1. The proof in Appendix C.3, however, yields (1 - 1/(4*L1*R))^T and (1 - c/(4*L0*R))^T, respectively. The missing factor 1/4 in the exponent cannot be absorbed into the big-O notation for geometric rates, so the theorem and Table 1 overclaim the proven rate. The qualitative phase-split conclusion is unaffected, but the stated rates must be corrected.
minor comments (5)
  1. [Appendix D.1, k not in K case] In the derivation before equation (45), the displayed implication states sum_{i in I_k} |grad_i f(x_k)|^2 >= F_k^2/(2R^2), but the preceding inequality bounds the complementary sum over i not in I_k; as written the displayed line is false and should be corrected to sum_{i not in I_k} |grad_i f(x_k)|^2 >= F_k^2/(2R^2).
  2. [Sections 3.2-3.3 and Table 1] Several narrative passages and table entries report rates such as O((1 - 1/(L1*R))^N F0) for NGD and Clip-GD, while the corresponding proofs in Appendices C.2-C.3 give contractions with constants 1/4 in the exponent; please standardize the constants between the theorems, the table, and the surrounding text.
  3. [Appendix C.1, Theorem 3.1 proof] The proof repeatedly uses the distance bound ||x_k - x*|| <= ||x_0 - x*|| = R, attributed to Gorbunov et al. (2024, proof of Theorem 3.3), without stating it as a lemma. Since this bound is load-bearing for every linear-rate estimate in the paper, it should be stated explicitly with a proof or a precise reference to the exact lemma.
  4. [Algorithm 5] The input line of Algorithm 5 mentions a 'random generator R_alpha(L0,L1)' that is neither defined nor used in the algorithm or its analysis; please remove it or clarify its role.
  5. [Appendix C.1, definition of T] The proof defines T as the first index with ||grad f(x_k)|| < L0/L1 and ||grad f(x_{k-1})|| >= L0/L1, while the theorem defines T as the smallest index satisfying only the first condition; these definitions should be aligned, especially to cover the case T=0 when the initial gradient is already below the threshold.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the convergence rates follow from the stated smoothness and convexity inequalities, with only minor self-citations for supporting lemmas.

full rationale

Walking the proof chain of Theorems 3.1-4.3, I find no step where a predicted quantity is defined in terms of itself, where a fitted parameter is renamed as a prediction, or where a uniqueness claim from the authors' prior work forces the choice. The bounds follow from the (L0,L1)-smoothness upper bounds (8)-(9), convexity (3), Cauchy-Schwarz, and the explicit step size eta_k=(L0+L1||grad f(x_k)||)^(-1); L0, L1, R and F0 are inputs rather than fitted quantities. The min-form bound in Theorem 3.1 separates the T iterations with ||grad f(x_k)|| >= L0/L1 and applies the standard 1/(N-T) telescope afterwards, so the final rate is not equivalent to its assumptions by construction. The paper does import ||x_k-x*|| <= ||x0-x*|| from Gorbunov et al. (2024, proof of Theorem 3.3) and the OrderRCD/GRM framework from Lobanov et al. (2024); these are supporting lemmas/algorithms by overlapping authors, not the target rates, and they are checkable independently of the present claims, so they count as normal self-citation rather than circular inputs. The informal conditioning on the random set K in Appendix D.1 and the mismatch between the stated exponent N and the proven exponent N/2 in Theorem 4.1 are proof-correctness gaps, not circularity: the proof never inserts the conclusion into the assumptions. The footnote about being unaware of the updated Vankov et al. version is a credit note, not a dependency. Score 2 reflects only the minor self-citations.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest only on standard problem parameters and imported textbook/prior results. There are no fitted constants, hand-chosen coefficients beyond the fixed 1/4 and 1/(4*sqrt(2)) factors, or new model entities.

assumptions (5)
  • domain assumption Function is (L0,L1)-smooth (Assumption 1.2)
    This is the central regularity condition defining the problem class; it is taken as given, not derived.
  • domain assumption Function is convex, and for Section 5 strongly convex (Assumption 1.4)
    Convexity is used in every descent-to-suboptimality conversion; strong convexity adds the Polyak-Lojasiewicz inequality.
  • domain assumption Coordinate-wise (L0,L1)-smoothness (Assumption 1.3)
    Needed for the RCD and OrderRCD analyses in Section 4.
  • standard math Distance bound ||x_k-x*||<=||x0-x*||=R, cited from Gorbunov et al. 2024
    Imported unproved in Appendix C.1 and used to bound f(x_k)-f* by ||grad f(x_k)||R.
  • standard math Golden Ratio Method converges for one-dimensional comparison-based line search
    Used in Appendix D.2 to implement the order oracle; its O(log(1/epsilon)) iteration count is a standard fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness." pith.science (2026). https://pith.science/paper/3R3BOGUB

@misc{pith2026241217050,
  author       = {Pith},
  title        = {Pith review of: Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3R3BOGUB}},
  note         = {Machine review of arXiv:2412.17050}
}
abstract

The gradient descent (GD) method -- is a fundamental and likely the most popular optimization algorithm in machine learning (ML), with a history traced back to a paper in 1847 (Cauchy, 1847). It was studied under various assumptions, including so-called $(L_0,L_1)$-smoothness, which received noticeable attention in the ML community recently. In this paper, we provide a refined convergence analysis of gradient descent and its variants, assuming generalized smoothness. In particular, we show that $(L_0,L_1)$-GD has the following behavior in the convex setup: as long as $\|\nabla f(x^k)\| \geq \frac{L_0}{L_1}$ the algorithm has linear convergence in function suboptimality, and when $\|\nabla f(x^k)\| < \frac{L_0}{L_1}$ is satisfied, $(L_0,L_1)$-GD has standard sublinear rate. Moreover, we also show that this behavior is common for its variants with different types of oracle: Normalized Gradient Descent as well as Clipped Gradient Descent (the case when the full gradient $\nabla f(x)$ is available); Random Coordinate Descent (when the gradient component $\nabla_{i} f(x)$ is available); Random Coordinate Descent with Order Oracle (when only $\text{sign} [f(y) - f(x)]$ is available). In addition, we also extend our analysis of $(L_0,L_1)$-GD to the strongly convex case.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Few Accelerated Algorithms for Convex Optimization under $(H_0,H_1)$-Smoothness

    math.OC 2026-08 conditional novelty 6.0 of 10

    For convex (H0,H1)-smooth objectives, restarted accelerated gradient methods achieve O(sqrt(H0 R^2/eps)+sqrt(H1 R^2) log(F0/eps)) iterations, and accelerated coordinate variants pay a standard factor d or use importan...

  2. Decentralized Stochastic Nonconvex Optimization under the $(L_0,L_1)$-Smoothness

    math.OC 2025-09 conditional novelty 6.0 of 10

    DNSGD is a decentralized normalized stochastic gradient method for (L0,L1)-smooth nonconvex optimization, with complexity bounds that match standard smooth decentralized results when L1=0.

  3. Power of Generalized Smoothness in Stochastic Convex Optimization: First- and Zero-Order Algorithms

    math.OC 2025-01 conditional novelty 6.0 of 10

    For convex stochastic optimization under (L0,L1)-smoothness, clipped and normalized SGD (and zero-order variants) obtain linear-rate terms in their convergence bounds, and in the L0=0 regime NSGD achieves logarithmic ...

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages · cited by 3 Pith papers

  1. [1]

    Allen-Zhu, Z., Qu, Z., Richt \'a rik, P., and Yuan, Y. (2016). Even faster accelerated coordinate descent using non-uniform sampling. In International Conference on Machine Learning , pages 1110--1119. PMLR

    work page 2016

  2. [2]

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    and Vandenberghe, L

    Boyd, S. and Vandenberghe, L. (2004). Convex optimization . Cambridge university press

    2004

  4. [4]

    Bubeck, S. et al. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8(3-4):231--357

    work page 2015

  5. [5]

    Cauchy, A. (1847). M \'e thode g \'e n \'e rale pour la r \'e solution des systemes d’ \'e quations simultan \'e es. Comp. Rend. Sci. Paris , 25(1847):536--538

    work page

  6. [6]

    Chen, Z., Zhou, Y., Liang, Y., and Lu, Z. (2023). Generalized-smooth nonconvex optimization is as efficient as smooth nonconvex optimization. In International Conference on Machine Learning , pages 5396--5427. PMLR

    work page 2023

  7. [7]

    Crawshaw, M., Liu, M., Orabona, F., Zhang, W., and Zhuang, Z. (2022). Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems , 35:9955--9968

    work page 2022

  8. [8]

    Faw, M., Rout, L., Caramanis, C., and Shakkottai, S. (2023). Beyond uniform smoothness: A stopped analysis of adaptive sgd. In The Thirty Sixth Annual Conference on Learning Theory , pages 89--160. PMLR

    work page 2023

Show all 36 references
  1. [9]

    H., and Richtarik, P

    Gorbunov, E., Bibi, A., Sener, O., Bergou, E. H., and Richtarik, P. (2019). A stochastic derivative free optimization method with momentum. In International Conference on Learning Representations

    2019

  2. [10]

    Gorbunov, E., Tupitsa, N., Choudhury, S., Aliev, A., Richt \'a rik, P., Horv \'a th, S., and Tak \'a c , M. (2024). Methods for convex (l\_0, l\_1) -smooth optimization: Clipping, acceleration, and adaptivity. arXiv preprint arXiv:2409.14989

    2024 arXiv

  3. [11]

    H \"u bler, F., Yang, J., Li, X., and He, N. (2024). Parameter-agnostic optimization under relaxed smoothness. In International Conference on Artificial Intelligence and Statistics , pages 4861--4869. PMLR

    2024

  4. [12]

    Koloskova, A., Hendrikx, H., and Stich, S. U. (2023). Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In International Conference on Machine Learning , pages 17343--17363. PMLR

    2023

  5. [13]

    Li, H., Qian, J., Tian, Y., Rakhlin, A., and Jadbabaie, A. (2024a). Convex and non-convex optimization under generalized smoothness. Advances in Neural Information Processing Systems , 36

    2024

  6. [14]

    Li, H., Rakhlin, A., and Jadbabaie, A. (2024b). Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems , 36

    2024

  7. [15]

    Lin, Q., Lu, Z., and Xiao, L. (2014). An accelerated proximal coordinate gradient method. Advances in Neural Information Processing Systems , 27

    2014

  8. [16]

    Lobanov, A., Gasnikov, A., and Krasnov, A. (2024). Acceleration exists! optimization problems when oracle can only compare objective function values. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  9. [17]

    ojasiewicz, S. (1963). A topological property of real analytic subsets. Coll. du CNRS, Les \'e quations aux d \'e riv \'e es partielles , 117:87--89

    1963

  10. [18]

    Nesterov, Y. (2012). Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341--362

    2012

  11. [19]

    Nesterov, Y. (2013). Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media

    2013

  12. [20]

    Nesterov, Y. (2018). Lectures on convex optimization , volume 137. Springer

    2018

  13. [21]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730--27744

    2022

  14. [22]

    Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In Dasgupta, S. and McAllester, D., editors, Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research ,...

    2013

  15. [23]

    Polyak, B. T. (1963). Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics , 3(4):864--878

    1963

  16. [24]

    and Tak \'a c , M

    Richt \'a rik, P. and Tak \'a c , M. (2016). Distributed coordinate descent method for learning with big data. Journal of Machine Learning Research , 17(75):1--25

    2016

  17. [25]

    Saha, A., Koren, T., and Mansour, Y. (2021). Dueling convex optimization. In International Conference on Machine Learning , pages 9245--9254. PMLR

    2021

  18. [26]

    and Tewari, A

    Shalev-Shwartz, S. and Tewari, A. (2009). Stochastic methods for l 1 regularized loss minimization. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 929--936

    2009

  19. [27]

    Takezawa, Y., Bao, H., Sato, R., Niwa, K., and Yamada, M. (2024). Polyak meets parameter-free clipped gradient descent. arXiv preprint arXiv:2405.15010

    2024 arXiv

  20. [28]

    Tang, Z., Rybin, D., and Chang, T.-H. (2024). Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. In The Twelfth International Conference on Learning Representations

    2024

  21. [30]

    Vankov, D., Rodomanov, A., Nedich, A., Sankar, L., and Stich, S. U. (2024b). Optimizing ( L _0, L _1) -smooth functions by gradient methods. arXiv preprint arXiv:2410.10800, version 1

    2024 arXiv

  22. [31]

    Vaswani, S., Bach, F., and Schmidt, M. (2019). Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics , pages 1195--1204. PMLR

    2019

  23. [32]

    Wang, B., Zhang, H., Ma, Z., and Chen, W. (2023). Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions. In The Thirty Sixth Annual Conference on Learning Theory , pages 161--190. PMLR

    2023

  24. [33]

    Wang, B., Zhang, Y., Zhang, H., Meng, Q., Ma, Z.-M., Liu, T.-Y., and Chen, W. (2022). Provable adaptivity in adam. arXiv preprint arXiv:2208.09900

    2022 arXiv

  25. [34]

    Zhang, B., Jin, J., Fang, C., and Wang, L. (2020a). Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems , 33:15511--15521

    2020

  26. [35]

    Zhang, J., He, T., Sra, S., and Jadbabaie, A. (2020b). Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations

    2020

  27. [36]

    and Xiao, L

    Zhang, Y. and Xiao, L. (2017). Stochastic primal-dual coordinate method for regularized empirical risk minimization. Journal of Machine Learning Research , 18(84):1--42

    2017

  28. [37]

    Zhao, S.-Y., Xie, Y.-P., and Li, W.-J. (2021). On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences , 64:1--13

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.