Pith. sign in

REVIEW 3 major objections 4 minor 42 references

Stable gradient-adjusted root mean square propagation on least squares problem

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A stable RMSProp variant is proved to converge R-linearly on linear least squares problems, with batch size controlling the final error floor in the noisy case.

desk verdict A plausible conditional linear-rate proof for a stable coordinate-wise RMSProp variant, but the abstract's claim runs far ahead of the batch-size regime the theorem actually covers. read the letter →

arxiv 2411.15877 v2 pith:VJK5WR67 submitted 2024-11-24 math.OC

Runze Li , Jintao Xu , Wenxun Xing This is my paper · ORCID
classification math.OC MSC 90C0690C3068T0968W20
keywords RMSProplinearleastsquaresR-linearconvergencestochasticgradientmini-batchadaptivemethodsmatrixBernsteininequality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SGA-RMSProp, a version of RMSProp that adaptively selects its discounting factor at each iteration so the per-coordinate scaling matrix varies only within a preset tolerance. It proves that on a consistent linear least squares problem the method converges R-linearly in expectation, and on an inconsistent problem it converges R-linearly to a neighborhood of the minimizer whose radius is set by the mini-batch size. This matters because RMSProp is widely used but previously had no linear-rate guarantee on least squares, while stochastic gradient descent has such guarantees. The proof gives an explicit contraction factor, an explicit step size depending on the extreme eigenvalues of $A^{\top}A$, and a batch-size condition that makes the contraction factor smaller than one.

What carries the argument

The load-bearing device is the stable gradient-adjusted mechanism: at each iteration $\beta_k$ is chosen so that the moving average $u_k$ stays in a bounded interval and the scaling matrix changes slowly, $\|D_k^{-1}D_{k-1}-I\|_2 \le \varepsilon$. This lets the proof factor the product of stochastic transition matrices into controlled diagonal factors times matrices of the form $I - \eta D_{k-1}^{1/2} M_k D_{k-1}^{1/2}$, bound the conditional expectation of each factor by $1 - 2(u\lambda_d(A^{\top}A)-u\sigma)/(u(\lambda_1(A^{\top}A)+\lambda_d(A^{\top}A))) + \varepsilon$ using the matrix Bernstein inequality, and chain the expectations through the filtration.

What would settle it

Take a small consistent least squares problem, say $d=2$ with known eigenvalues of $A^{\top}A$, set $B$ and $\varepsilon$ to satisfy Corollary 4.4, run SGA-RMSProp many times with the step size of Theorem 4.3, and compare the empirical mean error against $\rho\|x_1-x_*\|_2\gamma^K$ with $\gamma$ from (4.4); a single violation at large $K$ would refute the bound. As a control, computing $u\lambda_d(A^{\top}A)-u\sigma$ for $B=1$ shows the theorem is silent in that regime, so any observed linear convergence there would only show the batch-size condition is not necessary.

Watch

Extended reading notes

Core claim

The central claim is that with step size $\eta = 2/(u(\lambda_1(A^{\top}A)+\lambda_d(A^{\top}A)))$ and the $\beta$-selection rule of Algorithm 3.2, SGA-RMSProp satisfies $E[\|x_{K+1}-x_*\|_2] \le \rho \|x_1-x_*\|_2 \gamma^K$ on the consistent least squares problem, where $\gamma$ is given in (4.4) as the geometric mean of per-iteration factors times $1 - 2(u\lambda_d(A^{\top}A)-u\sigma)/(u(\lambda_1(A^{\top}A)+\lambda_d(A^{\top}A))) + \varepsilon$. On the inconsistent problem the same contraction holds up to an additive error term $R$ whose upper bound is proportional to the residual and decreases as the batch size increases (Theorem 4.7).

Load-bearing premise

The linear-rate guarantee is conditional on the batch being large enough that the noise term $u\sigma$ stays strictly below $u\lambda_d(A^{\top}A)$; for smaller batches the proof yields no rate, and the step size presumes exact knowledge of the extreme eigenvalues of $A^{\top}A$.

Editorial extensions

If this is right

  • On consistent least squares, SGA-RMSProp has an explicit R-linear rate whenever the batch size is large enough that the noise bound $\sigma$ in (4.5) satisfies $u\lambda_d(A^{\top}A) > u\sigma$.
  • On inconsistent least squares, the method converges R-linearly to a neighborhood of the minimizer whose radius is bounded by (4.25) and shrinks as the batch size grows.
  • The admissible range for the adjusted level is $0 < \varepsilon < 2(u\lambda_d(A^{\top}A)-u\sigma)/(u(\lambda_1(A^{\top}A)+\lambda_d(A^{\top}A)))$; within this range the linear rate holds.
  • Numerical experiments show SGA-RMSProp matches or beats SGD on small batches, and the proposed RMSP2SGD switch (SGA-RMSProp first, then SGD) generally reduces wall-clock time on large batches.
  • The analysis explains the fast initial phase of RMSProp-style methods as coming from per-iteration factors $\gamma_k < 1$, rather than from the asymptotic constant alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof requires exact knowledge of the extreme eigenvalues of $A^{\top}A$; estimating them online is the natural next step, and the numerical step-size heuristics suggest the method tolerates moderate deviations.
  • The linear-rate theorem is silent for small batches because $u\sigma$ can exceed $u\lambda_d(A^{\top}A)$, yet the experiments show the strongest early speedup at $B=50$; this gap suggests either the noise bound is loose or the true worst-case contraction is better than stated.
  • The mechanism of capping $\|D_k^{-1}D_{k-1}-I\|_2$ by $\varepsilon$ is portable: any adaptive-gradient method that controls its preconditioner's variation could plausibly get similar linear-rate arguments on quadratic objectives.
  • The RMSP2SGD switching rule (five consecutive $\beta_k = 1$) is heuristic; a criterion based on when the geometric-mean factor stops improving could make the switch principled.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SGA-RMSProp, a variant of RMSProp for linear least squares problems in which the discounting factor beta_k is selected adaptively at each iteration so that the diagonal scaling matrix D_k satisfies the stability bound ||D_k^{-1}D_{k-1}-I||_2 <= epsilon. The main theoretical results are Theorem 4.3, an R-linear convergence bound in expectation for the consistent case, and Theorem 4.7, an R-linear convergence bound to a neighborhood of the minimizer for the inconsistent case, with the neighborhood radius controlled by the batch size. The paper also reports numerical experiments comparing SGA-RMSProp against SGD and RMSProp, and proposes a heuristic switching strategy (RMSP2SGD) that starts with SGA-RMSProp and switches to SGD after a stability condition is met.

Significance. If the stated results hold as qualified, the paper contributes a rare convergence-rate analysis for an adaptive coordinate-wise preconditioned stochastic method on a concrete problem class, and the stability mechanism based on controlling ||D_k^{-1}D_{k-1}-I||_2 is a clean, potentially reusable idea. The proofs are detailed and largely self-contained, and the numerical study is organized and reproducible in structure. However, the central R-linear convergence claim is only proven under a large-batch condition that the experiments do not satisfy, so the practical reach of the theoretical result is much narrower than the abstract suggests.

major comments (3)
  1. [Abstract and Theorem 4.3 / Corollary 4.4] The abstract states that 'R-linear convergence of the algorithm is established on the consistent linear least squares problem' without qualification, and Theorem 4.3 states (4.3) with gamma given by (4.4). However, gamma < 1 is only guaranteed under the batch-size condition of Corollary 4.4 together with the stated bound on epsilon. For the paper's own sampling rule p_j = ||a_j||_2^2 / ||A||_F^2 (Remark 4.5), the Corollary 4.4 condition reads B >= 4 u^2 log(2d) ||A||_F^2 / u^2 * (2 sqrt(2) ||A||_2 / lambda_d(A^T A) + sqrt(u/(3 u lambda_d(A^T A))))^2. Plugging in the Section 5.1 parameters (n = 10^6, d = 102, lambda_d = 1, u/u = 5, and lambda_1 between 20 and 100) gives a lower bound on the order of 10^7 to 10^9, while the experiments use B = 50 and B = 1000. Thus the numerically observed linear convergence in Figure 1 occurs in a regime where Theorem 4.3 provides no guarantee, and the headline claim overstates what is proven. The abstract and theorem statements should be qualified with the batch-size condition, and the relation between the experiments and the proven regime should be discussed explicitly.
  2. [Theorem 4.3, proof around (4.6)] Theorem 4.3 states that the leading constant satisfies rho <= 1, but the proof bounds rho by ||D_K^{1/2}||_2 ||D_0^{-1/2}||_2, which by Proposition 3.1 is at most sqrt(u/u). Since u is a lower bound and u is an upper bound on the scaling, typically u < u, so sqrt(u/u) > 1. Therefore the claim rho <= 1 is not established; the correct statement would be rho <= sqrt(u/u) (or an explicit bound in terms of u and u). R-linear convergence remains meaningful with rho > 1, so this is correctable, but the theorem as stated is inaccurate.
  3. [Theorem 4.3, definition of gamma_k] The rate gamma in (4.4) depends on constants gamma_1,...,gamma_K that are never explicitly defined; the proof only says they are 'bounded above by some constants at most 1' after (4.6). Since Corollary 4.4 and the discussion after Theorem 4.3 rely on the product G(gamma_1,...,gamma_K) being at most 1, the argument would be self-contained if gamma_k were defined as explicit bounds, e.g., gamma_k = ||D_k^{1/2}D_{k-1}^{-1/2}||_2, which Proposition 3.1 already shows to be at most 1. As written, the theorem's rate contains unspecified quantities, which weakens its status as a quantitative convergence guarantee.
minor comments (4)
  1. [Throughout] The notation u and u (underline and overline) is visually difficult to distinguish in the typeset text; a different notation such as u_min and u_max would improve readability.
  2. [Section 5.1.2] The narrative states that for B = 50 the algorithm generally performs best with u = u1, but then selects u2 and u2 for subsequent experiments; the rationale for not using the empirically better u1 in the later comparisons is not explained.
  3. [Section 5.3] Figure 3 is described as having an inset plot in each panel, but the captions do not clearly identify which portion is the inset; this makes the claim about early-stage behavior harder to verify.
  4. [Algorithm 3.2] The interval in the selection rule is written as (max{...}, 1), which suggests an open interval; the algorithm presumably allows beta_k = 1 or values approaching 1, so the interval notation should be made precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SGA-RMSProp's stability bound is enforced by construction and then used honestly in the proof; the convergence claim rests on external concentration and eigenvalue results rather than on fitted inputs or self-citation.

full rationale

The derivation chain is self-contained. SGA-RMSProp is defined so that the β-selection rule in Algorithm 3.2 enforces the bounds u ≤ 1/√u_{k,j} ≤ u and ||D_k^{-1} D_{k-1} − I||_2 ≤ ε; Proposition 3.1 proves these bounds directly from the algorithm's update rule. Theorem 4.3 then uses those bounds, the step-size choice η = 2/(u(λ_1(A^T A)+λ_d(A^T A))), the matrix Bernstein inequality (Lemma 2.7), and Weyl/Ky Fan eigenvalue inequalities to bound E[||x_{K+1} − x*||_2]. The quantity σ in (4.5) is an honest data-dependent concentration bound from Lemma 4.2, not a fitted parameter renamed as a prediction. The constants ρ and γ_k are defined in the proof as norm products bounded via Proposition 3.1, not as quantities whose values are chosen to force the contraction. The inconsistent-case term R in Theorem 4.7 is derived from Lemma 4.6, an external concentration result, and is not assumed from the target conclusion. There are no load-bearing self-citations: the cited related results [5], [22], [29], etc. are independent prior work, and none is invoked as a substitute for the paper's own proof. The abstract's unqualified wording 'R-linear convergence' omits the batch-size and ε conditions in Corollary 4.4 that make the guaranteed rate γ < 1, and the numerical experiments at B = 50 and B = 1000 may lie outside that guaranteed regime; however, that is an overstatement of the theorem's scope, not a circular step. The proof would still stand unchanged if the hypotheses were stated explicitly. Therefore no circularity is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on user-chosen hyperparameters (epsilon, u, u) and the batch size; these are not fitted to the convergence result, but the theorem only holds for a specified range. The main structural assumptions are full column rank and independent sampling, which are standard for least squares analysis.

free parameters (4)
  • adjusted level epsilon = lambda_d/lambda_1, 5 lambda_d/lambda_1, or 10 lambda_d/lambda_1 in experiments
    Controls the stability condition ||D_k^{-1}D_{k-1}-I||_2 <= epsilon and enters the convergence rate gamma in (4.4); a user-chosen quantity.
  • upper bound u on scaling D_k = chosen from first stochastic gradient g1 in experiments, e.g., u_i = (10^{-i} min_j{g^2_{1,j} | g_{1,j} != 0})^{-1/2}
    Limits ||D_k||_2 and appears in the step size eta; set adaptively from data in the experiments.
  • lower bound u on scaling D_k = u/5 or u/10 in experiments
    Ensures each coordinate scaling is at least u, used in the lower bound lambda_d(D^{1/2} A^T A D^{1/2}) >= u lambda_d(A^T A) in (4.16).
  • mini-batch size B = 50 or 1000 in experiments; theorem requires a lower bound from Corollary 4.4
    Controls the noise level sigma; the linear convergence proof requires B sufficiently large.
assumptions (3)
  • domain assumption A has full column rank
    Guarantees lambda_d(A^T A) > 0 and a unique minimizer; stated in Section 1 before (1.4).
  • domain assumption Samples xi_i are i.i.d. with p_j > 0 for j = 1,...,n
    Used to establish unbiased stochastic gradient (3.1) and the matrix Bernstein bound in Lemma 2.7.
  • standard math Standard matrix concentration inequalities (Weyl, Ky Fan, matrix Bernstein)
    Borrowed from Horn-Johnson and Tropp; used in Lemmas 2.1, 2.2, and 2.7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stable gradient-adjusted root mean square propagation on least squares problem." pith.science (2026). https://pith.science/paper/VJK5WR67

@misc{pith2026241115877,
  author       = {Pith},
  title        = {Pith review of: Stable gradient-adjusted root mean square propagation on least squares problem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJK5WR67}},
  note         = {Machine review of arXiv:2411.15877}
}
read the original abstract

Root mean square propagation (abbreviated as RMSProp) is a first-order stochastic algorithm used in machine learning widely. In this paper, a stable gradient-adjusted RMSProp (abbreviated as SGA-RMSProp) with mini-batch stochastic gradient is proposed, and its properties are studied on the linear least squares problem. R-linear convergence of the algorithm is established on the consistent linear least squares problem. The algorithm is also proved to converge R-linearly to a neighborhood of the minimizer for the inconsistent case, with the region of the neighborhood being controlled by the batch size. Furthermore, numerical experiments are conducted to compare the performances of SGA-RMSProp, stochastic gradient descent (abbreviated as SGD), and the original RMSProp with different batch sizes. The faster initial convergence rate of SGA-RMSProp is observed through numerical experiments and an adaptive strategy for switching from SGA-RMSProp to SGD is proposed, which combines the benefits of these two algorithms.

Figures

Figures reproduced from arXiv: 2411.15877 by the authors.

Figure 1
Figure 1. The R-linear convergence of SGA-RMSProp with B = 1000. 5.1.4. Comparison of SGA-RMSProp and other algorithms. We first compare our al￾gorithm with the original RMSProp, whose iteration schemes follow (1.2)-(1.3) with all hy￾perparameters set as constants, on 12 instances derived from the problems (ED, 20, 0.7), (ED, 100, 0.7), (AD, 20, 2), and (AD, 100, 2). These instances cover both small and large condition number… view at source ↗
Figure 2
Figure 2. Comparison of SGA-RMSProp and SGD on (AD, 20, 1) with B = 1000. RMSProp and switching to SGD at an appropriate time may improve the performance of the algorithm. Based on this idea, we conduct the next experiment. 5.2. RMSP2SGD. We introduce RMSP2SGD as follows, a method that enhances SGA￾RMSProp by adding an adaptive switching rule to SGD. The rule is that if βk equals 1 for five consecutive times, then the algorit… view at source ↗
Figure 3
Figure 3. Comparison of the three algorithms for the inconsistent case. this dataset to predict song release years from timbre features through linear regression, and finally formulated it as LLSP [4]. The dataset contains n = 463715 records and d = 90 timbre features. We apply mean-centered standardization to prepare the data for linear regression. Next, we set ε to ε2 and use a batch size B = 2000, with all other parameters… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p028_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 34 canonical work pages

  1. [1]

    Bertin-Mahieux, Year Prediction MSD

    T. Bertin-Mahieux, Year Prediction MSD. UCI Machine Learning Repository, 2011, https://doi.org/ 10.24432/C50K61

    doi:10.24432/c50k61 2011

  2. [2]

    Bertsekas, Nonlinear Programming, Athena Scientific, Belmont, 1999

    D. Bertsekas, Nonlinear Programming, Athena Scientific, Belmont, 1999

    work page 1999

  3. [3]

    Bertsekas, Convex Optimization Algorithms , Athena Scientific, Belmont, 2015

    D. Bertsekas, Convex Optimization Algorithms , Athena Scientific, Belmont, 2015

    work page 2015

  4. [4]

    Bhardwaj, R

    S. Bhardwaj, R. R. Curtin, M. Edel, Y. Mentekidis, and C. Sanderson , Ensmallen: A flexible C++ library for efficient function optimization , in Workshop on Systems for ML and Open Source Software at NeurIPS, 2018

    work page 2018

  5. [5]

    Bollapragada, T

    R. Bollapragada, T. Chen, and R. W ard, On the fast convergence of minibatch heavy ball momentum, IMA J. Numer. Anal., drae033 (2024), https://doi.org/10.1093/imanum/drae033

    doi:10.1093/imanum/drae033 2024

  6. [6]

    Cassioli, A

    A. Cassioli, A. Chiavaioli, C. Manes, and M. Sciandrone , An incremental least squares algorithm for large scale linear classification , Eur. J. Oper. Res., 224 (2013), pp. 560–565, https://doi.org/10. 1016/j.ejor.2012.09.004

    work page 2013

  7. [7]

    C. Chen, L. Shen, F. Zou, and W. Liu , Towards practical Adam: Non-convexity, convergence theory, and mini-batch acceleration, J. Mach. Learn. Res., 23 (2022), pp. 1–47

    work page 2022

  8. [8]

    Durrett, Probability: Theory and Examples , Cambridge university press, Cambridge, 2019

    R. Durrett, Probability: Theory and Examples , Cambridge university press, Cambridge, 2019

    work page 2019

Show all 42 references
  1. [9]

    M. L. Eaton , Multivariate Statistics: A Vector Space Approach , Institute of Mathematical Statistics, Beachwood, 2007

    2007

  2. [10]

    Gorbunov, F

    E. Gorbunov, F. Hanzely, and P. Richt´arik, A unified theory of SGD: Variance reduction, sampling, quantization and coordinate descent, in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 680–690

    2020

  3. [11]

    Z. Guo, Y. Xu, W. Yin, R. Jin, and T. Yang , A novel convergence analysis for algorithms of the Adam family and beyond , preprint, arXiv:2104.14840, 2021

    2021 arXiv

  4. [12]

    D. Han, Y. Su, and J. Xie , Randomized Douglas-Rachford methods for linear systems: Improved accu- racy and efficiency, SIAM J. Optim., 34 (2024), pp. 1045–1070, https://doi.org/10.1137/23M1567503

    2024 doi

  5. [13]

    R. A. Horn and C. R. Johnson , Topics in Matrix Analysis , Cambridge university press, Cambridge, 1994

    1994

  6. [14]

    R. A. Horn and C. R. Johnson , Matrix Analysis, Cambridge university press, Cambridge, 2012

    2012

  7. [15]

    Khaled, O

    A. Khaled, O. Sebbouh, N. Loizou, R. M. Gower, and P. Richt ´arik, Unified analysis of stochastic gradient methods for composite convex and smooth optimization , J. Optim. Theory Appl., 199 (2023), pp. 499–540, https://doi.org/10.1007/s10957-023-02297-y

    2023 doi

  8. [16]

    D. P. Kingma and J. Ba, Adam: A method for stochastic optimization , preprint, arXiv:1412.6980, 2014

    2014 arXiv

  9. [17]

    Lacotte and M

    J. Lacotte and M. Pilanci , Fast convex quadratic optimization solvers with adaptive sketching-based preconditioners, preprint, arXiv:2104.14101, 2021

    2021 arXiv

  10. [18]

    T. L. Lai and H. Xing , Statistical models and methods for financial markets , Springer, New York, 2008

    2008

  11. [19]

    Lan and Y

    G. Lan and Y. Zhou, An optimal randomized incremental gradient method, Math. Program., 171 (2018), pp. 167–215, https://doi.org/10.1007/s10107-017-1173-0

    2018 doi

  12. [20]

    Li and Z

    H. Li and Z. Lin, On the O √ d T 1/4 convergence rate of RMSProp and its momentum extension measured by l1 norm: Better dependence on the dimension , preprint, arXiv:2402.00389, 2024

    2024 arXiv

  13. [21]

    Ling and T

    S. Ling and T. Strohmer , Self-calibration and bilinear inverse problems via linear least squares , SIAM J. Imag. Sci., 11 (2018), pp. 252–292, https://doi.org/10.1137/16M1103634

    2018 doi

  14. [22]

    J. Liu, D. Xu, H. Zhang, and D. Mandic , On hyper-parameter selection for guaranteed convergence of RMSProp, Cognit. Neurodyn., (2022), https://doi.org/10.1007/s11571-022-09845-8. 30 R. LI, J. XU, AND W. XING

    2022 doi

  15. [23]

    Y. Liu, F. Feng, and W. Yin , Acceleration of SVRG and Katyusha X by inexact preconditioning , in Proceedings of the 36th International Conference on Machine Learning, PMLR, 2019, pp. 4003–4012

    2019

  16. [24]

    Loizou and P

    N. Loizou and P. Richt ´arik, Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods , Comput. Optim. Appl., 77 (2020), pp. 653–710, https: //doi.org/10.1007/s10589-020-00220-z

    2020 doi

  17. [25]

    C. Ma, L. Wu, and W. E , A qualitative study of the dynamic behavior for adaptive gradient algorithms , in Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR, 2022, pp. 671–692

    2022

  18. [26]

    J. D. Moorman, T. K. Tu, D. Molitor, and D. Needell , Randomized Kaczmarz with averaging , BIT Numer. Math., 61 (2021), pp. 337–359, https://doi.org/10.1007/s10543-020-00824-1

    2021 doi

  19. [27]

    Moulines and F

    E. Moulines and F. Bach, Non-asymptotic analysis of stochastic approximation algorithms for machine learning, in Advances in Neural Information Processing Systems, 2011, pp. 451–459

    2011

  20. [28]

    Naseem, R

    I. Naseem, R. Togneri, and M. Bennamoun , Linear regression for face recognition , IEEE Trans. Pattern Anal. Mach. Intell., 32 (2010), pp. 2106–2112, https://doi.org/10.1109/TPAMI.2010.128

    2010 doi

  21. [29]

    Needell, N

    D. Needell, N. Srebro, and R. W ard , Stochastic gradient descent, weighted sampling, and the ran- domized Kaczmarz algorithm , Math. Program., 155 (2016), pp. 549–573, https://doi.org/10.1007/ s10107-015-0864-7

    2016

  22. [30]

    Paszke , Pytorch: An imperative style, high-performance deep learning library , preprint arXiv:1912.01703, 2019

    A. Paszke , Pytorch: An imperative style, high-performance deep learning library , preprint arXiv:1912.01703, 2019

    1912 arXiv

  23. [31]

    Richt´arik and M

    P. Richt´arik and M. Tak´ac, Stochastic reformulations of linear systems: Algorithms and convergence theory, SIAM J. Matrix Anal. Appl., 41 (2020), pp. 487–524, https://doi.org/10.1137/18M1179249

    2020 doi

  24. [32]

    Strakoˇs, On the real convergence rate of the conjugate gradient method , Linear Algebra Appl., 154 (1991), pp

    Z. Strakoˇs, On the real convergence rate of the conjugate gradient method , Linear Algebra Appl., 154 (1991), pp. 535–549, https://doi.org/10.1016/0024-3795(91)90393-B

    1991 doi

  25. [33]

    Strohmer and R

    T. Strohmer and R. Vershynin , A randomized Kaczmarz algorithm with exponential convergence , J. Fourier Anal. Appl., 15 (2009), pp. 262–278, https://doi.org/10.1007/s00041-008-9030-4

    2009 doi

  26. [34]

    Tieleman and G

    T. Tieleman and G. Hinton , Lecture 6.5 RMSProp: Divide the gradient by a running average of its recent magnitude, COURSERA: Neural networks for machine learning, 4 (2012), pp. 26–31

    2012

  27. [35]

    J. A. Tropp, An Introduction to Matrix Concentration Inequalities , now Publishers Inc., Hanover, 2015

    2015

  28. [36]

    W ang, M

    X. W ang, M. Johansson, and T. Zhang, Generalized Polyak step size for first order optimization with momentum, in Proceedings of the 40th International Conference on Machine Learning, PMLR, 2023, pp. 35836–35863

    2023

  29. [37]

    W ard, X

    R. W ard, X. Wu, and L. Bottou , Adagrad stepsizes: Sharp convergence over nonconvex landscapes , J. Mach. Learn. Res., 21 (2020), pp. 1–30

    2020

  30. [38]

    X. Wu, Y. Xie, S. S. Du, and R. W ard, Adaloss: A computationally-efficient and provably convergent adaptive gradient method , in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8691–8699, https://doi.org/10.1609/aaai.v36i8.20848

    2022 doi

  31. [39]

    Y. Xie, X. Wu, and R. W ard, Linear convergence of adaptive stochastic gradient descent, in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 1475– 1485

    2020

  32. [40]

    Y. Zeng, D. Han, Y. Su, and J. Xie, On adaptive stochastic heavy ball momentum for solving linear sys- tems, SIAM J. Matrix Anal. Appl., 45 (2024), pp. 1259–1286, https://doi.org/10.1137/23M1575883

    2024 doi

  33. [41]

    Zhang, Y

    Z. Zhang, Y. Li, L. Li, Z. Li, and S. Liu , Multiple linear regression for high efficiency video intra coding, in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2019, pp. 1832–1836

    2019

  34. [42]

    F. Zou, L. Shen, Z. Jie, W. Zhang, and W. Liu , A sufficient condition for convergences of Adam and RMSProp , in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Los Alamitos, 2019, IEEE Computer Society, pp. 11119–11127

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.