Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models

Aleksandr Katrutsa; Evgeny Frolov; Tatyana Matveeva

REVIEW 4 major objections 4 minor 21 references

Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AdaGram computes full-matrix AdaGrad-style updates by tracking a rank-r inverse-Cholesky factor, and its experiments show that rank 1–5 factors match or beat diagonal optimizers on correlated logistic regression tasks.

desk verdict Neat combination of a known recursive identity with a projector-splitting integrator, but the empirical evidence is thin and the low-rank truncation error is never analyzed. read the letter →

arxiv 2508.21106 v1 pith:VSST7PJK submitted 2025-08-28 cs.LG cs.AI

Tatyana Matveeva , Aleksandr Katrutsa , Evgeny Frolov This is my paper · ORCID

classification cs.LGcs.AI

keywords AdaGramfull-matrixpreconditioningadaptivegradientmethodslow-rankapproximationprojector-splittingintegratorCholeskyfactorizationlogisticregressionsecond-orderoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes AdaGram, an optimizer that aims to make full-matrix adaptive gradient updates practical by maintaining only a low-rank factor of the inverse Cholesky factor of the accumulated gradient matrix. The key mathematical claim is a recursive identity: the preconditioned gradient at step t+1 is a scalar multiple of the preconditioned gradient at step t, with the scalar determined entirely by the norm of the current transformed gradient. Because the inverse-Cholesky factor itself is updated by a projector-splitting low-rank integrator, the method avoids O(n^3) inversions and O(n^2) storage. If correct, the correlation information that full-matrix preconditioning uses can be obtained at a fraction of the cost, and the experiments on logistic regression with up to 60 features support that claim.

What carries the argument

The load-bearing object is the inverse Cholesky factor L^{-1}_t of the accumulated gradient matrix G_t = εI + Σ g_τ g_τ^T, represented as (I − P_t Q_t^T)L_0^{-1}. Theorem 1's recursion turns preconditioning into a scalar-normalized update of the previous preconditioned gradient; Lemma 1 converts that recursion into growing P and Q factors; and Algorithm 1, the projector-splitting integrator for dynamical low-rank approximation, truncates A_t = P_t Q_t^T to rank r at each step. Together these replace exact matrix inversion with low-rank SVD/QR updates, reducing memory and per-step cost from quadratic in n to roughly linear in n times r.

What would settle it

Run AdaGram at rank 1, 5, and 10 against exact full-matrix AdaGrad on a logistic regression with roughly 10^5 features and designed pairwise correlations; if the loss-versus-time curves diverge with dimension, or if the spectral norm of L^{-1}_t − (I − P_tQ_t^T)L_0^{-1} grows without bound along a fixed trajectory, the rank-truncation assumption fails.

Watch

Extended reading notes

Core claim

AdaGram computes AdaGrad's full-matrix direction θ ← θ − ηG^{-1/2}g without forming n×n G. It uses the inverse Cholesky factor L^{-1} and the identity L^{-1}_{t+1} g_{t+1} = (1+||L^{-1}_t g_{t+1}||^2)^{-1/2} L^{-1}_t g_{t+1} (Theorem 1), so each preconditioned gradient follows from the previous one and its norm. Lemma 1 stores L^{-1}_t as (I − P_tQ_t^T)L_0^{-1}; the projector-splitting integrator keeps P_tQ_t^T at fixed rank r. Experiments on synthetic and UCI logistic regression show rank-1–5 AdaGram matching or beating diagonal AdaGrad, Shampoo, KATE, and SGD on correlated losses.

Load-bearing premise

The rank-r truncation done by the projector-splitting integrator is accurate enough to preserve the convergence benefit of full-matrix preconditioning; the paper gives no error bound and only tests up to 60 features.

Editorial extensions

If this is right

Full-matrix AdaGrad-style updates become feasible for models where O(n^2) storage is unacceptable, because memory and per-iteration cost scale with n times r rather than n^2.
The recursion identity means the preconditioned gradient can be computed from the previous step's transformed gradient and its norm, so no matrix square root needs to be recomputed at each step.
Rank r acts as a dial between diagonal preconditioning and full-matrix behavior, with the paper's experiments finding best performance at very low ranks (1–2 on real datasets).
The low-rank integrator viewpoint gives a principled way to 'forget' old gradient directions as training progresses, since the magnitude of each gradient update controls how strongly the stored factor changes.
The same machinery can be attached to other adaptive methods that use gradient second-moment matrices, such as Adam- and RMSProp-style updates, replacing diagonal estimates with low-rank full-matrix estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test whether the rank-r truncation error stays bounded as the feature dimension grows beyond 60; a direct high-dimensional experiment with strong correlations would reveal whether the low-rank preconditioner keeps tracking exact full-matrix AdaGrad.
If truncation error does accumulate, a remedy not explored in the paper is a periodic full-rank refresh (recomputing L^{-1} exactly every K steps) to keep the low-rank factors from drifting away from the true preconditioner.
The scalar-normalized recursion in Theorem 1 could be useful for distributed or Hessian-free solvers, since exchanging only rank-r factors or preconditioned gradient vectors avoids communicating full matrices.
Existing techniques for error control in dynamical low-rank approximation could supply the missing bound and turn the rank-truncation heuristic into a certified algorithm with per-step error guarantees.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this paper combines a known rank-1 update identity for the inverse Cholesky factor with a projector-splitting integrator to keep a rank-r factorization of the preconditioner. That combination is new, and the derivation of the recursive update (Theorem 1) is correct. But the evidence for the central claim—that rank 5 or lower is enough—is thin, and the truncation error from the projector-splitting step is never bounded. The experiments stop at 60 features, so the abstract's 'scalable solution for large models' language is not supported.

What it does well: the proof is self-contained, including the appendix, and the isometry argument that lets them switch from the symmetric square root to the Cholesky factor is clean. The algorithm is simple to implement, and the code is available. The connection to dynamical low-rank approximation is a legitimate new idea, even if the pieces are not new on their own.

Soft spots: the projector-splitting algorithm is taken from an ODE integrator but applied with a fixed step h=1 to a discrete update. There is no error analysis for this step, and since the same approximate preconditioned gradient is used in the normalization on line 7 of Algorithm 2, the error also perturbs the effective learning rate. That's a real gap. The experiments are on three small UCI datasets with hyperparameters selected per method to minimize final training loss; there are no repeated runs or variance estimates. The synthetic experiments are fine but do not rescue the scale claim. Also, minor citation issues: ref [12] for SOAP appears to point to a different paper, and ref [10] does not match the text describing online matrix estimation.

Bottom line: this is a reasonable idea that deserves a serious look, but the paper currently overclaims. The right referee would ask for larger experiments with statistical replicates and at least an informal argument about the truncation error. I'd take it to peer review, with the expectation of major revision.

Referee Report

4 major / 4 minor

Summary. The paper proposes AdaGram, an optimizer that maintains a low-rank approximation of the inverse Cholesky factor of the accumulated outer-product gradient matrix G_t, with the goal of realizing full-matrix adaptive preconditioning at reduced memory and computational cost. The theoretical core is Theorem 1, which gives an exact recurrence for the action of the inverse Cholesky factor on the current gradient: L_{t+1}^{-1} g_{t+1} = (1 + ||L_t^{-1} g_{t+1}||^2)^{-1/2} L_t^{-1} g_{t+1}. Because the exact representation grows unbounded in rank, the authors use a projector-splitting integrator (Algorithm 1, from Lubich--Oseledets) to keep a rank-r factorization, controlled by a memory parameter µ. The method is evaluated on synthetic correlation-structured data and three UCI logistic-regression benchmarks (Heart, Australian, Splice), comparing against SGD, diagonal AdaGrad, Shampoo, KATE, and full AdaGrad.

Significance. The exact recurrence in Theorem 1 is clean, self-contained, and could be of independent use, and the connection to dynamical low-rank integrators is original. The development from Eq. (3) to Eq. (7) is a useful observation that avoids forming G_t explicitly. The paper also makes code available, which aids reproducibility. However, the central empirical claim---that rank 1--5 AdaGram variants match or beat diagonal adaptive optimizers---rests on a small and narrowly tuned experimental base, and the algorithm's interpretation as a full-matrix AdaGrad method is not fully supported. If the low-rank truncation were supplied with an error bound or a descent/convergence guarantee, this would be a solid contribution; as it stands, it is mainly an interesting heuristic with a clean exact recurrence behind it.

major comments (4)

[§3.3, Eq. (7) and §3.2] The recursive update computes L^{-1} g, not G^{-1/2} g. The isometry argument in §3.2 preserves only the norm (g^T G^{-1} g = ||L^{-1} g||^2), not the direction of the preconditioned gradient. For positive definite G = L L^T, L^{-1} and G^{-1/2} differ by an orthogonal factor, so the update is a rotated version of AdaGrad. This rotation can turn the step into an ascent direction. Example: take L = [[1, 0], [0.9, 0.1]], G = L L^T (positive definite), and g = (1, 0.2); then g^T L^{-1} g = -0.4 < 0. Thus AdaGram is not equivalent to full-matrix AdaGrad, and the comparisons to FullAdaGrad in §4 compare two different optimizers. The authors should either prove a descent/convergence property for the L^{-1} update or revise the claim that AdaGram enables full-matrix adaptive gradient updates.
[§3.4, Eq. (13), Algorithm 1] The rank-r truncation is uncontrolled. Algorithm 1 is a projector-splitting integrator designed for an ODE; here it is applied with a fixed step h=1 to a discrete rank-(r+1) update, with no error analysis. The increment ΔA_{t+1} in Eq. (12) is rank-one but not small in norm, and the approximation error in A_t feeds directly into the normalization 1/sqrt(1+||\bar g_t||^2) on line 7 of Algorithm 2. Without an error bound or a stability analysis, there is no guarantee that the low-rank method tracks the exact recurrence of Lemma 1 or preserves even the (problematic) direction of the exact update.
[§4.2, §4.4] The experimental evidence is too narrow for the abstract's claim. Hyperparameters are selected per method by minimizing the final training loss, which is a training-time selection criterion and can bias comparisons, especially for AdaGram with the extra rank r and memory µ hyperparameters. No run-to-run variance or error bars are reported. The largest real dataset (Splice) has only 60 features. Demonstrating that rank 1--2 works on these three problems does not establish that rank-five (or smaller) approximations are sufficient for scalable full-matrix preconditioning in large models.
[Algorithm 2] The algorithm as written is underspecified and contains a dimension inconsistency. Line 1 sets L_0 = sqrt(ε) I_m, while gradients are in R^{mn}; if the weight matrix is m×n, the preconditioner should act on R^{mn}. The initial P, Q, U, S, V are not defined, and it is not stated how the memory parameter µ enters Algorithm 1's steps (Eq. (14) gives A_{t+1} = µ A_t + (1-µ) ΔA_{t+1}, but Algorithm 1 expects a rank-r factorization and an increment). These details are necessary for a reproducible optimizer.

minor comments (4)

[Eq. (5)] The parameter α is defined through 1 + α||\bar g||^2 = sqrt(1+||\bar g||^2); when ||\bar g||=0 the formula for α involves division by zero. The zero case should be handled separately.
[Appendix A.3] The proof of Lemma 1 contains a wrong algebra line: 'P_t Q_t^T \bar g_{t+1} = Q_t(P_t^T \bar g_{t+1})' should be P_t(Q_t^T \bar g_{t+1}). The lemma statement can be verified independently, but the proof as printed is not reliable.
[§4.3, Figure 2] The text states that the optimal rank for AdaGramPS is consistently rank 1 or 2, but Figure 2 plots AdaGramPS rank 3, 4, and 5. Clarify which ranks were actually selected by grid search and which are shown for comparison.
[Algorithm 2] The time indexing is off by one relative to Eq. (3). At iteration t, the preconditioner G_t includes g_t, but the algorithm computes \bar g_t = L_{t-1}^{-1} g_t and then normalizes; it would help to state explicitly that the update uses L_t^{-1} g_t following Theorem 1.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; derivation is self-contained, with minor non-load-bearing self-citations.

full rationale

The central recursion (Theorem 1) is a mathematical identity proven from the Cholesky update G_{t+1}=G_t+g_{t+1}g_{t+1}^T using the Sherman-Morrison formula; it is not assumed as the definition of the preconditioner. The low-rank update is taken from the external projector-splitting integrator of [18], and the paper does not claim error bounds for the rank-r truncation—this is a rigor/scalability gap, not circularity. Algorithm 2 applies Eq. 7 using the approximate factor, which is the intended algorithm construction, not a fitted quantity relabeled as a prediction. Experimental comparisons are against external baselines on UCI datasets; rank and memory parameter µ are tuned by grid search and then evaluated, which is standard model selection rather than circular prediction. Self-citations [3] and [21] appear only for background and synthetic correlation generation respectively; they do not support the central claim. Thus no load-bearing circular step exists.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. The central parameters are the rank and the memory weight, both tuned by grid search. The key mathematical backbone is a known factorization identity combined with a cited low-rank integrator; the paper contributes the algorithmic combination.

free parameters (4)

rank r = 1-5 (from grid search)
Controls the rank of the low-rank preconditioner. Optimal rank was selected per dataset to minimize final training objective.
memory parameter µ = grid-selected, not reported
Weights the relative contribution of historical and current gradient information in equation (14). Chosen by grid search.
learning rate η = grid-selected, not reported
Scales the parameter update in Algorithm 2. Chosen per dataset by grid search.
numerical stability parameter ε = grid-selected, not reported
Diagonal shift in G_0 = εI to ensure invertibility. Chosen by grid search.

assumptions (3)

domain assumption The fast symmetric factorization of I + gg^T from [8] produces a valid factor P = I + αgg^T with α satisfying equation (5).
Used to derive equation (8) and Theorem 1. Cited from [8]; the paper provides a proof sketch in Appendix A.2 but relies on the cited result.
domain assumption The projector-splitting integrator (Algorithm 1, from [18]) provides a sufficiently accurate rank-r approximation of the evolving matrix A_t = P_t Q_t^T.
Core of the low-rank maintenance; no error bound is provided in this paper, and the approximation quality is only evaluated indirectly through final loss and accuracy.
ad hoc to paper The optimizer update with the approximate low-rank preconditioner remains convergent; truncation does not break the descent property.
No convergence analysis is given. The paper assumes the truncated preconditioner still yields useful update directions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models." pith.science (2026). https://pith.science/paper/VSST7PJK

@misc{pith2026250821106,
  author       = {Pith},
  title        = {Pith review of: Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSST7PJK}},
  note         = {Machine review of arXiv:2508.21106}
}

read the original abstract

Adaptive gradient methods like Adagrad and its variants are widespread in large-scale optimization. However, their use of diagonal preconditioning matrices limits the ability to capture parameter correlations. Full-matrix adaptive methods, approximating the exact Hessian, can model these correlations and may enable faster convergence. At the same time, their computational and memory costs are often prohibitive for large-scale models. To address this limitation, we propose AdaGram, an optimizer that enables efficient full-matrix adaptive gradient updates. To reduce memory and computational overhead, we utilize fast symmetric factorization for computing the preconditioned update direction at each iteration. Additionally, we maintain the low-rank structure of a preconditioner along the optimization trajectory using matrix integrator methods. Numerical experiments on standard machine learning tasks show that AdaGram converges faster or matches the performance of diagonal adaptive optimizers when using rank five and smaller rank approximations. This demonstrates AdaGram's potential as a scalable solution for adaptive optimization in large models.

Figures

Figures reproduced from arXiv: 2508.21106 by the authors.

**Figure 1.** Correlation matrices The performance of our proposed methods on synthetic data with varying correlation structures is presented in Figure 2. As expected, on isotropic data where feature correlations are absent, our methods are outperformed by simpler optimizers like AdaGrad and SGD. However, as correlations are introduced, the advantages of our approach become apparent. On data with simple correlations, our AdaGram… view at source ↗

**Figure 2.** The plots compare test loss versus training epochs on three types of data: (left) isotropic data with no feature [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Comparison of optimization algorithms on the Heart, Australian, and Splice datasets. The proposed Ada [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Performance of optimizers on the Heart, Australian, and Splice datasets. Our proposed methods (Ada [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

[12]

SOAP: One-shot pruning of generative pre-trained transformer checkpoints with a second-order method, 2024

Abhimanyu Vyas, Andrea Vedaldi, and Andrew Zisserman. SOAP: One-shot pruning of generative pre-trained transformer checkpoints with a second-order method, 2024

work page 2024
[10]

Ravikumar

Prateek Jain, Pavitra Krishnaswamy, Prateek Mehta, and P. Ravikumar. Neural taylor approximations: Con- vergence and exploration in rectifier nets. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 2710–2719. PMLR, 2018

work page 2018
[1]

A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights

Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1–43, 2016

work page 2016
[2]

A lyapunov analysis for accelerated gradient methods: From determinis- tic to stochastic case

Maxime Laborde and Adam Oberman. A lyapunov analysis for accelerated gradient methods: From determinis- tic to stochastic case. In International conference on artificial intelligence and statistics, pages 602–612. PMLR, 2020

work page 2020
[3]

NAG-GS: Semi-implicit, accelerated and robust stochastic optimizer

Valentin Leplat, Daniil Merkulov, Aleksandr Katrutsa, Daniel Bershatsky, Olga Tsymboi, and Ivan Oseledets. NAG-GS: Semi-implicit, accelerated and robust stochastic optimizer. arXiv preprint arXiv:2209.14937, 2022

arXiv 2022
[4]

A method of solving a convex programming problem with convergence rate O(1/k2)

Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k2). Soviet Mathematics Doklady, 27:372–376, 1983

work page 1983
[5]

Adaptive subgradient methods for online learning and stochastic optimization

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011

2011
[6]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015

work page 2015

Show all 21 references

[7]

Hinton, Nitish Srivastava, and Kevin Swersky

Geoffrey E. Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning, lecture 6e, rmsprop. Coursera, 2012

2012
[8]

Fast symmetric factorization of hierarchical matrices with applications

Sivaram Ambikasaran, Michael O’Neil, and Karan Raj Singh. Fast symmetric factorization of hierarchical matrices with applications. arXiv preprint arXiv:1405.0223, 2014

2014 arXiv
[9]

Buhmann, and Nicolai Meinshausen

Gabriel Krummenacher, Brian McWilliams, Yannic Kilcher, Joachim M. Buhmann, and Nicolai Meinshausen. Scalable adaptive stochastic optimization using random projections, 2016

2016
[11]

Shampoo: Preconditioned stochastic tensor optimization, 2018

Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018

2018
[13]

Efficient full-matrix adaptive regularization, 2020

Naman Agarwal, Brian Bullins, Xinyi Chen, Elad Hazan, Karan Singh, Cyril Zhang, and Yi Zhang. Efficient full-matrix adaptive regularization, 2020

2020
[14]

Remove that square root: A new efficient scale-invariant version of adagrad

Sayantan Choudhury, Nazarii Tupitsa, Nicolas Loizou, Samuel Horv ´ath, Martin Takac, and Eduard Gorbunov. Remove that square root: A new efficient scale-invariant version of adagrad. Advances in Neural Information Processing Systems, 37:47400–47431, 2024

2024
[15]

Matthew D. Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012

2012 arXiv
[16]

Incorporating nesterov momentum into adam

Timothy Dozat. Incorporating nesterov momentum into adam. Technical report, Stanford University, 2016

2016
[17]

Fast low-rank modifications of the thin singular value decomposition

Matthew Brand. Fast low-rank modifications of the thin singular value decomposition. Linear Algebra and its Applications, 415(1):20–30, May 2006

2006
[18]

A projector-splitting integrator for dynamical low-rank approximation, 2013

Christian Lubich and Ivan Oseledets. A projector-splitting integrator for dynamical low-rank approximation, 2013

2013
[19]

J. A. Nelder and R. W. M. Wedderburn. Generalized linear models. Journal of the Royal Statistical Society. Series A (General), 135(3):370–384, 1972. Stable URL: https://www.jstor.org/stable/2344614

1972
[20]

UCI machine learning repository, 2017

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017

2017
[21]

Stress test procedure for feature selection algorithms

AM Katrutsa and VV Strijov. Stress test procedure for feature selection algorithms. Chemometrics And Intelli- gent Laboratory Systems, 142:172–183, 2015. 9 A PREPRINT A. Appendix A.1 Algorithms Algorithm 3 AdaGrad, matrix case Require: η >0, ϵ >0 1: Initialize W1 = 0m×n; G0 = ...

2015

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [12]

SOAP: One-shot pruning of generative pre-trained transformer checkpoints with a second-order method, 2024

Abhimanyu Vyas, Andrea Vedaldi, and Andrew Zisserman. SOAP: One-shot pruning of generative pre-trained transformer checkpoints with a second-order method, 2024

work page 2024

[2] [10]

Ravikumar

Prateek Jain, Pavitra Krishnaswamy, Prateek Mehta, and P. Ravikumar. Neural taylor approximations: Con- vergence and exploration in rectifier nets. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 2710–2719. PMLR, 2018

work page 2018

[3] [1]

A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights

Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17(153):1–43, 2016

work page 2016

[4] [2]

A lyapunov analysis for accelerated gradient methods: From determinis- tic to stochastic case

Maxime Laborde and Adam Oberman. A lyapunov analysis for accelerated gradient methods: From determinis- tic to stochastic case. In International conference on artificial intelligence and statistics, pages 602–612. PMLR, 2020

work page 2020

[5] [3]

NAG-GS: Semi-implicit, accelerated and robust stochastic optimizer

Valentin Leplat, Daniil Merkulov, Aleksandr Katrutsa, Daniel Bershatsky, Olga Tsymboi, and Ivan Oseledets. NAG-GS: Semi-implicit, accelerated and robust stochastic optimizer. arXiv preprint arXiv:2209.14937, 2022

arXiv 2022

[6] [4]

A method of solving a convex programming problem with convergence rate O(1/k2)

Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k2). Soviet Mathematics Doklady, 27:372–376, 1983

work page 1983

[7] [5]

Adaptive subgradient methods for online learning and stochastic optimization

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011

2011

[8] [6]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015

work page 2015