Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization

Corrado Coppola; Irene Amerini; Laura Palagi; Lorenzo Papa

REVIEW 3 major objections 6 minor 92 references

Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read F-CMA, a random-reshuffling optimizer with a derivative-free line search, claims deterministic convergence to a stationary point for nonconvex finite-sum problems without the memory overhead of adaptive gradient methods.

desk verdict F-CMA has a genuinely new line-search minibatch design and an honest empirical sweep, but the main convergence theorem is not proven as written: the learning-rate sequence can increase, and the final stationarity argument has a sign/evaluation error. read the letter →

arxiv 2411.15795 v3 pith:RNDOEFIH submitted 2024-11-24 cs.LG math.OC

Corrado Coppola , Lorenzo Papa , Irene Amerini , Laura Palagi This is my paper · ORCID

classification cs.LGmath.OC

keywords nonconvexoptimizationmini-batchrandomreshufflinglinesearchconvergencetheorydeeplearningcomputervision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

F-CMA is a mini-batch optimizer for the unconstrained minimization of a sum of smooth, possibly non-convex loss functions. Its central claim is a deterministic global convergence guarantee: even though samples are randomly reshuffled every epoch, the learning rate is adjusted by a sufficient-decrease test and a derivative-free line search, and the sequence of iterates admits a stationary limit point with inf_k ‖∇f(w_k)‖ → 0. The paper argues this needs neither convexity of the objective nor a gradient-related search direction, and replaces the individual coercivity assumption of prior controlled mini-batch methods with gradient clipping. If correct, the method delivers a convergence guarantee comparable to adaptive methods while using less memory, and the reported experiments claim training time cuts of up to 68%, per-epoch efficiency gains up to 20%, and accuracy gains up to 5% on CIFAR-10 and CIFAR-100.

What carries the argument

The load-bearing object is the learning-rate sequence {ζ_k} paired with the per-epoch sufficient-decrease test at Step 6 of Algorithm 3. Each epoch runs random reshuffling (Algorithm 1) producing a tentative point w̃^k, a direction d_k accumulating the per-batch gradients, and an estimate f̃^k of f; if f̃^{k+1} ≤ min{φ_k − γ ζ_k, f(w0)} the point is accepted unchanged, otherwise the algorithm checks whether ‖d_k‖ ≤ τ ζ_k and, failing that, launches the derivative-free line search (DFL, Algorithm 2). DFL scales ζ_k by η, extrapolates a step by repeatedly dividing by δ while a cheap surrogate model ψ satisfies an Armijo-type reduction, and then evaluates the true objective once; the returned step α̃_k and threshold τ decide whether ζ_{k+1} is left unchanged, decreased by θ, or set to max{α̃_k, α_min}. The proof mechanics are: Lemma 3 bounds |f̃^{k+1} − f(w_k)| by $P^{2}$ L_f ζ_k(CM + D), so the estimate tracks the true function as ζ_k shrinks; Assumption 4 (a growth condition, enforced by gradient clipping) keeps gradients bounded; coercivity keeps iterates in compact level sets; and Proposition 5's claimed monotone decrease ζ_k → 0 lets RR's cumulative direction approach the full gradient, which via Proposition 1 yields stationarity.

What would settle it

Run F-CMA on a smooth nonconvex problem with logging of ζ_k, and check whether any iteration at Step 16 sets ζ_{k+1} = max{α̃_k, α_min} with α̃_k > ζ_k; a single such increase contradicts the monotone-decrease premise of Proposition 5, and a run where ζ_k does not converge to zero would directly refute the claimed theorem.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the controlled mini-batch idea can be made faster and memory-lighter while keeping a deterministic convergence proof. For the nonconvex finite-sum problem min_w f(w) = Σ_i f_i(w), the authors prove (Proposition 6) that the points produced by F-CMA admit limit points and that at least one is stationary, i.e., inf_k ‖∇f(w_k)‖ → 0. The mechanism is a learning-rate control loop: after each random-reshuffling epoch, the accumulated batch-loss estimate f̃^k must satisfy a sufficient decrease bound; if it does not, the algorithm either shrinks the learning rate or invokes a derivative-free line search (DFL) that uses an arbitrary approximation model ψ of f to extrapolate a trial step, verifying it on the true objective. The convergence argument relies on three lemmas: the gap between the estimated and true objective is O(ζ_k) (Lemma 3), the iterates stay bounded (Proposition 4), and the learning rate tends to zero (Proposition 5).

Load-bearing premise

The convergence theorem stands on the claim that the learning rate never increases, yet the algorithm's line-search branch can raise it, and if that happens the proof that the learning rate tends to zero no longer goes through.

Editorial extensions

If this is right

If the convergence theorem holds, deep networks trained with mini-batch random reshuffling can be certified to reach a stationary point deterministically, closing part of the gap between adaptive-method practice and convergence theory.
The gradient-clipping substitution for individual coercivity lets the theory apply to standard training pipelines without per-component lower-bound assumptions.
The learning-rate-driven early stopping rule (ε threshold) yields an automatic stopping signal, which the authors tie to lower energy consumption and carbon footprint during training.
Because ψ can be any approximation model, the line search can be decoupled from the exact loss, opening the scheduler to cheaper surrogates while preserving the true-objective verification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the authors do not draw: if F-CMA's monotone learning-rate premise fails (Step 16 can raise ζ_{k+1} above ζ_k), the proof of ζ_k → 0 is not supported, and one should instrument a run to see whether ζ_k ever increases in practice.
The 68% time reduction is reported as total training time with the early-stopping rule; a stricter comparison that fixes the number of epochs, or reports time-to-accuracy curves, would separate the effect of the optimizer's step quality from the effect of stopping earlier.
The surrogate model ψ could be a subsampled batch; using the same permutation for ψ and the true check might couple the two evaluations, so an ablation varying the size of ψ would clarify how much of the line-search benefit comes from the proxy.
Because the proof only needs a coercive approximation ψ, other cheap models could be plugged into DFL without changing the convergence argument.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper is not a crank submission: F-CMA is a concrete optimizer that extends the controlled-minibatch line (CMA/CMAL) with a derivative-free extrapolation line search, an internal η scheduler, and gradient clipping in place of per-function coercivity. The line search uses at most two full objective evaluations, and the early-stopping rule conditioned on the learning rate is a sensible energy-saving idea. The authors also ship code and run a 54-combination benchmark across six architectures and two datasets. That is real work.

Second, the central proof that the method converges to a stationary point does not hold as written. Proposition 5 claims ζ_k is monotonically non-increasing because each step either leaves it or multiplies by θ<1. But Algorithm 3 Step 16 sets ζ_{k+1}=max{α~_k, α_min} whenever the DFL step size fails the small-direction test, and the DFL can return α~_k > ζ_k. So ζ can increase, the dichotomy is false, and the argument that ζ_k→0 collapses. Proposition 4's boundedness proof explicitly leans on ζ being non-increasing, so that conclusion is also unsupported. Proposition 6 then applies a mean-value theorem with a sign error and evaluates the gradient at w_k instead of an intermediate point, so the claimed contradiction ∇f(w̄)=0 doesn't follow. These are not typos; they are load-bearing.

The empirical section is also weaker than the abstract implies. The headline time savings come mostly from early stopping (F-CMA stops at 78-122 epochs while baselines run 250), not from per-epoch efficiency—Table 2 shows T/Ep for F-CMA is comparable to Adam and worse than SGD. There are no error bars, only one run per configuration, and the comparison omits the line-search optimizers cited in Section 2. The accuracy gains are modest and within what single-seed noise could produce.

On the positive side, the work is not circular: no fitted constants enter the theorem, and the paper does not define the target result into existence. The code link is a plus. But a proof that fails at two consecutive propositions cannot support a deterministic convergence claim. I would want the authors to fix the learning-rate update (or the proof) and redo the experiments with multiple seeds and proper baselines before believing the main conclusions.

For a journal, I would send this to review rather than desk-reject outright — the algorithmic idea is worth an expert look and the flaws are specific enough for a referee to pinpoint. But my own recommendation is reject in current form; it needs major revision. Reading group: maybe, as a case study in checking algorithm/proof consistency.

Referee Report

3 major / 6 minor

Summary. The manuscript introduces F-CMA, a random-reshuffling minibatch algorithm for the nonconvex finite-sum problem (1)/(3). It combines a per-epoch sufficient-decrease check with a derivative-free line search (DFL) that uses a coercive surrogate model and at most two evaluations of the true objective. The authors claim a deterministic global convergence result, inf_k ||∇f(w_k)||→0, without convexity of f or a gradient-related search direction, and they report experiments on CIFAR-10 and CIFAR-100 with six architectures showing up to 68% training-time reduction, up to 20% per-epoch efficiency gain, and up to 5% accuracy improvement over eight baselines. The paper also proposes a learning-rate-based early stopping rule and argues that the method has lower memory requirements than adaptive optimizers.

Significance. If the proof were correct, F-CMA would be a notable contribution: a line-search optimizer for deep networks with a deterministic nonconvex convergence guarantee, lower memory than adaptive methods, and a derivative-free search that can reuse any surrogate model. The empirical campaign is broad (54 model-optimizer combinations, six architectures), the code is promised in a public repository, and the theoretical analysis is self-contained, with no fitted constants entering the theorem. These strengths are real. However, the convergence proof contains load-bearing errors: the claimed monotonicity of the step-size sequence is contradicted by the algorithm itself, and the final stationarity argument has sign and evaluation-point mistakes. These flaws invalidate the paper's central theoretical claim as it stands.

major comments (3)

[Section 3.3, Proposition 5; Algorithm 3 Step 16] Proposition 5 asserts that {ζ_k} is monotonically non-increasing because at every iteration either ζ_{k+1}=ζ_k or ζ_{k+1}=θζ_k<ζ_k. This dichotomy is incomplete: when the condition at Step 13 of Algorithm 3 fails, Step 16 sets ζ_{k+1}=max{α~_k, α_min}. Algorithm 2 (DFL) can return α~_k>ζ_k because it initializes α=ζ_kη and repeatedly multiplies by 1/δ>1, while the failure of Step 13 only requires α~_k||d_k||²>τζ_k and is compatible with α~_k>ζ_k. Hence ζ_k can increase, the claimed monotonicity fails, and the contradiction argument showing ζ_k→0 collapses. Since Proposition 1 requires ζ_k→0 and Proposition 4 explicitly uses the monotonicity to prove boundedness of {w_k}, the global convergence theorem (Proposition 6) is not established.
[Section 3.3, Proposition 6] The proof of Proposition 6 contains multiple sign and evaluation-point errors. In the case where ¯K′ is infinite, the proof compares the vector d_k with the scalar τζ_k in an inequality, which is not a valid vector inequality; the stationarity conclusion requires passing to norms. In the case where ¯K′′ is infinite, the proof states that ˜α_k≤0, but the line-search return ˜α_k is a nonnegative step size. The Mean-Value theorem is also misapplied: the correct identity is f(w_k+˜α_k d_k)-f(w_k)=˜α_k ∇f(w_k+ξ_k d_k)^T d_k with ξ_k∈(0,˜α_k), and the Armijo inequality gives ∇f(w_k+ξ_k d_k)^T d_k ≤ -γ||d_k||², not ∇f(w_k)^T d_k ≤ γ||d_k||² as written. Once the signs and evaluation points are corrected, the displayed limit argument does not yield ||∇f(w̄)||=0 without additional work; as printed, the proof is invalid.
[Section 3.3, Proposition 4] The boundedness proof depends crucially on the assertion that '{ζ_k} is by definition a non-increasing sequence' when taking limits along a divergent subsequence. Because that monotonicity is false (see the comment on Proposition 5), the boundedness of {w_k} is not proved. Independently, even under a monotonicity assumption, the proof replaces ζ_k by its limit ζ̄ in an upper bound of the form f(w_k)≤f(w0)+P²L_fζ_k(CM+D); since a non-increasing sequence satisfies ζ_k≥ζ̄, the correct upper bound would use the initial value ζ_0, so the displayed inequality has the wrong direction.

minor comments (6)

[Section 3.3, Proposition 1] Proposition 1 states 'lim_{k→0}ζ_k=0'; the limit should be as k→∞. It also says the sequence {w_k} is produced by Algorithm 1, but the relevant generating method is Algorithm 3.
[Section 3.3, Proposition 5] In the proof of Proposition 5 the text refers to 'Step 17' as the case where the algorithm sets ζ_{k+1}, whereas the instruction that sets ζ_{k+1}=max{α~_k,α_min} is Step 16; Step 17 is only the closing 'end if'.
[Section 3.1, Assumption 3] Assumption 3 is labeled 'f_p bounded below' but defines compact sublevel sets, which is a coercivity condition; Proposition 5 then uses Assumption 3 to conclude φ_k≥0, which requires a specific lower bound (such as f≥0) that is not stated.
[Section 3.1, Lemma 3] In the proof of Lemma 3, the chain of inequalities introduces an extra factor C: from Assumption 4 and ||∇f||≤M the correct bound is ||∇f_p||≤CM+D, so the middle expression should be L_fζ_kp(C||∇f||+D), not L_fζ_kpC(||∇f||+D).
[Equation (16)] Equation (16) contains the unexplained placeholder '=CO'; this appears to be a typesetting artifact of the bound that follows in the proof of Proposition 1.
[Section 4] The dataset names are typeset as 'CIF AR10' and 'CIF AR100' in several sentences; they should be 'CIFAR-10' and 'CIFAR-100'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the convergence proof is self-contained, though Proposition 5 contains a non-circular proof gap.

full rationale

The paper's derivation chain is self-contained with respect to the circularity patterns enumerated. The central convergence argument (Proposition 1, Lemma 3, Propositions 4-6) is proved inside the manuscript using stated assumptions (L-smoothness, coercivity, boundedness of gradients, and the algorithm's own update rules); no external fitted constants enter the theorem, and no quantity is defined in terms of the target result. The main self-referential element is that the closest baseline, CMAL, is the authors' own prior work, but CMAL is used as an experimental benchmark and as a source of the line-search template, not as the load-bearing justification for convergence. Assumption 4 is introduced as a replacement for CMAL's coercivity condition; this is an explicit modeling assumption, not a result derived from the conclusion being proved. The reader-identified issue in Proposition 5, namely that the claim 'either zeta_{k+1}=zeta_k or zeta_{k+1}=theta zeta_k < zeta_k' ignores Algorithm 3 Step 16 where zeta_{k+1}=max{alpha_tilde_k, alpha_min} can increase zeta, is a genuine mathematical gap in the proof of zeta_k -> 0, but it is not circularity: the proof does not assume its own conclusion, fit a parameter and rename it a prediction, or import a uniqueness result from the authors' prior work. It is a correctness risk, not a circular-derivation risk. No self-definitional, fitted-input-called-prediction, load-bearing self-citation, imported uniqueness, ansatz-smuggled-via-citation, or renamed-known-result pattern is present. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 9 free parameters · 6 assumptions · 0 invented entities

The algorithm depends on eight hand-chosen hyperparameters and on four assumptions about the objective, one of which is introduced specifically for this paper and is only informally linked to gradient clipping. The monotonicity of the learning rate is a proof premise that is contradicted by the algorithm's own Step 16. No new physical or mathematical entities, such as a new force or dimension, are introduced.

free parameters (9)

zeta_0 = 0.05
Initial learning rate chosen by hand; experimental results depend on it.
theta = 0.75
Learning-rate decrease factor chosen by hand in Algorithm 3.
tau = 0.01
Direction-norm threshold that triggers learning-rate decrease in Step 9.
gamma = 0.01
Sufficient-decrease parameter in the Armijo-like condition.
delta = 0.90
Extrapolation factor in DFL; because delta < 1, the search step can grow beyond zeta_k.
eta = 0.50
Scaling factor applied to the learning rate before DFL starts.
alpha_min = 1e-10
Floor for zeta_{k+1} in Step 16.
epsilon = 1e-10
Early-stopping threshold: training stops when zeta_k falls below epsilon.
gradient clip norm = not specified
Gradient clipping is said to enforce Assumption 4, but the clipping value is not reported in the paper.

assumptions (6)

domain assumption Assumption 1: f is coercive, so all level sets are compact.
Used to prove boundedness of iterates in Proposition 4; not verified for neural-network losses.
domain assumption Assumption 2: each f_p is C^1 with L-Lipschitz gradient.
Standard smoothness condition used throughout the convergence proofs.
domain assumption Assumption 3: each f_p is bounded below and all level sets are compact.
Used to bound the sequence phi_k from below in Proposition 5; the compact-level-set part is strong for deep networks.
ad hoc to paper Assumption 4: ||grad f_p(w)|| <= C||grad f(w)|| + D and ||grad f(w)|| <= M for all w.
Central to Lemma 3 and Proposition 4; the paper claims gradient clipping enforces it, but clipping alters the objective and is not proved to satisfy the condition for the original f.
ad hoc to paper The approximation psi used in DFL is coercive.
Proposition 2 requires psi to be coercive for the line search to be well defined; in the implementation psi is a subset sum of batch losses, which need not be coercive in weight space.
ad hoc to paper The learning-rate sequence zeta_k is monotonically non-increasing.
Proposition 5 asserts this, but Step 16 can increase zeta_k to max{alpha_tilde_k, alpha_min}, so the premise is false as written.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization." pith.science (2026). https://pith.science/paper/RNDOEFIH

@misc{pith2026241115795,
  author       = {Pith},
  title        = {Pith review of: Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNDOEFIH}},
  note         = {Machine review of arXiv:2411.15795}
}

read the original abstract

Adaptive gradient methods have been increasingly adopted by deep learning community due to their fast convergence and reduced sensitivity to hyper-parameters. However, these methods come with limitations, such as increased memory requirements for elements like moving averages and a poorly understood convergence theory. To overcome these challenges, we introduce F-CMA, a Fast-Controlled Mini-batch Algorithm with a random reshuffling method featuring a sufficient decrease condition and a line-search procedure to ensure loss reduction per epoch, along with its deterministic proof of global convergence to a stationary point. To evaluate the F-CMA, we integrate it into conventional training protocols for classification tasks involving both convolutional neural networks and vision transformer models, allowing for a direct comparison with popular optimizers. Computational tests show significant improvements, including a decrease in the overall training time by up to 68%, an increase in per-epoch efficiency by up to 20%, and in model accuracy by up to 5%.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 73 canonical work pages

[1]

D. P. Kingma, J. Ba, Adam: A method for stochastic optimiz ation, CoRR abs/1412.6980 (2015). 19

arXiv 2015
[2]

Shazeer, M

N. Shazeer, M. Stern, Adafactor: Adaptive learning rate s with sublinear memory cost, in: International Conference on Machine Learn ing, PMLR, 2018, pp. 4596–4604

2018
[3]

Zaheer, H

R. Zaheer, H. Shaziya, A study of the optimization algori thms in deep learning, in: 2019 third international conference on inven tive systems and control (ICISC), IEEE, 2019, pp. 536–539

2019
[4]

T. Lin, Y. Wang, X. Liu, X. Qiu, A survey of transformers, A I open 3 (2022) 111–132

2022
[5]

S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, M. Sha h, Trans- formers in vision: A survey, ACM computing surveys (CSUR) 54 (10s) (2022) 1–41

2022
[6]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-t raining of deep bidirectional transformers for language understandi ng, arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018
[7]

Radford, K

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training (2018)

2018
[8]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones , A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advance s in neural information processing systems 30 (2017)

2017

Show all 92 references

[9]

Allen-Zhu, Y

Z. Allen-Zhu, Y. Li, Z. Song, A convergence theory for dee p learning via over-parameterization, in: International conference on m achine learning, PMLR, 2019, pp. 242–252

2019
[10]

Buhai, Y

R.-D. Buhai, Y. Halpern, Y. Kim, A. Risteski, D. Sontag, Empirical study of the beneﬁts of overparameterization in learning latent v ariable models, in: International Conference on Machine Learning, PMLR, 20 20, pp. 1211– 1219
[11]

J. Xu, D. J. Hsu, A. Maleki, Beneﬁts of over-parameteriz ation with em, Advances in Neural Information Processing Systems 31 (2018 )

2018
[12]

Chang, Y

X. Chang, Y. Li, S. Oymak, C. Thrampoulidis, Provable be neﬁts of over- parameterization in model compression: From double descen t to pruning neural networks, in: Proceedings of the AAAI Conference on A rtiﬁcial In- telligence, Vol. 35, 2021, pp. 6974–6983

2021
[13]

Robbins, S

H. Robbins, S. Monro, A stochastic approximation metho d, The annals of mathematical statistics (1951) 400–407

1951
[14]

Bottou, F

L. Bottou, F. E. Curtis, J. Nocedal, Optimization metho ds for large-scale machine learning, Siam Review 60 (2) (2018) 223–311

2018
[15]

Ruder, An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747 (2016)

S. Ruder, An overview of gradient descent optimization algorithms, arXiv preprint arXiv:1609.04747 (2016). 20

2016 arXiv
[16]

Sutskever, J

I. Sutskever, J. Martens, G. E. Dahl, G. E. Hinton, On the importance of initialization and momentum in deep learning, in: ICML, 201 3
[17]

D. P. Bertsekas, J. N. Tsitsiklis, Gradient convergenc e in gradient methods with errors, SIAM Journal on Optimization 10 (3) (2000) 627– 642

2000
[18]

Gurbuzbalaban, A

M. Gurbuzbalaban, A. Ozdaglar, P. A. Parrilo, On the con vergence rate of incremental aggregated gradient algorithms, SIAM Journal on Optimiza- tion 27 (2) (2017) 1035–1048

2017
[19]

N. D. Vanli, M. Gurbuzbalaban, A. Ozdaglar, Global conv ergence rate of proximal incremental aggregated gradient methods, SIAM Journal on Optimization 28 (2) (2018) 1282–1300

2018
[20]

Mishchenko, A

K. Mishchenko, A. Khaled, P. Richtárik, Random reshuﬄi ng: Simple anal- ysis with vast improvements, Advances in Neural Informatio n Processing Systems 33 (2020) 17309–17320

2020
[21]

J. Yang, X. Li, I. Fatkhullin, N. He, Two sides of one coin : the limits of untuned sgd and the power of adaptive methods, Advances in Neural Information Processing Systems 36 (2024)

2024
[22]

Bengio, P

Y. Bengio, P. Simard, P. Frasconi, Learning long-term d ependencies with gradient descent is diﬃcult, IEEE transactions on neural ne tworks 5 (2) (1994) 157–166

1994
[23]

LeCun, Y

Y. LeCun, Y. Bengio, G. Hinton, Deep learning, nature 52 1 (7553) (2015) 436–444

2015
[24]

H. B. McMahan, A survey of algorithms and analysis for ad aptive online learning, Journal of Machine Learning Research 18 (90) (201 7) 1–50
[25]

S. H. Haji, A. M. Abdulazeez, Comparison of optimizatio n techniques based on gradient descent algorithm: A review, PalArch’s Journal of Archaeology of Egypt/Egyptology 18 (4) (2021) 2715–2743

2021
[26]

Loshchilov, F

I. Loshchilov, F. Hutter, Decoupled weight decay regul arization, arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[27]

Loshchilov, F

I. Loshchilov, F. Hutter, Fixing weight decay regulari zation in adam (2018)

2018
[28]

Duchi, E

J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient met hods for online learning and stochastic optimization., Journal of machine learning research 12 (7) (2011) 2121–2159

2011
[29]

Dozat, Incorporating nesterov momentum into Adam, i n: ICLR Work- shop, 2016

T. Dozat, Incorporating nesterov momentum into Adam, i n: ICLR Work- shop, 2016

2016
[30]

L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, J. Han, On t he variance of the adaptive learning rate and beyond, arXiv preprint arX iv:1908.03265 (2019). 21

2019 arXiv
[31]

Y. Zhou, B. Karimi, J. Yu, Z. Xu, P. Li, Towards better gen eralization of adaptive gradient methods, Advances in Neural Informati on Processing Systems 33 (2020) 810–821

2020
[32]

C. Ma, L. Wu, E. Weinan, A qualitative study of the dynami c behavior for adaptive gradient algorithms, in: Mathematical and Sci entiﬁc Machine Learning, PMLR, 2022, pp. 671–692

2022
[33]

Kovalev, A

D. Kovalev, A. Salim, P. Richtárik, Optimal and practic al algorithms for smooth and strongly convex decentralized optimization, Ad vances in Neural Information Processing Systems 33 (2020) 18342–18352

2020
[34]

Hanzely, P

F. Hanzely, P. Richtarik, L. Xiao, Accelerated bregman proximal gradient methods for relatively smooth convex optimization, Comput ational Opti- mization and Applications 79 (2021) 405–440

2021
[35]

Kovalev, E

D. Kovalev, E. Gasanov, A. Gasnikov, P. Richtarik, Lowe r bounds and opti- mal algorithms for smooth and strongly convex decentralize d optimization over time-varying networks, Advances in Neural Informatio n Processing Systems 34 (2021) 22325–22335

2021
[36]

Gratton, S

S. Gratton, S. Jerad, P. L. Toint, Convergence properti es of an objective- function-free optimization regularization algorithm, in cluding an complex- ity bound, SIAM Journal on Optimization 33 (3) (2023) 1621–1 646

2023
[37]

Gratton, S

S. Gratton, S. Jerad, P. L. Toint, Complexity of a class o f ﬁrst-order objective-function-free optimization algorithms, Optim ization Methods and Software (2024) 1–31

2024
[38]

Richards, M

D. Richards, M. Rabbat, Learning with gradient descent and weakly convex losses, in: International Conference on Artiﬁcial Intelli gence and Statistics, PMLR, 2021, pp. 1990–1998

2021
[39]

Kawaguchi, Deep learning without poor local minima, Advances in neu- ral information processing systems 29 (2016)

K. Kawaguchi, Deep learning without poor local minima, Advances in neu- ral information processing systems 29 (2016)

2016
[40]

Jiang, S

X. Jiang, S. U. Stich, Adaptive sgd with polyak stepsize and line-search: Robust convergence and variance reduction, Advances in Neu ral Informa- tion Processing Systems 36 (2024)

2024
[41]

Mutschler, A

M. Mutschler, A. Zell, Parabolic approximation line se arch for dnns, Ad- vances in Neural Information Processing Systems 33 (2020) 5 405–5416

2020
[42]

Mutschler, A

M. Mutschler, A. Zell, Empirically explaining sgd from a line search perspec- tive, in: Artiﬁcial Neural Networks and Machine Learning–I CANN 2021: 30th International Conference on Artiﬁcial Neural Network s, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part II 30...

2021
[43]

Mahsereci, P

M. Mahsereci, P. Hennig, Probabilistic line searches f or stochastic optimiza- tion, Journal of Machine Learning Research 18 (119) (2017) 1 –59

2017
[44]

Gulcehre, J

C. Gulcehre, J. Sotelo, M. Moczulski, Y. Bengio, A robus t adaptive stochas- tic gradient method for deep learning, in: 2017 Internation al Joint Confer- ence on Neural Networks (IJCNN), IEEE, 2017, pp. 125–132

2017
[45]

Gürbüzbalaban, A

M. Gürbüzbalaban, A. Ozdaglar, P. A. Parrilo, Why rando m reshuﬄing beats stochastic gradient descent, Mathematical Programm ing 186 (2021) 49–84

2021
[46]

B. Ying, K. Yuan, S. Vlaski, A. H. Sayed, On the performan ce of random reshuﬄing in stochastic learning, in: 2017 Information The ory and Appli- cations Workshop (ITA), IEEE, 2017, pp. 1–5

2017
[47]

Sharma, J

P. Sharma, J. Li, G. Joshi, On improved distributed rand om reshuﬄing over networks, in: ICASSP 2024-2024 IEEE International Con ference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 20 24, pp. 13211– 13215

2024
[48]

Safran, O

I. Safran, O. Shamir, How good is sgd with random shuﬄing ?, in: Confer- ence on Learning Theory, PMLR, 2020, pp. 3250–3284

2020
[49]

Mishchenko, A

K. Mishchenko, A. Khaled, P. Richtárik, Proximal and fe derated random reshuﬄing, in: International Conference on Machine Learni ng, PMLR, 2022, pp. 15718–15749

2022
[50]

Liuzzi, L

G. Liuzzi, L. Palagi, R. Seccia, Convergence under lips chitz smoothness of ease-controlled random reshuﬄing gradient algorithms, arXiv preprint arXiv:2212.01848 (2022)

2022 arXiv
[51]

Coppola, G

C. Coppola, G. Liuzzi, L. Palagi, Cma light: a novel mini batch algo- rithm for large-scale non convex ﬁnite sum optimization, ar Xiv preprint arXiv:2307.15775 (2023)

2023 arXiv
[52]

J. Qian, Y. Wu, B. Zhuang, S. Wang, J. Xiao, Understandin g gradient clipping in incremental gradient methods, in: Internation al Conference on Artiﬁcial Intelligence and Statistics, PMLR, 2021, pp. 150 4–1512

2021
[53]

X. Chen, S. Z. Wu, M. Hong, Understanding gradient clipp ing in private sgd: A geometric perspective, Advances in Neural Informati on Processing Systems 33 (2020) 13773–13782

2020
[54]

Y. Li, Y. Yuan, Convergence analysis of two-layer neura l networks with relu activation, Advances in neural information processing sys tems 30 (2017)

2017
[55]

R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulg in, P. Richtárik, Sgd: General analysis and improved rates, in: Internationa l conference on machine learning, PMLR, 2019, pp. 5200–5209. 23

2019
[56]

J. Liu, Y. Yuan, On almost sure convergence rates of stoc hastic gradient methods, in: Conference on Learning Theory, PMLR, 2022, pp. 2963–2983

2022
[57]

Nguyen, P

L. Nguyen, P. H. Nguyen, M. Dijk, P. Richtárik, K. Schein berg, M. Takác, Sgd and hogwild! convergence without the bounded gradients assumption, in: International Conference on Machine Learning, PMLR, 20 18, pp. 3750– 3758
[58]

Bolte, T

J. Bolte, T. P. Nguyen, J. Peypouquet, B. W. Suter, From e rror bounds to the complexity of ﬁrst-order descent methods for convex fun ctions, Mathe- matical Programming 165 (2017) 471–507

2017
[59]

Ghadimi, G

S. Ghadimi, G. Lan, Stochastic ﬁrst-and zeroth-order m ethods for noncon- vex stochastic programming, SIAM journal on optimization 2 3 (4) (2013) 2341–2368

2013
[60]

X. Li, A. Milzarek, J. Qiu, Convergence of random reshuﬄ ing under the kurdyka–Łojasiewicz inequality, SIAM Journal on Optimization 33 (2) (2023) 1092–1120. doi:10.1137/21M1468048. URL https://doi.org/10.1137/21M1468048

2023 doi
[61]

Blatt, A

D. Blatt, A. O. Hero, H. Gauchman, A convergent incremen tal gradient method with a constant step size, SIAM Journal on Optimizati on 18 (1) (2007) 29–51

2007
[62]

Gurbuzbalaban, A

M. Gurbuzbalaban, A. Ozdaglar, P. A. Parrilo, Converge nce rate of in- cremental gradient and incremental newton methods, SIAM Jo urnal on Optimization 29 (4) (2019) 2542–2565

2019
[63]

Mokhtari, M

A. Mokhtari, M. Gurbuzbalaban, A. Ribeiro, Surpassing gradient descent provably: A cyclic incremental method with linear converge nce rate, SIAM Journal on Optimization 28 (2) (2018) 1420–1447

2018
[64]

H. Li, A. Rakhlin, A. Jadbabaie, Convergence of adam und er relaxed as- sumptions, Advances in Neural Information Processing Syst ems 36 (2024)

2024
[65]

S. Bock, M. Weiß, A proof of local convergence for the ada m optimizer, in: 2019 international joint conference on neural networks (IJCNN), IEEE, 2019, pp. 1–8

2019
[66]

Zhang, C

Y. Zhang, C. Chen, N. Shi, R. Sun, Z.-Q. Luo, Adam can conv erge without any modiﬁcation on update rules, Advances in neural informa tion process- ing systems 35 (2022) 28386–28399

2022
[67]

R. Ward, X. Wu, L. Bottou, Adagrad stepsizes: Sharp conv ergence over nonconvex landscapes, Journal of Machine Learning Researc h 21 (219) (2020) 1–30. 24

2020
[68]

Z. Liu, T. D. Nguyen, A. Ene, H. Nguyen, On the convergenc e of adagrad (norm) on rˆ d: Beyond convexity, non-asymptotic rate and ac celeration, in: International Conference on Learning Representations , International Conference on Learning Representations, 2023

2023
[69]

Kairouz, M

P. Kairouz, M. R. Diaz, K. Rush, A. Thakurta, (nearly) di mension indepen- dent private erm with adagrad rates, in: Conference on Learn ing Theory, PMLR, 2021, pp. 2717–2746

2021
[70]

X. Li, F. Orabona, On the convergence of stochastic grad ient descent with adaptive stepsizes, in: The 22nd international conference on artiﬁcial intel- ligence and statistics, PMLR, 2019, pp. 983–992

2019
[71]

Y. Xie, X. Wu, R. Ward, Linear convergence of adaptive st ochastic gradient descent, in: International conference on artiﬁcial intell igence and statistics, PMLR, 2020, pp. 1475–1485

2020
[72]

Vaswani, A

S. Vaswani, A. Mishkin, I. Laradji, M. Schmidt, G. Gidel , S. Lacoste-Julien, Painless stochastic gradient: Interpolation, line-searc h, and convergence rates, Advances in neural information processing systems 3 2 (2019)

2019
[73]

Loizou, S

N. Loizou, S. Vaswani, I. H. Laradji, S. Lacoste-Julien , Stochastic polyak step-size for sgd: An adaptive learning rate for fast conver gence, in: Inter- national Conference on Artiﬁcial Intelligence and Statist ics, PMLR, 2021, pp. 1306–1314

2021
[74]

Défossez, L

A. Défossez, L. Bottou, F. Bach, N. Usunier, A simple con vergence proof of Adam and Adagrad, arXiv preprint arXiv:2003.02395 (2020 )

2003 arXiv
[75]

Sun, Optimization for deep learning: An overview , Journal of the Operations Research Society of China 8 (2) (2020) 249–294

R.-Y. Sun, Optimization for deep learning: An overview , Journal of the Operations Research Society of China 8 (2) (2020) 249–294

2020
[76]

M. V. Solodov, Incremental gradient algorithms with st epsizes bounded away from zero, Computational Optimization and Applicatio ns 11 (1998) 23–35

1998
[77]

Richtárik, M

P. Richtárik, M. Takáč, Distributed coordinate descen t method for learning with big data, Journal of Machine Learning Research 17 (75) ( 2016) 1–25

2016
[78]

Fercoq, P

O. Fercoq, P. Richtárik, Accelerated, parallel, and pr oximal coordinate descent, SIAM Journal on Optimization 25 (4) (2015) 1997–20 23

2015
[79]

K. Levy, A. Kavis, V. Cevher, Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization, Advances in Neural In formation Processing Systems 34 (2021) 20571–20582

2021
[80]

Dorfman, N

R. Dorfman, N. Yehya, K. Y. Levy, Dynamic byzantine-rob ust learning: Adapting to switching byzantine workers, arXiv preprint ar Xiv:2402.02951 (2024). 25

2024 arXiv
[81]

Allen-Zhu, E

Z. Allen-Zhu, E. Hazan, Variance reduction for faster n on-convex optimiza- tion, in: International conference on machine learning, PM LR, 2016, pp. 699–707

2016
[82]

Armijo, Minimization of functions having lipschitz continuous ﬁrst par- tial derivatives, Paciﬁc Journal of mathematics 16 (1) (196 6) 1–3

L. Armijo, Minimization of functions having lipschitz continuous ﬁrst par- tial derivatives, Paciﬁc Journal of mathematics 16 (1) (196 6) 1–3
[83]

Grippo, F

L. Grippo, F. Lampariello, S. Lucidi, Global convergen ce and stabilization of unconstrained minimization methods without derivative s, Journal of Op- timization Theory and Applications 56 (3) (1988) 385–406

1988
[84]

Grippo, M

L. Grippo, M. Sciandrone, Nonmonotone derivative-fre e methods for non- linear equations, Computational Optimization and applica tions 37 (2007) 297–328

2007
[85]

Krizhevsky, G

A. Krizhevsky, G. Hinton, et al., Learning multiple lay ers of features from tiny images (2009)

2009
[86]

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning fo r image recog- nition, in: Proceedings of the IEEE conference on computer v ision and pattern recognition, 2016, pp. 770–778

2016
[87]

Zagoruyko, N

S. Zagoruyko, N. Komodakis, Wide residual networks, ar Xiv preprint arXiv:1605.07146 (2016)

2016 arXiv
[88]

Sandler, A

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Mobilenetv2: Inverted residuals and linear bottlenecks, in: Proceeding s of the IEEE con- ference on computer vision and pattern recognition, 2018, p p. 4510–4520

2018
[89]

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Gu o, Swin trans- former: Hierarchical vision transformer using shifted win dows, in: Proceed- ings of the IEEE/CVF international conference on computer v ision, 2021, pp. 10012–10022

2021
[90]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, I magenet: A large- scale hierarchical image database, in: 2009 IEEE conferenc e on computer vision and pattern recognition, Ieee, 2009, pp. 248–255

2009
[91]

Zhuang, J

B. Zhuang, J. Liu, Z. Pan, H. He, Y. Weng, C. Shen, A survey on eﬃcient training of transformers, in: E. Elkind ( Ed.), Proceedings of the Thirty-Second International Joint Conf erence on Artiﬁcial Intelligence, IJCAI-23, International Joint Co nferences on Artiﬁcial Intellige...

2023 doi
[92]

L. Papa, P. Russo, I. Amerini, L. Zhou, A survey on eﬃcien t vision trans- formers: algorithms, techniques, and performance benchma rking, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024). 26

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

D. P. Kingma, J. Ba, Adam: A method for stochastic optimiz ation, CoRR abs/1412.6980 (2015). 19

arXiv 2015

[2] [2]

Shazeer, M

N. Shazeer, M. Stern, Adafactor: Adaptive learning rate s with sublinear memory cost, in: International Conference on Machine Learn ing, PMLR, 2018, pp. 4596–4604

2018

[3] [3]

Zaheer, H

R. Zaheer, H. Shaziya, A study of the optimization algori thms in deep learning, in: 2019 third international conference on inven tive systems and control (ICISC), IEEE, 2019, pp. 536–539

2019

[4] [4]

T. Lin, Y. Wang, X. Liu, X. Qiu, A survey of transformers, A I open 3 (2022) 111–132

2022

[5] [5]

S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, M. Sha h, Trans- formers in vision: A survey, ACM computing surveys (CSUR) 54 (10s) (2022) 1–41

2022

[6] [6]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-t raining of deep bidirectional transformers for language understandi ng, arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018

[7] [7]

Radford, K

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training (2018)

2018

[8] [8]

Vaswani, N

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones , A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, Advance s in neural information processing systems 30 (2017)

2017