Pith. sign in

REVIEW 2 major objections 3 minor 1 cited by

Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters

T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that a unified momentum-based stochastic-optimization template, covering Stochastic Heavy Ball and Stochastic Nesterov Accelerated Gradient with time-varying parameters, converges almost surely under…

desk verdict The main theorem is not proven as written: the Lyapunov argument drops the δ_t coupling terms, and the stated condition δ_t→0 is likely too weak. read the letter →

arxiv 2506.11904 v2 pith:2JZ5MTG3 submitted 2025-06-13 math.OC cs.LGstat.ML

Mathukumalli Vidyasagar This is my paper
classification math.OCcs.LGstat.ML MSC 65K1068Q2568W20
keywords heavy-ballalgorithmstochasticmomentummethodstime-varyingalmostsureconvergenceRobbins-MonroconditionsKiefer-Wolfowitz-Blumzeroth-orderoptimizationKurdyka-Lojasiewiczproperty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes convergence conditions for a single stochastic-optimization template that contains the Stochastic Heavy Ball and the Stochastic Nesterov Accelerated Gradient algorithms as special cases, with the momentum parameter allowed to change from one iteration to the next. The central result is that, under natural analogues of the Robbins–Monro and Kiefer–Wolfowitz–Blum conditions, the unified momentum update has $\liminf_{t\to\infty}\|\nabla J(\theta_t)\|_2=0$ almost surely; if the objective satisfies a Kurdyka–Łojasiewicz-type condition, $J(\theta_t)\to 0$ and $\nabla J(\theta_t)\to 0$ almost surely. The assumed stochastic gradient may be biased and may have conditional variance that grows without bound, which is what makes the result applicable to zeroth-order methods such as simultaneous-perturbation stochastic approximation that estimate gradients from just two function evaluations. A separate analysis shows that a previously proposed transformation for time-varying heavy-ball parameters is impractical, because its synthetic step sizes either violate the summability conditions or become negative.

What carries the argument

The argument uses an eigenvector diagonalization of the coupled update. With $k_t:=a_t/(1-\mu_t)$, the change of variables $u_t:=w_t+k_t v_t$ turns the update into $u_{t+1}=u_t+\delta_t\mu_t v_t-(b_t+k_{t+1})\alpha_t h_{t+1}$ and $v_{t+1}=\mu_t v_t-\alpha_t h_{t+1}$, where $\delta_t=k_{t+1}-k_t$; when $\delta_t\to0$, the equations are asymptotically decoupled. The proof tracks the Lyapunov function $V_t=J(u_t)+\|v_t\|_2^2$ and applies the Robbins–Siegmund almost-supermartingale theorem to $V_t$, obtaining boundedness of $J(\theta_t)$ and summability of the dissipation terms, from which the lim-inf and almost-sure convergence conclusions follow.

What would settle it

Run the SHB update on a one-dimensional quadratic with unbiased unit-variance noise, $\alpha_t=1/t$, $a_t=1$, and $\mu_t=0.5+0.25\sin t$ (so $\mu_t$ stays below $1$ and the Robbins–Monro sums hold, but $k_t=a_t/(1-\mu_t)$ oscillates and $\delta_t\not\to0$). If the simulated $\liminf_{t\to\infty}|\nabla J(\theta_t)|$ stays above zero, the $\delta_t\to0$ condition is doing load-bearing work; if it converges, the condition is not necessary for the conclusion, and Theorem 4.4's sufficient conditions are not tight.

Watch

Extended reading notes

Core claim

On the paper's own terms, the claim is that the unified momentum update $w_{t+1}=w_t+a_t v_t-b_t\alpha_t h_{t+1}$, $v_{t+1}=\mu_t v_t-\alpha_t h_{t+1}$ converges almost surely under standing assumptions (S1)–(S2) on $J$, the biased-noise bounds (4.10)–(4.11) on the stochastic gradient, and the parameter conditions (4.12)–(4.15). Concretely, $\sum_t\alpha_t^2<\infty$, $\sum_t\alpha_t B_t<\infty$, $\sum_t\alpha_t^2 M_t^2<\infty$, and $\sum_t\alpha_t=\infty$ imply $\liminf_{t\to\infty}\|\nabla J(\theta_t)\|_2=0$; if $J$ satisfies the KL' property, then $J(\theta_t)\to0$ and $\nabla J(\theta_t)\to0$ almost surely; and if $J$ satisfies the stronger Polyak–Łojasiewicz property and the bias-and-variance sequences decay or grow at controlled rates, explicit almost-sure rates $J(\theta_t)=o(t^{-\lambda})$ follow. A distinctive feature is that all parameters, including $\mu_t$ and $\alpha_t$, may themselves be time-varying or random, with the only added requirement being that the transformed coupling ratio $k_t=a_t/(1-\mu_t)$ converges so that the updates need only be asymptotically decoupled.

Load-bearing premise

The load-bearing premise is that the ratio $a_t/(1-\mu_t)$ settles down — that is, $\delta_t=k_{t+1}-k_t\to0$ — so the coupling term $\delta_t\mu_t v_t$ can be absorbed into the Lyapunov argument; if the step-size and momentum schedules oscillate forever, the proof loses control even when every Robbins–Monro-type sum converges.

Editorial extensions

If this is right

  • The Stochastic Heavy Ball and the nonstandard Stochastic Nesterov Accelerated Gradient with $\mu_t$ bounded away from $1$ inherit the same almost-sure convergence guarantees as SGD in the unbiased, bounded-variance case, since the conditions reduce exactly to the Robbins–Monro conditions.
  • Zeroth-order estimates of the SPSA type, with bias $B_t=O(c_t)$ and variance $M_t=O(1/c_t)$, satisfy Corollary 4.6, so convergence holds under the Kiefer–Wolfowitz–Blum conditions with only two function evaluations per iteration.
  • The block-update variants — single-coordinate, multi-coordinate, and Bernoulli updates — inherit the conclusions of Theorems 4.4 and 4.7, extending the theory to randomized partial-gradient updates.
  • Under the PL property, the rate bound $J(\theta_t)=o(t^{-\lambda})$ for every $\lambda<\min\{1-2\delta,\gamma\}$ shows that time-varying momentum does not degrade the best known rates when the bias and variance sequences are controlled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\delta_t\to0$ condition is a design rule the paper leaves implicit: choose $a_t$ and $\mu_t$ so that $a_t/(1-\mu_t)$ settles, not merely so that each sequence is individually well-behaved; oscillating schedules that pass every Robbins–Monro sum fall outside the theorem.
  • A natural quantitative extension is to replace $\delta_t\to0$ by $\sum_t|\delta_t|<\infty$; the Lyapunov argument might then absorb the coupling term with a slightly weaker dissipation term, a modification the paper does not explore.
  • The appendix's failure analysis suggests a broader caution for practitioners: exact decoupling transformations for momentum methods with time-varying coefficients appear inherently unstable, so asymptotic decoupling of the type used here is the more promising design principle.
  • For schedules with $\mu_t\to1$, the paper leaves open a bounded-variance regime worth testing numerically; a simulation with two-evaluation SPSA gradients and a momentum schedule approaching $1$ would show whether the divergence observed for standard NAG is avoidable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes a unified momentum-based stochastic optimization algorithm (2.2)-(2.3) that encompasses Stochastic Heavy Ball and a nonstandard Stochastic Nesterov Accelerated Gradient, with all parameters allowed to vary in time and even to be random. Under standing assumptions on the objective function and very general bias/variance conditions (4.10)-(4.11) on the stochastic gradient, the paper claims almost-sure convergence of liminf ||∇J(θ_t)|| to zero, and, under a KL-type property, full almost-sure convergence of J(θ_t) and ∇J(θ_t). The proof uses a variable transformation that diagonalizes the noiseless dynamics when parameters are constant, and treats the time-varying case as asymptotically decoupled. The paper also contains a rate result under the PL property, a block-update extension, and an appendix criticizing the variable transformation of Sebbouh-Gower-Defazio for time-varying momentum.

Significance. If the main theorem is correct, the paper is a valuable contribution: it unifies SHB and SNAG under one set of sufficient conditions that generalize Robbins-Monro and Kiefer-Wolfowitz-Blum conditions, and it handles biased stochastic gradients with conditionally unbounded variance, which is relevant to zeroth-order methods. The claims are strong: liminf convergence of the gradient norm, not merely of a running minimum, and almost-sure convergence under KL'. The paper also gives rate bounds under PL and extends the results to block updates. These are substantive claims. However, the central Lyapunov proof in Section 5.3 contains a coupling-term omission that is load-bearing for the main theorem; the fix requires either a stronger condition on the time-variation of k_t or a genuinely different argument. The surrounding structure, including the Robbins-Siegmund framework and the use of WBF bookkeeping, is otherwise coherent and mostly carefully executed.

major comments (2)
  1. [Section 5.3, Eqs. (5.42)-(5.50)] The term δ_t μ_t ⟨∇J(u_t), v_t⟩ in (5.42) is bounded in (5.43) as |δ_t| μ̄ ||∇J(u_t)|| ||v_t||, with no factor α_t. In the aggregated inequality (5.50) and in the quadratic form K_t of (5.52), every cross term between ||∇J(u_t)|| and ||v_t|| carries a factor α_t. Thus the proof silently requires |δ_t| ≤ C α_t, or at least δ_t² = O(α_t), in order to absorb this term into the negative definite part for large t. Condition (4.15), which only requires δ_t → 0, is insufficient. Concretely, with d=1, J(θ)=θ²/2, μ_t=1/2, α_t=1/t, b_t=1, ε_t=0, and a_t=0.5 Σ_{s≤t} (-1)^s/s^{0.4}, the stated assumptions hold but δ_t = Θ(t^{-0.4}) and δ_t²/α_t → ∞, so the Lyapunov argument as written does not apply. Please either strengthen (4.15) and the theorem statements accordingly, or provide an alternative argument that controls this coupling term without an additional condition.
  2. [Section 5.3, Item 3 proof] The contradiction argument after (5.57) establishes a lower bound on ||∇J(w_t(ω))||₂, but the summability conclusion (5.56) concerns ||∇J(u_t(ω))||₂². Since w_t = u_t - k_t v_t and v_t → 0, a short Lipschitz-continuity step is needed to pass from ||∇J(w_t)|| ≥ q to ||∇J(u_t)|| ≥ q/2 eventually; this step is implicit in the text and should be stated explicitly. In the same paragraph, the definition of q uses an unspecified quantity M(ω); it should presumably be G(ω) or a related bound, and this should be clarified.
minor comments (3)
  1. [Section 5.3, Eq. (5.53)] Equation (5.53) writes V_{t+1} on the right-hand side where V_t is meant; the correct inequality is E_t(V_{t+1}) ≤ V_t - ... .
  2. [Section 5.3, Eq. (5.20)] The sentence introducing V_t says 'We will find an upper bound for E_t(V_t)' but the subsequent derivation bounds E_t(V_{t+1}); the notation should be aligned.
  3. [Throughout] There are numerous typographical and grammatical slips, e.g., 'convrgence', 'objectve', 'gneralizations', and missing punctuation in displayed equations; a careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central convergence proof is a direct application of independently checkable stochastic-approximation lemmas and a re-derived variable transformation, not a renaming or re-fitting of its inputs.

full rationale

Theorem 4.4 is proved from Robbins–Siegmund almost-supermartingale theory (Lemma 5.1, [27]) together with the extensions stated as Theorems 5.2 and 5.4 and credited to the author's prior work [14,15]. Those extensions are general statements about nonnegative processes satisfying (5.13); their hypotheses do not include the target momentum-algorithm convergence, so applying them is a genuine deduction, not a circular import. The variable transformation in Section 5.1 is re-derived in the paper via the eigenvector matrix Z_t (5.3)–(5.10), even though the approach originates in [26]; the time-varying coupling parameter δ_t is explicitly identified and not hidden. The block-updating Theorem 4.8 cites [26, Theorem 8] and omits a proof, but this is a peripheral extension of the main theorem; the cited theorem is a checkable prior result rather than an unverified premise tailored to the present claim. The Appendix's negative analysis of [30] is a new, self-contained computation. No fitted quantity is renamed as a prediction, and no uniqueness theorem is imported to force the chosen assumptions. A skeptical reader's objection that condition (4.15) δ_t→0 may be insufficient for the Lyapunov argument—because the δ_t coupling terms in (5.42)–(5.46) may require δ_t^2=O(α_t) for positive definiteness—is a mathematical correctness/completeness concern, not a circularity; the theorem's conclusion is not equivalent by construction to its hypotheses. Hence no circular step is identified, and the small score reflects only the presence of self-citations that remain independently checkable.

Assumptions & free parameters 0 free parameters · 10 assumptions · 0 invented entities

The paper introduces no fitted constants and no new postulates. It relies on standard Lipschitz and noise assumptions, the KL'/PL/NSC conditions from prior literature, and the technical condition δ_t→0 that enables the decoupling transformation. The core stochastic tools (Robbins-Siegmund and its extensions) are cited from [14,15] and stated as lemmas.

assumptions (10)
  • domain assumption J is C1 with L-Lipschitz gradient (S1, Section 4.1.1)
    Used throughout for the descent lemma (Lemma 4.2) and the gradient bound (Lemma 4.1).
  • domain assumption J is bounded below and its infimum is attained (S2, Section 4.1.1)
    Ensures J* is finite and the set S_J is nonempty, required for Lemma 4.1.
  • domain assumption KL' property (4.5) with Class B function ψ
    Used in Theorem 4.4 Item 3 to force J(θ_t)→0 from liminf of gradient norm.
  • domain assumption NSC property: compact level sets and a bound on distance to the minimizer set (Section 4.1.1)
    Used in Item 4 to convert J(θ_t)→0 into ρ(θ_t)→0.
  • domain assumption PL property (4.4) in Theorem 4.7
    Gives rates of convergence.
  • domain assumption Momentum bounded away from 1: μ_t ≤ μ̄ < 1 (4.12)
    Ensures uniform contraction in v_t; excludes standard NAG with μ_t→1.
  • domain assumption δ_t = k_{t+1} - k_t → 0 (4.15)
    Key technical condition; makes the transformed equations asymptotically decoupled in (5.10).
  • domain assumption Stochastic gradient bias and variance bounds (4.10)-(4.11) with summable sequences α_tB_t and α_t^2 M_t^2
    Generalizes RM/KWB conditions; enables biased, unbounded-variance noise and zero-order methods.
  • standard math Robbins-Siegmund almost supermartingale theorem (Lemma 5.1)
    Core tool for almost sure convergence.
  • standard math Karandikar-Vidyasagar extensions (Theorems 5.2 and 5.4, from [14,15])
    Stated without proof; provide the refined convergence and rate results used in Sections 5.3 and 5.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters." pith.science (2026). https://pith.science/paper/2JZ5MTG3

@misc{pith2026250611904,
  author       = {Pith},
  title        = {Pith review of: Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2JZ5MTG3}},
  note         = {Machine review of arXiv:2506.11904}
}
read the original abstract

In this paper, we present a unified algorithm for stochastic optimization that makes use of a "momentum" term; in other words, the stochastic gradient depends not only on the current true gradient of the objective function, but also on the true gradient at the previous iteration. Our formulation includes the Stochastic Heavy Ball (SHB) and the Stochastic Nesterov Accelerated Gradient (SNAG) algorithms as special cases. In addition, in our formulation, the momentum term is allowed to vary as a function of time (i.e., the iteration counter). The assumptions on the stochastic gradient are the most general in the literature, in that it can be biased, and have a conditional variance that grows in an unbounded fashion as a function of time. This last feature is crucial in order to make the theory applicable to "zero-order" methods, where the gradient is estimated using just two function evaluations. We present a set of sufficient conditions for the convergence of the unified algorithm. These conditions are natural generalizations of the familiar Robbins-Monro and Kiefer-Wolfowitz-Blum conditions for standard stochastic gradient descent. We also analyze another method from the literature for the SHB algorithm with a time-varying momentum parameter, and show that it is impracticable.

Figures

Figures reproduced from arXiv: 2506.11904 by the authors.

Figure 1
Figure 1. Computational results for the numerical example with a variety of algorithms among others, the following function, referred to as J2(·) there: J(θ) = θ ⊤Aθ + 3 sin2 (h1, θi), where θt is a vector of 1 million parameters, A is a block-diagonal matrix of size (106 × 106 ), consisting of 100 Hilbert matrices, each of dimension 104 × 104 , and 1 denotes a column vector of all ones. This function is not convex, but it do… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

    stat.ML 2026-01 conditional novelty 7.0 of 10

    Momentum SGD pays a provable tracking penalty under distribution shift — lag growing as (1−β)⁻¹ and tracking floors worse than vanilla SGD in drift-dominated regimes.

Reference graph

Works this paper leans on

35 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [30]

    Alm ost sure convergence rates for stochastic gradient descent and stochastic heavy ball

    Othmane Sebbouh, Robert M Gower, and Aaron Defazio. Alm ost sure convergence rates for stochastic gradient descent and stochastic heavy ball. In Proceedings of Thirty Fourth Conference on Learning Theory, PMLR , volume 134, pages 3935–3971, 2021

    work page 2021

  2. [1]

    Convergence rates for the heavy-ball continuous dynamics for non-convex opti mization, under polyak–lojasiewicz condition

    Vassilis Apidopoulos, Nicol` o Ginatta, and Silvia Vill a. Convergence rates for the heavy-ball continuous dynamics for non-convex opti mization, under polyak–lojasiewicz condition. Journal of Global Optimization , 84:563–589, 2022

    work page 2022

  3. [2]

    Duchi, Dylan J

    Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Fost er, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathemat- ical Programming, 199(1–2):165–214, 2023

    work page 2023

  4. [3]

    Optimal co nvergence rates for nesterov acceleration

    Jean-Franc cois Aujol, Charles Dossal, and Aude Rondepierre. Optimal co nvergence rates for nesterov acceleration. SIAM Journal on Optimization , 29(4):3131–3153, 2019

    work page 2019

  5. [4]

    Convergen ce rates of the heavy-ball method under the lojasiewicz property

    Jean-Franc cois Aujol, Charles Dossal, and Aude Rondepierre. Convergen ce rates of the heavy-ball method under the lojasiewicz property. Mathematical Programming, 198(1):198–254, 2023

    work page 2023

  6. [5]

    Advances in optimizing recurrent networks

    Yoshua Bengio, Nicolas Boulanger-Lewandowski, and Raz van Pascanu. Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , pages 8624–8628, 2013

    work page 2013

  7. [6]

    Adaptive Algorithms and Stochastic Approximation

    Albert Benveniste, Michel M´ etivier, and Pierre Priour et. Adaptive Algorithms and Stochastic Approximation. Springer-Verlag, 1990

    work page 1990

  8. [7]

    Bertsekas and John N

    Dmitri P. Bertsekas and John N. Tsitsiklis. Global conve rgence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627–642, 2000

    work page 2000

Show all 35 references
  1. [8]

    Julius R. Blum. Multivariable stochastic approximatio n methods. Annals of Mathe- matical Statistics, 25(4):737–744, 1954

    1954

  2. [9]

    Curtis, and Jorge Nocedal

    L´ eon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimi zation methods for large- scale machine learning. SIAM Review, 60(2):223–311, 2018

    2018

  3. [10]

    Probability: Theory and Examples (5th Edition)

    Rick Durrett. Probability: Theory and Examples (5th Edition) . Cambridge University Press, 2019

    2019

  4. [11]

    Convergence of s equences: A survey

    Barbara Franci and Sergio Grammatico. Convergence of s equences: A survey. arxiv:2111.11374, 2011

    2011 arXiv

  5. [12]

    Elec- tronic Journal of Statistics , 12:461–529, 2018

    S´ ebastien Gadat, Fabien Panloup, and Sofiane Saadane.Stochastic heavy ball. Elec- tronic Journal of Statistics , 12:461–529, 2018

    2018

  6. [13]

    Global con- vergence of the heavy-ball method for convex optimization.In Proceedings of the 2015 European Control Conference, pages 311–316, 2015

    Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global con- vergence of the heavy-ball method for convex optimization.In Proceedings of the 2015 European Control Conference, pages 311–316, 2015. MOMENTUM-BASED ALGORITHMS WITH TIME-V ARYING PARAMETERS 3 5

    2015

  7. [14]

    Karandikar and M

    Rajeeva L. Karandikar and M. Vidyasagar. Convergence r ates for stochas- tic approximation: Biased noise with unbounded variance, a nd applications. https://arxiv.org/pdf/2312.02828v3.pdf, May 2024

    2024 arXiv

  8. [15]

    Karandikar and M

    Rajeeva L. Karandikar and M. Vidyasagar. Convergence r ates for stochastic approx- imation: Biased noise with unbounded variance, and applica tions. Journal of Opti- mization Theory and Applications , 203:2412–2450, October 2024

    2024

  9. [16]

    Linear co nvergence of gradient and proximal-gradient methods under the polyak- lojasiewicz c ondition

    Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear co nvergence of gradient and proximal-gradient methods under the polyak- lojasiewicz c ondition. Lecture Notes in Computer Science, 9851:795–811, 2016

    2016

  10. [17]

    Better Theory for SGD in the Nonconvex World

    Ahmed Khaled and Peter Richt´ arik. Better Theory for SGD in the Nonconvex World. arXiv:2002.03329, February 2020

    2002 arXiv

  11. [18]

    Kiefer and J

    J. Kiefer and J. Wolfowitz. Stochastic estimation of th e maximum of a regression function. Annals of Mathematical Statistics , 23(3):462–466, 1952

    1952

  12. [19]

    Jinlan Liu, Dongpo Xu, Yinghua Lu, Jun Kong, and Danilo P . Mandic. Last-iterate convergence analysis of stochastic momentum methods for ne ural networks. Neuro- computing, 527:27–35, 2023

    2023

  13. [20]

    On almost sure convergence rates of s tochastic gradient meth- ods

    Jun Liu and Ye Yuan. On almost sure convergence rates of s tochastic gradient meth- ods. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Con- ference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2963–2983. PMLR, 02–0...

    2022

  14. [21]

    A method for unconstrained convex mini mization problem with the rate of convergence o(1/k2)n (in russian)

    Yurii Nesterov. A method for unconstrained convex mini mization problem with the rate of convergence o(1/k2)n (in russian). Soviet Mathematics Doklady , 269:543–547, 1983

    1983

  15. [22]

    Introductory Lectures on Convex Optimization: A Basic Cour se, vol- ume 87

    Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Cour se, vol- ume 87. Springer Scientific+Business Media, 2004

    2004

  16. [23]

    Random Gradient -Free Minimization of Con- vex Functions

    Yurii Nesterov and Vladimir Spokoiny. Random Gradient -Free Minimization of Con- vex Functions. Foundations of Computational Mathematics , 17(2):527–566, 2017

    2017

  17. [24]

    B. T. Polyak. Gradient methods for the minimisation of f unctionals. USSR Compu- tational Mathematics and Mathematical Physics , 3(4):864–878, 1963

    1963

  18. [25]

    B.T. Polyak. Some methods of speeding up the convergenc e of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1–17, 1964

    1964

  19. [26]

    Convergence of momentum-based heavy ball method with batch updating and/o r approximate gra- dients

    Tadipatri Uday Kiran Reddy and Mathukumalli Vidyasaga r. Convergence of momentum-based heavy ball method with batch updating and/o r approximate gra- dients. https://arxiv.org/pdf/2303.16241V4.pdf, April 2025

    2025 arXiv

  20. [27]

    Robbins and D

    H. Robbins and D. Siegmund. A convergence theorem for non negative almost super- martingales and some applications , pages 233–257. Elsevier, 1971

    1971

  21. [28]

    A stochastic approxi mation method

    Herbert Robbins and Sutton Monro. A stochastic approxi mation method. Annals of Mathematical Statistics, 22(3):400–407, 1951

    1951

  22. [29]

    Sadegh and J

    P. Sadegh and J. C. Spall. Optimal random perturbations for stochastic approxima- tion using a simultaneous perturbation gradient approxima tion. IEEE transactions on automatic control , 43(10):1480–1484, 1998

    1998

  23. [31]

    A uni- fied analysis of adagrad with weighted aggregation and momen tum acceleration

    Li Shen, Congliang Chen, Fangyu Zou, Zequn Jie, Ju Sun, a nd Wei Liu. A uni- fied analysis of adagrad with weighted aggregation and momen tum acceleration. arxiv:1808.03408, August 2018

    2018 arXiv

  24. [32]

    Cand` es

    Weijie Su, Stephen Boyd, and Emmanuel J. Cand` es. A differential equation for model- ing nesterov’s accelerated gradient method: Theory and ins ights. Journal of Machine Learning Research, 17:1–43, 2016. 36 M. VIDYASAGAR

    2016

  25. [33]

    On the impor- tance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffre y Hinton. On the impor- tance of initialization and momentum in deep learning. In Proceedings of the 30th international conference on machine learning (ICML-13) , pages 1139–1147, 2013

    2013

  26. [34]

    Probability with Martingagles

    David Williams. Probability with Martingagles . Cambridge University Press, 1991

    1991

  27. [35]

    A u nified analysis of stochastic momentum methods for deep learning

    Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A u nified analysis of stochastic momentum methods for deep learning. In Proceedings of the 27th Interna- tional Joint Conference on Artificial Intelligence , IJCAI’18, page 2955–2961. AAAI Press, 2018. (M. Vidyasagar) Indi...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.