Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations
read the original abstract
In this paper, we propose a continuous-time formulation for the AdaGrad, RMSProp, and Adam optimization algorithms by modeling them as first-order integro-differential equations. We perform numerical simulations of these equations, along with stability and convergence analyses, to demonstrate their validity as accurate approximations of the original algorithms. Our results indicate a strong agreement between the behavior of the continuous-time models and the discrete implementations, thus providing a new perspective on the theoretical understanding of adaptive optimization methods.
This paper has not been read by Pith yet.
Forward citations
Cited by 2 Pith papers
-
Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
Adam-HNAG is a splitting-based reformulation of Adam that yields the first convergence proof for Adam-type methods, including accelerated rates, in convex smooth optimization.
-
Global Stability and Step Size Robustness of RMSProp
An input-to-state Lyapunov function is introduced to prove global asymptotic stability of RMSProp for constant step sizes and robustness to arbitrary bounded time-varying step size rules.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.