A coupled ODE system, discretized with a decreasing learning rate, yields accelerated SGD algorithms with proven optimal last-iterate rates and improved constants.
Stochastic Gradient Descent with Polyak's Learning Rate
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Stochastic gradient descent (SGD) for strongly convex functions converges at the rate $\bO(1/k)$. However, achieving good results in practice requires tuning the parameters (for example the learning rate) of the algorithm. In this paper we propose a generalization of the Polyak step size, used for subgradient methods, to Stochastic gradient descent. We prove a non-asymptotic convergence at the rate $\bO(1/k)$ with a rate constant which can be better than the corresponding rate constant for optimally scheduled SGD. We demonstrate that the method is effective in practice, and on convex optimization problems and on training deep neural networks, and compare to the theoretical rate.
fields
math.OC 1years
2019 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Nesterov's method with decreasing learning rate leads to accelerated stochastic gradient descent
A coupled ODE system, discretized with a decreasing learning rate, yields accelerated SGD algorithms with proven optimal last-iterate rates and improved constants.