Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

Jie Liu; Katya Scheinberg; Lam M. Nguyen; Martin Tak\'a\v{c}

arxiv: 1705.07261 · v1 · pith:Q6BIYYISnew · submitted 2017-05-20 · 📊 stat.ML · cs.LG· math.OC

Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

Lam M. Nguyen , Jie Liu , Katya Scheinberg , Martin Tak\'a\v{c} This is my paper

classification 📊 stat.ML cs.LGmath.OC

keywords gradientnonconvexstochasticrecursivealgorithmconvergencefunctionslosses

0 comments

read the original abstract

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Accelerating Mini-batch SARAH by Step Size Rules
cs.LG 2019-06 unverdicted novelty 4.0

MB-SARAH-RBB uses a random Barzilai-Borwein step size to accelerate mini-batch SARAH, with a linear convergence proof and improved complexity for strongly convex objectives.