Under the Polyak-Lojasiewicz condition, a proximal primal-dual algorithm and an AdaGrad-style variant maximize AUC with deep networks at O~(1/epsilon) sample complexity, with adaptive iteration complexity under slow cumulative gradient growth.
Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.
representative citing papers
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
MARS combines STORM-style variance reduction with preconditioned updates and reports improved token efficiency over AdamW on GPT-2 pretraining, though the analyzed and evaluated variants differ.
MB-SARAH-RBB uses a random Barzilai-Borwein step size to accelerate mini-batch SARAH, with a linear convergence proof and improved complexity for strongly convex objectives.
citing papers explorer
-
Stochastic AUC Maximization with Deep Neural Networks
Under the Polyak-Lojasiewicz condition, a proximal primal-dual algorithm and an AdaGrad-style variant maximize AUC with deep networks at O~(1/epsilon) sample complexity, with adaptive iteration complexity under slow cumulative gradient growth.
-
OptMuon: Closed-Loop Orthogonalized Momentum Methods for Stochastic Optimization with Zero-Noise Optimality
OptMuon combines orthogonalized momentum with trajectory-dependent AdaGrad-Norm adaptation to obtain expected-stationarity rates of order T^{-1/2} + sigma^{1/2}T^{-1/4} or T^{-1/2} + sigma^{1/3}T^{-1/3} that reduce to near-optimal deterministic first-order rates in the zero-noise regime.
-
MARS: Unleashing the Power of Variance Reduction for Training Large Models
MARS combines STORM-style variance reduction with preconditioned updates and reports improved token efficiency over AdamW on GPT-2 pretraining, though the analyzed and evaluated variants differ.
-
Accelerating Mini-batch SARAH by Step Size Rules
MB-SARAH-RBB uses a random Barzilai-Borwein step size to accelerate mini-batch SARAH, with a linear convergence proof and improved complexity for strongly convex objectives.