PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems without extra gradient evaluations.
Optimal non-asymptotic bound of the Ruppert-Polyak averaging without strong convexity
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper is devoted to the non-asymptotic control of the mean-squared error for the Ruppert-Polyak stochastic averaged gradient descent introduced in the seminal contributions of [Rup88] and [PJ92]. In our main results, we establish non-asymptotic tight bounds (optimal with respect to the Cramer-Rao lower bound) in a very general framework that includes the uniformly strongly convex case as well as the one where the function f to be minimized satisfies a weaker Kurdyka-Lojiasewicz-type condition [Loj63, Kur98]. In particular, it makes it possible to recover some pathological examples such as on-line learning for logistic regression (see [Bac14]) and recursive quan- tile estimation (an even non-convex situation).
citation-role summary
background 1
citation-polarity summary
fields
math.OC 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems without extra gradient evaluations.