ADAGB2 achieves O(epsilon^{-2}) probabilistic iteration complexity for nonconvex bound-constrained stochastic optimization under a directional gradient-noise condition, with a counterexample showing unbiased oracles need not transfer approximate to true optimality.
Fully stochastic trust-region methods with Barzilai-Borwein steplengths
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method.
fields
math.OC 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Fast Stochastic Second-Order Adagrad for Nonconvex Bound-Constrained Optimization
ADAGB2 achieves O(epsilon^{-2}) probabilistic iteration complexity for nonconvex bound-constrained stochastic optimization under a directional gradient-noise condition, with a counterexample showing unbiased oracles need not transfer approximate to true optimality.