prunAdag separates parameters into optimisable and decreasable sets, updates them with Adagrad-like rules, and provably drives the average gradient norm to zero at rate O(log(k)/sqrt(k+1)).
Complexity and performance for two classes of noise-tolerant first-order algorithms
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Two classes of algorithms for optimization in the presence of noise are presented, that do not require the evaluation of the objective function. The first generalizes the well-known Adagrad method. Its complexity is then analyzed as a function of its parameters. A second class of algorithms is then derived whose complexity is at least as good as that of the first class. Initial numerical experiments on finite-sum problems arising from deep-learning applications suggest that methods of the second class may outperform those of the first.
fields
math.OC 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
prunAdag: an adaptive pruning-aware gradient method
prunAdag separates parameters into optimisable and decreasable sets, updates them with Adagrad-like rules, and provably drives the average gradient norm to zero at rate O(log(k)/sqrt(k+1)).