AutoSGD runs three parallel SGD streams at nearby learning rates, uses paired noisy objective estimates to pick the winner, and is claimed to converge with little user tuning.
Anytime Tail Averaging
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Tail averaging consists in averaging the last examples in a stream. Common techniques either have a memory requirement which grows with the number of samples to average, are not available at every timestep or do not accomodate growing windows. We propose two techniques with a low constant memory cost that perform tail averaging with access to the average at every time step. We also show how one can improve the accuracy of that average at the cost of increased memory consumption.
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent
AutoSGD runs three parallel SGD streams at nearby learning rates, uses paired noisy objective estimates to pick the winner, and is claimed to converge with little user tuning.