REVIEW 4 cited by
DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning rate'' parameter. Theoretically, we show that a slight variation of the DoG formula enjoys strong parameter-free convergence guarantees for stochastic convex optimization assuming only \emph{locally bounded} stochastic gradients. Empirically, we consider a broad range of vision and language transfer learning tasks, and show that DoG's performance is close to that of SGD with tuned learning rate. We also propose a per-layer variant of DoG that generally outperforms tuned SGD, approaching the performance of tuned Adam. A PyTorch implementation is available at https://github.com/formll/dog
Forward citations
Cited by 4 Pith papers
-
Clipping the Price of Adaptivity at the Tail
Under a model-loss decomposition, clipping model outputs in tail events yields adaptive SCO bounds matching known-parameter optima up to logarithmic factors in uncertainty.
-
Projected gradient methods for nonconvex and stochastic smooth optimization: new complexities and auto-conditioned stepsizes
Develops constant-stepsize and auto-conditioned projected gradient methods plus stochastic variants that achieve new iteration complexity bounds for finding approximate stationary points in nonconvex smooth optimization.
-
Fantastic Pretraining Optimizers and Where to Find Them
After careful tuning, alternative optimizers speed up LLM pretraining by only 1.1x to 1.4x over AdamW, and the advantage shrinks with model scale.
-
Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models
HDET lets data-parallel replicas explore a spread of learning rates independently before averaging parameters, with an auto-LR controller driven by inter-replica loss differences to produce a self-adapting schedule wi...
Discussion (0). Sign in to comment.