REVIEW 6 cited by
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
read the original abstract
Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability.We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer. Code is available at https://github.com/juntang-zhuang/Adabelief-Optimizer
Forward citations
Cited by 6 Pith papers
-
Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
Bounded trajectories of a broad class of GD optimizers (Adam, RMSprop, NAG, Adan, etc.) converge with polynomial rates to critical points of KL objectives with locally Lipschitz gradients, covering analytic-activation...
-
Gaussian processes on ray-guided transformed uniform grids for fast, flexible, and auto-differentiable adaptive source reconstruction in lens modelling
A new RTU grid method models the lensing source as a Gaussian process on a ray-transformed uniform grid, achieving comparable fits with roughly half the pixels per dimension and higher ELBOs on mock data.
-
WideDepth: Millimeter-Accurate Benchmark for Fisheye Depth Estimation
WideDepth supplies the first millimeter-accurate indoor fisheye depth benchmark together with a stereo generation pipeline and model adaptation technique.
-
On the Convergence of Muon and Beyond
Muon-MVR2 attains the optimal anytime convergence rate of ~O(T^{-1/3}) in stochastic non-convex settings under horizon-free schedules.
-
Disentangling the dark and stellar mass through precise lens modelling of the JWST observation of lensed quasar WFI2033--4723
Composite time-delay lens modelling of JWST WFI2033–4723 yields intermediate stellar IMF normalisation and a steep gNFW inner slope γ_in≃1.3 that is robust to the choice of external cosmology.
-
Ligandformer: A Graph Neural Network for Predicting Compound Property with Robust Interpretation
Ligandformer is a self-attention graph neural network framework that predicts compound properties, outputs attention maps for local structural interpretation, and claims improved robustness and generalization over pri...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.