REVIEW 6 cited by
Optimization for deep learning: theory and algorithms
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.
Forward citations
Cited by 6 Pith papers
-
Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness
Gradient descent on any feedforward network with Lipschitz-smooth, linearly bounded activations drives the minimum squared gradient norm to zero at rate O(1/T^(1/L)) without boundedness or overparameterization assumptions.
-
DeePoly: A High-Order Accuracy Scientific Machine Learning Framework for Function Approximation and Solving PDEs
DeePoly combines rough neural-network features with local polynomial bases and a linear solve to reach high-order accuracy on function approximation and PDEs.
-
Sharp higher order convergence rates for the Adam optimizer
Adam can achieve the accelerated momentum convergence rate locally on smooth strongly convex problems when its momentum and step size are tuned to the condition number, while RMSprop is shown to converge at the slower...
-
Mathematical analysis of the gradients in deep learning
For deep feedforward networks with piecewise-smooth activations, the autodiff gradient is shown to be the unique limit of gradients of smoothed activations, a limiting Frechet subgradient, and equal to the true gradie...
-
Bridging Jensen Gap for Max-Min Group Fairness Optimization in Recommendation
A dual-optimization method for group max-min fairness in recommender systems is proposed to reduce the mini-batch Jensen gap, but the central convergence theorem is internally inconsistent.
-
Averaged Adam accelerates stochastic optimization in the training of deep neural network approximations for partial differential equation and optimal control problems
Averaged variants of Adam, using a sliding-window arithmetic average or an exponential moving average of iterates, are reported to beat Adam and SGD on the tested PDE, optimal control, and CIFAR-10 benchmarks.
Discussion (0). Continue with ORCID to comment.