REVIEW 8 cited by
Toward Understanding Why Adam Converges Faster Than SGD for Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
While stochastic gradient descent (SGD) is still the most popular optimization algorithm in deep learning, adaptive algorithms such as Adam have established empirical advantages over SGD in some deep learning applications such as training transformers. However, it remains a question that why Adam converges significantly faster than SGD in these scenarios. In this paper, we propose one explanation of why Adam converges faster than SGD using a new concept directional sharpness. We argue that the performance of optimization algorithms is closely related to the directional sharpness of the update steps, and show SGD has much worse directional sharpness compared to adaptive algorithms. We further observe that only a small fraction of the coordinates causes the bad sharpness and slow convergence of SGD, and propose to use coordinate-wise clipping as a solution to SGD and other optimization algorithms. We demonstrate the effect of coordinate-wise clipping on sharpness reduction and speeding up the convergence of optimization algorithms under various settings. We show that coordinate-wise clipping improves the local loss reduction when only a small fraction of the coordinates has bad sharpness. We conclude that the sharpness reduction effect of adaptive coordinate-wise scaling is the reason for Adam's success in practice and suggest the use of coordinate-wise clipping as a universal technique to speed up deep learning optimization.
Forward citations
Cited by 8 Pith papers
-
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
A theoretical analysis claims attention networks fail to learn residual features when time series steps have opposite signs, giving a possible explanation for the known advantage of linear residual models.
-
Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.
-
Avoiding spurious sharpness minimization broadens applicability of SAM
SAM's failure in language modeling is traced to a dominant 'logit path' that minimizes sharpness spuriously, and the proposed Functional-SAM, which removes that path, improves validation loss over AdamW and SAM.
-
Artificial Neural Networks for Magnetoencephalography: A review of an emerging field
A systematic review of 119 ANN-MEG studies shows rapid growth across decoding, BCI, clinical, modeling, and source-localization applications, with recurring reproducibility gaps.
-
Mechanistic Insights into Grokking from the Embedding Layer
Trainable embeddings in a simple MLP cause delayed generalization (grokking) on modular arithmetic, and a higher embedding learning rate plus balanced sampling accelerates it.
-
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Adaptive batch size schedules based on the norm test can run with FSDP-distributed Llama-style pretraining, but the reported runs do not consistently beat constant batch sizes and the convergence theorem does not cove...
-
Revisiting the Initial Steps in Adaptive Gradient Descent Optimization
Initializing Adam's second-order moment estimate to a non-zero value, rather than the default zero, reduces early training instability and improves generalization across several deep learning tasks.
-
Spatial navigation in preclinical Alzheimer's disease: A review
Spatial navigation performance, particularly path integration and wayfinding, correlates with AD biomarkers such as p-tau in cognitively unimpaired at-risk individuals and may enable earlier detection than episodic me...
Discussion (0). Continue with ORCID to comment.