REVIEW 7 cited by
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Adam has been shown to outperform gradient descent on large language models by a larger margin than on other tasks, but it is unclear why. We show that a key factor in this performance gap is the heavy-tailed class imbalance found in language tasks. When trained with gradient descent, the loss of infrequent words decreases more slowly than the loss of frequent ones. This leads to a slow decrease on the average loss as most samples come from infrequent words. On the other hand, Adam and sign-based methods are less sensitive to this problem. To establish that this behavior is caused by class imbalance, we show empirically that it can be reproduced across architectures and data types, on language transformers, vision CNNs, and linear models. On a linear model with cross-entropy loss, we show that class imbalance leads to imbalanced, correlated gradients and Hessians that have been hypothesized to benefit Adam. We also prove that, in continuous time, gradient descent converges slowly on low-frequency classes while sign descent does not.
Forward citations
Cited by 7 Pith papers
-
Gradient Multi-Normalization for Stateless and Scalable LLM Training
SinkGD is a stateless optimizer that balances gradients via square-root Sinkhorn iterations, matching or beating Adam and memory-efficient baselines on LLaMA pretraining with SGD-level memory.
-
Simple Convergence Proof of Adam From a Sign-like Descent Perspective
Adam's O(1/T^1/4) convergence is proven from a sign-like descent perspective, but the dimension-free claim depends on restrictive coordinate-wise assumptions.
-
Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
SGD with momentum can match Adam's performance in language modeling when trained with small batches and careful tuning, a result that contradicts several popular explanations for the optimizer gap.
-
Nonlinearly Preconditioned Gradient Methods under Generalized Smoothness
The paper introduces (L,\bar L)-anisotropic smoothness and proves O(1/K) convergence rates for nonlinearly preconditioned gradient methods, unifying gradient clipping, Adam, and Adagrad under one theory.
-
On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.
-
SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
S3, an optimizer with a p-th order momentum denominator, equal EMA coefficients, and Nesterov acceleration, is claimed to match AdamW's 100k-step perplexity at 50k steps while avoiding loss spikes.
-
Improving Adaptive Moment Optimization via Preconditioner Diagonalization
Rotating gradients into their SVD coordinate system before Adam-style updates can roughly halve the number of steps LLaMA models need to reach a given perplexity.
Discussion (0). Continue with ORCID to comment.