Revisiting the Initial Steps in Adaptive Gradient Descent Optimization

Abulikemu Abuduweili; Changliu Liu

REVIEW 3 major objections 4 minor 52 references

Revisiting the Initial Steps in Adaptive Gradient Descent Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adam's default zero initialization of the second-order moment makes its first update a pure sign-descent step; replacing that zero with a data-driven or random positive value stabilizes training and improves final performance.

desk verdict A simple, empirically solid v0-initialization trick for Adam with a theory section that doesn't hold up — worth refereeing for the experiments, but the drift analysis needs to go. read the letter →

arxiv 2412.02153 v2 pith:BRWM2Y7Z submitted 2024-12-03 cs.LG cs.AI

Abulikemu Abuduweili , Changliu Liu This is my paper · ORCID

classification cs.LGcs.AI

keywords Adamoptimizersecond-ordermomentinitializationsigndescentadaptivegradientmethodstrainingstabilityTransformeroptimizationlearningratewarmup

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that Adam's default initialization of the second-order moment estimate ($v_0=0$) is not harmless: it makes the first update a pure sign-descent step whose per-coordinate magnitude is the learning rate, which is disproportionately large when early gradients are small. The proposed fix is to set $v_0$ to a positive value, either computed from gradient statistics on a small data sample ($v_{0,\mathrm{data}}$) or drawn from a scaled chi-squared distribution ($v_{0,\mathrm{rnd}}$). Across image classification, language modeling, machine translation, and GAN training, this change stabilizes the early phase and improves final accuracy, perplexity, BLEU, and FID for Adam and several adaptive optimizers. The practical consequence is that one default can be replaced by a cheap initialization, often matching or beating recent Adam variants and learning-rate warmup.

What carries the argument

The load-bearing object is the second-order moment estimate $v_t$ and its initial value $v_0$ in Adam's update (Eq. 2). The mechanism is the first-step collapse in Eq. 5: because $v_0=0$, the bias-corrected $\hat v_1$ is just $g_1^2/(1-\beta_2)$, so $\Delta\theta_1=-\alpha\,\operatorname{sign}(g_1)$. The paper quantifies the disturbance with the drift term $\mathrm{drift}_{v_t}(v_0)=\|\mathbb{E}[v_\infty]-\mathbb{E}[v_0]\|$ (Eq. 8), which is large for $v_0=0$ and zero for the ideal $v_0=\bar g^2+\sigma^2 I$ in the linear-loss Gaussian oracle; non-zero $v_0$ removes the sign-descent degeneracy at step one and keeps $v_t$ at the right scale from the start.

What would settle it

Ignoring the $\epsilon$ term, the mechanism predicts that with $v_0=0$ every first-step coordinate update has magnitude $\alpha$ (Eq. 5), while with non-zero $v_0$ every first-step coordinate satisfies $|\Delta\theta_1|<\alpha$ (Eq. 12); finding a single coordinate with $|\Delta\theta_1|>\alpha$, or observing identical first-step distributions under $v_0=0$ and $v_{0,\mathrm{data}}$, would falsify the claimed mechanism.

Watch

Extended reading notes

Core claim

With $m_0=v_0=0$, the first Adam update collapses to $\Delta\theta_1=-\alpha\,\operatorname{sign}(g_1)$ (Eq. 5): every coordinate is moved by the learning rate $\alpha$ regardless of gradient magnitude. In deep networks, especially Transformers, early gradients are small, so this sign-descent step is overly large and pushes parameters out of the initial flat region. Initializing $v_0$ to the expected squared gradient plus variance (Eq. 10) or to a scaled chi-squared random value (Eq. 11) makes the first step depend on gradient magnitude, keeps $|\Delta\theta_1|<\alpha$, and reduces the drift of $v_t$ toward its steady state. The paper shows empirically that this change stabilizes and improves Adam, AdamW, AdaBound, RAdam, and AdaBelief on CIFAR-10 and ImageNet classification, Penn Treebank LSTM language modeling, IWSLT'14 DE-EN Transformer translation, and CIFAR-10 DCGAN generation, and that it can replace or complement warmup.

Load-bearing premise

The analysis assumes the average size of the gradients stays about the same throughout training, so a $v_0$ chosen from early samples remains the right scale for the whole run.

Editorial extensions

If this is right

With non-zero $v_0$, Adam can train the IWSLT'14 Transformer without a warmup phase, whereas vanilla Adam without warmup fails to converge.
On ResNet-34/CIFAR-10, Adam with $v_{0,\mathrm{data}}$ raises test accuracy from 95.25% to 96.02%, and on IWSLT'14 DE-EN it raises BLEU from 30.14 to 33.64; similar gains appear for AdamW, AdaBound, RAdam, and AdaBelief.
The scale factor $\sigma$ is tuning-friendly: for both initialization strategies, values in $[1,1000]$ beat the zero baseline in the CIFAR-10 ablation.
Switching $v_0$ changes the solution basin: networks trained with $v_0=0$ and $v_{0,\mathrm{rnd}}$ are not linearly connected, and the $v_{0,\mathrm{rnd}}$ solution sits in a flatter loss landscape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test it, but the same first-step argument suggests that initializing the first moment $m_0$ away from zero, or setting the random scale per layer from measured gradient variances, could close the remaining gap between warmup and non-warmup training.
Because $v_{0,\mathrm{rnd}}$ injects random scale into each weight's update, it may act as a mild implicit regularizer; that could explain the flatter minima and the lack of linear mode connectivity, though the paper does not isolate this effect.
If the stationarity assumption holds even approximately, data-driven $v_0$ could be computed once and reused across runs or related tasks as a cheap calibration step; this is an extension beyond the paper's claims.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — here's the short version: this paper reports a cheap, broadly effective trick — initialize Adam's second-moment estimate v0 to a non-zero data-driven or random value — and backs it with consistent gains across five optimizers and four tasks. The first-step sign-descent observation (Eq. 5) is correct and already partly in Balles & Hennig and Kunstner et al.; the new bit is the systematic evaluation of non-zero v0 initialization and the two concrete schemes. Credit where due: the experiments are honest, five seeds, reasonable baselines, and the gains are real (e.g., Adam on IWSLT BLEU from 30.1 to 33.7; consistent but smaller gains elsewhere). The ablation on σ shows robustness from 1 to 1000, so the trick doesn't require careful tuning.

The soft spots are in the theory, not the data. The drift analysis in Sec 2.3 is ad hoc: defining drift as ||E[v∞]-E[v0]|| makes the ideal v0=E[g²] optimal by construction, which is circular. More seriously, the random initialization recommended with σ=100 gives mean v0 ≈ σ/(fan_in+fan_out) ≈ 0.1 for typical layers. The paper's own inequality says random init reduces drift only if 0 < λ < 2(ḡ²+σ²), and the paper also says gradients at initialization are small. For ḡ²+σ² ~ 1e-2 (generous), the bound is λ < 0.02, so the recommended λ≈0.1 violates it. That means the advertised theoretical mechanism cannot explain the empirical success of the very configuration tested. The same issue affects the standard-deviation claim: they state the std of vt is O((1-β2)σ²), missing a sqrt factor; it's actually O(σ²√(1-β2)). These are fixable — the likely real explanation is that a non-zero v0 simply reduces the early effective step size, which is exactly the sign-descent observation, and the drift metric is unnecessary.

Verdict: the paper is worth refereeing. The empirical finding is useful and easy to check, and the proposed init could reduce dependence on warmup. But the authors should either fix Sec 2.3 or cut the drift theory and present the step-size argument alone. I'd recommend a conditional accept after revision, not a desk reject.

Referee Report

3 major / 4 minor

Summary. The paper argues that Adam's default initialization of the second-order moment estimate, v0 = 0, causes the first update to degenerate into pure sign descent with step magnitude equal to the learning rate, and that this destabilizes training and degrades final performance in deep networks. It proposes two non-zero initialization schemes for v0: a data-driven one based on empirical gradient statistics (Eq. 10) and a random one based on a scaled chi-squared distribution (Eq. 11). The authors support the proposal with a linear-loss Gaussian-oracle model and a drift metric (Sec. 2.3), and report improvements across CIFAR-10, ImageNet, Penn Treebank, IWSLT'14, and CIFAR-10 GAN experiments for Adam, AdamW, AdaBound, RAdam, and AdaBelief.

Significance. If the empirical findings hold, this is a simple, computationally cheap modification with broad applicability: it requires changing only the initialization of the optimizer state and improves stability and final performance across multiple architectures and optimizers. The paper's strengths are its extensive empirical evaluation (five seeds, several tasks, several optimizers, an ablation over the scale parameter) and the release of code. The theoretical analysis, however, is heuristic and contains an internal inconsistency: the recommended random-initialization scale appears to violate the paper's own drift-reduction condition in the small-gradient regime it describes. The empirical results are valuable, but the theoretical framing needs substantial correction before the paper is publishable in its current form.

major comments (3)

[Sec. 2.3, Eq. (11)] The recommended random initialization contradicts the drift-reduction condition stated just above it. For v0 = λI, the paper derives that drift(v0 = λI) < drift(v0 = 0) iff 0 < λ < 2(gbar^2 + σ^2), where gbar^2 + σ^2 is the per-coordinate steady-state second moment of the gradient. With Eq. (11) and σ = 100, the mean of v0 is λ = 100/(fan_in + fan_out); for a 512×512 weight matrix this is about 0.098. But Sec. 2.2 motivates the problem precisely by saying the initial Transformer landscape is flat and 'gradients are correspondingly small,' implying that the per-coordinate gbar^2 + σ^2 is far below 0.05. Under the paper's own inequality, v0_rnd would then have a larger drift than zero initialization, not a smaller one, which is the opposite of the results in Tables 1–5. The ablation in Table 6 strengthens the contradiction: σ = 1000 (λ ≈ 1) still improves over σ = 0. This is not a minor approximation; either the drift metric is not the operative mechanism, or the recommended scale is in the wrong regime. The authors should measure the actual per-coordinate gradient second-moment scale and reconcile Eq. (11) with the inequality, or explicitly reclassify the drift analysis as heuristic.
[Sec. 2.3, Eqs. (6)–(7)] The steady-state argument assumes the stochastic gradient is generated by a stationary oracle with constant mean gbar and covariance σ^2I. Deep-network training is non-stationary: the weights change, and the distribution of g_t changes with them. Consequently, the claim that v0_data 'keeps E[vt] constant' and 'eliminates the need for adjustments' from the start is not established for the networks in Sec. 3; it holds only if the initial gradient statistics remain representative of the whole trajectory, which is not shown. This is load-bearing because the theoretical justification for both proposals rests on this transfer. I recommend either tracking per-coordinate second-moment statistics through training (e.g., comparing v0 to the running v_t) or presenting Eqs. (6)–(9) explicitly as intuition for a simplified model, not as a proof for deep networks.
[Sec. 2.3, Eq. (8)] The drift metric is introduced as 'reflecting how much the optimizer must adapt' and 'a smaller drift term indicates better stability,' but no derivation or empirical correlation links drift to update-step variance, loss spikes, or final performance. Since v0 = gbar^2 + σ^2I makes drift exactly zero by construction, the superiority of the data-driven rule in this metric is built in; as a theoretical justification it is circular. The experiments verify the proposal independently, but they do not test the drift metric itself. I would like to see either a direct test (e.g., compare drift values across v0 settings against measured instability) or a modest reformulation of the contribution as an empirical finding with a heuristic explanation.

minor comments (4)

[Sec. 2.3 (after Eq. (8))] The statement that the standard deviation of each coordinate of vt is of scale O((1−β2)σ^2) misses a square-root factor: for an exponential moving average with smoothing (1−β2), the fluctuation scale is O(sqrt(1−β2)·sd(g_t^2)), not O((1−β2)σ^2). The concentration conclusion still holds for β2 close to 1, but the stated scale is wrong.
[Secs. 2.3 and 3] The symbol σ is overloaded: it denotes the gradient noise covariance scale in Eq. (7) and a user-set multiplier in Eqs. (10) and (11). This makes the condition 0 < λ < 2(gbar^2 + σ^2) appear to involve the same σ as in Eq. (11), which it does not.
[Appendix B, Algorithm 2] The pseudocode divides accumulated gradients by num_samples = len(dataloader.dataset), which is inconsistent with the text's statement that only 5,000 random samples are used. Moreover, if the dataloader uses batches of size B, param.grad is the batch-averaged gradient, so the pseudocode estimates (1/B)·E[g^2], not E[g^2] as claimed in Eq. (10).
[Various] There are several typos: 'IWSTL' in the caption of Table 4, 'dataset dataset' in the caption of Table 5, and 'are are non linearly connected' in Appendix C.5.

Circularity Check

1 steps flagged · score 3.0 of 10

The drift-based 'stability' argument makes v0 = E[g^2] optimal by construction, but the paper's empirical claims remain independently tested.

self definitional [Section 2.3, Eqs. (8)-(9)]
"To assess the stability of the optimization process and the influence of the initial state, we define the drift of the second-order moment as: driftvt (v0) = ∥E[v∞] − E[v0]∥. ... Since vt directly determines the adaptive learning rate, a smaller drift term indicates better stability of optimization process. ... For non-zero initialization, v0 = ¯g2 + σ2I, the expected second moment remains constant for all E[vt] = ¯g2 + σ2I. Thus driftvt (v0 = ¯g2 + σ2I) = 0."

The paper defines stability operationally as the distance from the steady-state second moment, E[v∞] = ¯g2 + σ2I. Under that definition, any v0 equal to the steady-state value has zero drift by construction, so the 'optimality' of the data-driven initialization under the drift metric is a tautology rather than an independently derived prediction about training stability. The subsequent statement that this 'ensures a stable adaptive learning rate' restates the chosen metric rather than proving it from a more basic principle. This is partial circularity: the theoretical justification for v0,data is built into the definition of drift. The paper's empirical evaluations are independent of this metric, which is why the circularity score is moderate rather than high.

full rationale

The paper's main claim is empirical: non-zero initialization of Adam's second moment stabilizes training and improves final performance, verified across CIFAR-10, ImageNet, PTB, IWSLT'14, and GAN tasks. Those experiments are not predictions from the theory, and they do not reduce to the paper's fitted inputs. The Eq. (5) derivation of Adam's first step as sign descent is direct algebra from the update rule, not circular. The choice σ=100 for v0,rnd is an ablation-tuned hyperparameter, and although the paper frames it as 'tuning-friendly' rather than derived, that is not a fitted input renamed as a prediction. The only genuinely circular element is the drift analysis in Sec. 2.3: the metric in Eq. (8) is defined so that v0 = E[g^2] is optimal by construction, so the theoretical claim that data-driven initialization is maximally stable is self-definitional. The paper also asserts without derivation that a smaller drift implies better optimization stability; that is a modeling assumption, not a theorem. Separate from circularity, the skeptic's concern that the random initialization's scale σ=100 may violate the paper's own drift bound λ<2(¯g²+σ²) for small-gradient regimes is a correctness and internal-consistency issue, not a circularity issue. Overall, the central empirical contribution is self-contained, and the circularity is confined to the supporting drift-based justification.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on two free scaling hyperparameters (sigma values), a heuristic drift metric that is defined to favor the proposed initialization, and the assumption that gradient statistics at initialization remain representative during training. No new physical or mathematical entities are introduced.

free parameters (4)

sigma_data = 1
Scaling factor in data-driven initialization (Eq. 10); chosen by the authors and applied to all tasks; ablation in Table 6 shows performance depends on it.
sigma_rnd = 100
Scaling factor in random initialization (Eq. 11); selected based on CIFAR-10 performance and used for all other tasks; a free hyperparameter.
num_samples_data = 5000
Number of training samples used to estimate gradient statistics for v0,data; arbitrary and not ablated.
optimizer_hyperparameters = task-dependent (e.g., lr=0.001, beta1=0.9, beta2=0.999)
Tuned per task following prior protocols; not specific to the proposed method but impact the comparisons.

assumptions (4)

domain assumption Gradients in the analysis follow g_t ~ N(gbar, sigma^2 I) with fixed gbar and sigma (Eq. 6-7).
Used to derive expected moments and drift in Sec. 2.3; this stationary Gaussian model is not representative of deep network training.
ad hoc to paper The drift metric ||E[v_infinity] - E[v0]|| measures optimization instability (Eq. 8).
Introduced without proof that reducing this drift improves convergence; it is a heuristic proxy.
domain assumption Deep networks, especially Transformers, have flat loss landscapes and small initial gradients, making sign-descent steps disproportionately large (Sec. 2.2).
Relies on prior observations and citations [24-26]; the paper does not verify it beyond the IWSLT experiment.
standard math The Taylor expansion and independence assumption for 1/sqrt(vt) in Appendix A.1 are valid.
Used to approximate expected update steps; the uncorrelatedness of gt and vt is asserted, not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting the Initial Steps in Adaptive Gradient Descent Optimization." pith.science (2026). https://pith.science/paper/BRWM2Y7Z

@misc{pith2026241202153,
  author       = {Pith},
  title        = {Pith review of: Revisiting the Initial Steps in Adaptive Gradient Descent Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BRWM2Y7Z}},
  note         = {Machine review of arXiv:2412.02153}
}

abstract

Adaptive gradient optimization methods, such as Adam, are prevalent in training deep neural networks across diverse machine learning tasks due to their ability to achieve faster convergence. However, these methods often suffer from suboptimal generalization compared to stochastic gradient descent (SGD) and exhibit instability, particularly when training Transformer models. In this work, we show the standard initialization of the second-order moment estimation ($v_0 =0$) as a significant factor contributing to these limitations. We introduce simple yet effective solutions: initializing the second-order moment estimation with non-zero values, using either data-driven or random initialization strategies. Empirical evaluations demonstrate that our approach not only stabilizes convergence but also enhances the final performance of adaptive gradient optimizers. Furthermore, by adopting the proposed initialization strategies, Adam achieves performance comparable to many recently proposed variants of adaptive gradient optimization methods. Our code is available at https://github.com/Walleclipse/Adam_Initialization.

Figures

Figures reproduced from arXiv: 2412.02153 by the authors.

**Figure 2.** Histogram of update step distribution across coordinates. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Optimization of the saddle objective function with different methods. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Comparison of Vanilla Adam and Adam v0,rnd on (a) CIFAR-10 image classification task. (b) Penn Treebank language modeling task. (c) IWSTL’14 machine translation task. Training curve. We compare the training curves of Vanilla Adam and Adam with random initialization v0…

**Figure 5.** Figure 5: Comparison of the loss landscape around the convergent points of Transformer trained by [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Comparison of Vanilla Adam, Adam with warmup, and Adam v0,rnd on CIFAR-10 image classification task. The test perplexity of a 2-layer LSTM on the Penn Treebank language modeling task is shown in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 7.** Figure 7: Linear Mode Connectivity Analysis. (a) Error observed when linearly interpolating be [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 33 canonical work pages

[1]

SIAM review, 60(2):223–311, 2018

LéonBottou,FrankECurtis,andJorgeNocedal.Optimizationmethodsforlarge-scalemachine learning. SIAM review, 60(2):223–311, 2018

work page 2018
[2]

Adaptivesubgradientmethodsforonlinelearning and stochastic optimization.Journal of machine learning research, 12(7), 2011

JohnDuchi,EladHazan,andYoramSinger. Adaptivesubgradientmethodsforonlinelearning and stochastic optimization.Journal of machine learning research, 12(7), 2011

work page 2011
[3]

Neuralnetworksformachinelearning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

GeoffreyHinton,NitishSrivastava,andKevinSwersky. Neuralnetworksformachinelearning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

work page 2012
[4]

Adam: A method for stochastic optimization

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

arXiv 2014
[5]

Why adam outperforms gradient descent on language models: A heavy-tailed class imbalance problem

Robin Yadav, Frederik Kunstner, Mark Schmidt, and Alberto Bietti. Why adam outperforms gradient descent on language models: A heavy-tailed class imbalance problem. InOPT 2023: Optimization for Machine Learning

work page 2023
[6]

Toward understanding why adam converges faster than sgd for transformers

Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than sgd for transformers. arXiv preprint arXiv:2306.00204, 2023

arXiv 2023
[7]

KaiqiJiang,DhruvMalik,andYuanzhiLi.Howdoesadaptiveoptimizationimpactlocalneural network geometry?Advances in Neural Information Processing Systems, 36, 2024

work page 2024
[8]

Why transformers need adam: A hessian perspective.arXiv preprint arXiv:2402.16788, 2024

Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective.arXiv preprint arXiv:2402.16788, 2024

arXiv 2024

Show all 52 references

[9]

Adam can converge without any modification on update rules.Advances in neural information processing systems, 35:28386–28399, 2022

Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules.Advances in neural information processing systems, 35:28386–28399, 2022

2022
[10]

Ontheconvergenceofadamandbeyond

SashankJReddi,SatyenKale,andSanjivKumar. Ontheconvergenceofadamandbeyond. In International Conference on Learning Representations, 2018

2018
[11]

Onthevarianceoftheadaptivelearningrateandbeyond

LiyuanLiu,HaomingJiang,PengchengHe,WeizhuChen,XiaodongLiu,JianfengGao,andJi- aweiHan. Onthevarianceoftheadaptivelearningrateandbeyond. In InternationalConference on Learning Representations, 2020

2020
[12]

Adaptive gradient methods with dy- namic bound of learning rate

Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dy- namic bound of learning rate. InInternational Conference on Learning Representations, 2018

2018
[13]

Adabelief optimizer: Adapting stepsizes by the belief in observed gradients.Advances in neural information processing systems, 33:18795–18806, 2020

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients.Advances in neural information processing systems, 33:18795–18806, 2020

2020
[14]

Momentum is all you need for data-driven adaptive optimization

YizhouWang,YueKang,CanQin,HuanWang,YiXu,YulunZhang,andYunFu. Momentum is all you need for data-driven adaptive optimization. In2023 IEEE International Conference on Data Mining (ICDM), pages 1385–1390. IEEE, 2023

2023
[15]

Survey of optimization algo- rithms in modern neural networks.Mathematics, 11(11):2466, 2023

Ruslan Abdulkadirov, Pavel Lyakhov, and Nikolay Nagornov. Survey of optimization algo- rithms in modern neural networks.Mathematics, 11(11):2466, 2023

2023
[16]

No train no gain: Revisitingefficienttrainingalgorithmsfortransformer-basedlanguagemodels

Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisitingefficienttrainingalgorithmsfortransformer-basedlanguagemodels. Advances in Neural Information Processing Systems, 36, 2024. 10

2024
[17]

Attention is all you need.Advances in Neural Information Processing Systems, 2017

A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

2017
[18]

Dissecting adam: The sign, magnitude and variance of stochastic gradients

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pages 404–413. PMLR, 2018

2018
[19]

Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. InThe Eleventh International Conference on Learning Representations, 2023

2023
[20]

Heavy- tailedclassimbalanceandwhyadamoutperformsgradientdescentonlanguagemodels

Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy- tailedclassimbalanceandwhyadamoutperformsgradientdescentonlanguagemodels. arXiv preprint arXiv:2402.19449, 2024

2024 arXiv
[21]

Bert: Pre-training of deep bidirectional transformers for language understand- ing

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[22]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

2021
[23]

Escaping the big data paradigm with compact transformers

Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704, 2021

2021 arXiv
[24]

Improving transformer op- timization through better initialization

Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer op- timization through better initialization. InInternational Conference on Machine Learning, pages 4475–4483. PMLR, 2020

2020
[25]

Scan and snap: Understanding trainingdynamicsandtokencompositionin1-layertransformer

Yuandong Tian, Yiping Wang, Beidi Chen, and Simon S Du. Scan and snap: Understanding trainingdynamicsandtokencompositionin1-layertransformer. AdvancesinNeuralInformation Processing Systems, 36:71911–71947, 2023

2023
[26]

On the difficulty of training recurrent neural networks

R Pascanu. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2013

2013 arXiv
[27]

Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets.Advances in neural information processing systems, 31, 2018

2018
[28]

On the sdes and scal- ing rules for adaptive gradient algorithms.Advances in Neural Information Processing Systems, 35:7697–7711, 2022

Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scal- ing rules for adaptive gradient algorithms.Advances in Neural Information Processing Systems, 35:7697–7711, 2022

2022
[29]

ZhiyuanLi,SadhikaMalladi,andSanjeevArora.Onthevalidityofmodelingsgdwithstochas- tic differential equations (sdes).Advances in Neural Information Processing Systems, 34:12712– 12725, 2021

2021
[30]

Understanding the difficulty of training deep feedforward neural networks

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. InProceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

2010
[31]

On the convergence of adaptive gradient methods for nonconvex optimization.arXiv preprint arXiv:1808.05671, 2018

Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization.arXiv preprint arXiv:1808.05671, 2018

2018 arXiv
[32]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition ,pages 770–778, 2016. 11

2016
[33]

Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

IanGoodfellow,JeanPouget-Abadie,MehdiMirza,BingXu,DavidWarde-Farley,SherjilOzair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks.Communications of the ACM, 63(11):139–144, 2020

2020
[34]

Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

Sepp Hochreiter, J urgen Schmidhuber, and Corso Elvezia. Long short-term memory.Neural Computation, 9(8):1735–1780, 1997

1997
[35]

A stochastic approximation method.The annals of mathe- matical statistics, pages 400–407, 1951

Herbert Robbins and Sutton Monro. A stochastic approximation method.The annals of mathe- matical statistics, pages 400–407, 1951

1951
[36]

Some methods of speeding up the convergence of iteration methods.Ussr computational mathematics and mathematical physics, 4(5):1–17, 1964

Boris T Polyak. Some methods of speeding up the convergence of iteration methods.Ussr computational mathematics and mathematical physics, 4(5):1–17, 1964

1964
[37]

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

2019
[38]

Learningmultiplelayersoffeaturesfromtinyimages

AlexKrizhevsky,GeoffreyHinton,etal. Learningmultiplelayersoffeaturesfromtinyimages. 2009

2009
[39]

Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.International journal of computer vision, 115:211–252, 2015

2015
[40]

Building a large annotated corpus of english: The penn treebank.Computational linguistics, 19(2):313–330, 1993

Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank.Computational linguistics, 19(2):313–330, 1993

1993
[41]

fairseq: A fast, extensible toolkit for sequence modeling

MyleOtt,SergeyEdunov,AlexeiBaevski,AngelaFan,SamGross,NathanNg,DavidGrangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. InProceedings of NAACL-HLT 2019: Demonstrations, 2019

2019
[42]

Bleu: amethodforautomatic evaluation of machine translation

KishorePapineni,SalimRoukos,ToddWard,andWei-JingZhu. Bleu: amethodforautomatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

2002
[43]

Unsupervised representation learning with deep convolutional generative ad- versarial networks.arXiv preprint arXiv:1511.06434, 2015

Alec Radford. Unsupervised representation learning with deep convolutional generative ad- versarial networks.arXiv preprint arXiv:1511.06434, 2015

2015 arXiv
[44]

Gans trained by a two time-scale update rule converge to a local nash equilibrium.Ad- vances in neural information processing systems, 30, 2017

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Ad- vances in neural information processing systems, 30, 2017

2017
[45]

Asymmetric valleys: Beyond sharp and flat local minima

Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. Advances in neural information processing systems, 32, 2019

2019
[46]

Towards the- oretically understanding why sgd generalizes better than adam in deep learning.Advances in Neural Information Processing Systems, 33:21285–21296, 2020

Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards the- oretically understanding why sgd generalizes better than adam in deep learning.Advances in Neural Information Processing Systems, 33:21285–21296, 2020

2020
[47]

On the adequacy of untuned warmup for adaptive optimization

Jerry Ma and Denis Yarats. On the adequacy of untuned warmup for adaptive optimization. InProceedingsoftheAAAIConferenceonArtificialIntelligence ,volume35,pages8828–8836,2021

2021
[48]

Re- thinkingtheinceptionarchitectureforcomputervision

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinkingtheinceptionarchitectureforcomputervision. In ProceedingsoftheIEEEconferenceon computer vision and pattern recognition, pages 2818–2826, 2016

2016
[49]

SGDR: Stochastic gradient descent with warm restarts

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017

2017
[50]

Linear mode connectivity and the lottery ticket hypothesis

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InInternational Conference on Machine Learning, pages 3259–3269. PMLR, 2020. 12 A. Additional Details about Second-order Moment Initializat...

2020
[51]

The small denominator leads to excessively large initial updates, particularly when¯g is small orσ2 is large

approaching 0 ast → 0. The small denominator leads to excessively large initial updates, particularly when¯g is small orσ2 is large. This instability can cause erratic optimization behavior, especially in the early stages of training. Case 2: non-zero initialization (v0 = ¯g2 ...

2012
[52]

The model is trained with a length penalty of 1.0, a beam size of 5, and an initial warmup step size of10−7

implemented using the Fairseq package [41] on the IWSLT’14 German-to-English machine translation dataset. The model is trained with a length penalty of 1.0, a beam size of 5, and an initial warmup step size of10−7. Training is conducted for 55 epochs, and results are reported ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

SIAM review, 60(2):223–311, 2018

LéonBottou,FrankECurtis,andJorgeNocedal.Optimizationmethodsforlarge-scalemachine learning. SIAM review, 60(2):223–311, 2018

work page 2018

[2] [2]

Adaptivesubgradientmethodsforonlinelearning and stochastic optimization.Journal of machine learning research, 12(7), 2011

JohnDuchi,EladHazan,andYoramSinger. Adaptivesubgradientmethodsforonlinelearning and stochastic optimization.Journal of machine learning research, 12(7), 2011

work page 2011

[3] [3]

Neuralnetworksformachinelearning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

GeoffreyHinton,NitishSrivastava,andKevinSwersky. Neuralnetworksformachinelearning lecture 6a overview of mini-batch gradient descent.Cited on, 14(8):2, 2012

work page 2012

[4] [4]

Adam: A method for stochastic optimization

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

arXiv 2014

[5] [5]

Why adam outperforms gradient descent on language models: A heavy-tailed class imbalance problem

Robin Yadav, Frederik Kunstner, Mark Schmidt, and Alberto Bietti. Why adam outperforms gradient descent on language models: A heavy-tailed class imbalance problem. InOPT 2023: Optimization for Machine Learning

work page 2023

[6] [6]

Toward understanding why adam converges faster than sgd for transformers

Yan Pan and Yuanzhi Li. Toward understanding why adam converges faster than sgd for transformers. arXiv preprint arXiv:2306.00204, 2023

arXiv 2023

[7] [7]

KaiqiJiang,DhruvMalik,andYuanzhiLi.Howdoesadaptiveoptimizationimpactlocalneural network geometry?Advances in Neural Information Processing Systems, 36, 2024

work page 2024

[8] [8]

Why transformers need adam: A hessian perspective.arXiv preprint arXiv:2402.16788, 2024

Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective.arXiv preprint arXiv:2402.16788, 2024

arXiv 2024