Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A stateless optimizer that pre-processes each SGD gradient with row-wise normalization and whitening claims Adam-level LLM performance at half the memory.

desk verdict Solid idea, honest paper, but the 2x speedup is only as good as the Adam baseline you choose; needs a fully re-run, code-released revision. read the letter →

arxiv 2412.13148 v3 pith:YTJYAWIU submitted 2024-12-17 cs.LG cs.AI

Chao Ma , Wenbo Gong , Meyer Scetbon , Edward Meeds This is my paper · ORCID
classification cs.LGcs.AI
keywords statelessoptimizergradientwhiteningnormalizationLLMpre-trainingmemory-efficienttrainingNewton-SchulziterationSGDAdam-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWAN is a stochastic optimizer that pre-processes each raw mini-batch gradient with two stateless matrix operations, row-wise normalization and whitening, and then applies a plain SGD update. The paper's central claim is that this suffices to replicate the benefits of Adam for pre-training large language models without storing any optimizer state. Empirically, SWAN uses the same memory as SGD, roughly half Adam's end-to-end footprint, and reaches the same validation perplexity as Adam while consuming about half as many tokens on 350M- and 1.3B-parameter models. The authors argue the two operations respectively stabilize the time-varying covariance of transformer gradients and counteract the local curvature of the loss landscape.

What carries the argument

The central object is the composed operator $\text{SWAN}(G) = \text{GradWhitening}(\text{GradNorm}(G))$ applied to the current gradient matrix $G \in \mathbb{R}^{m \times n}$, followed by a norm-preserving rescale. GradNorm($G$) divides each row by its root-mean-square across columns, while GradWhitening($G$) = $(GG^\top)^{-1/2} G$ is the closest orthogonal matrix to $G$, so the update uses only current-batch statistics. The fast variant replaces the exact inverse square root with a diagonal-substitution Newton\textendash{}Schulz iteration whose per-step cost is $O(m^2)$ instead of $O(m^3)$, bringing raw throughput close to Adam's without distributed computation.

What would settle it

Run the 350M or 1.3B pre-training twice: once with the full SWAN pipeline and once with GradWhitening replaced by element-wise diagonal whitening, or with the whitening matrix computed from a different mini-batch than the gradient. If the 2x token-speedup over Adam persists, the claim that matrix-level whitening counteracts local curvature is not the operative mechanism.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that the memory and convergence advantages of adaptive optimizers can be obtained by instantaneous, stateless pre-processing of SGD gradients. GradNorm standardizes each row of the gradient matrix by its root-mean-square across output dimensions, and GradWhitening multiplies the normalized gradient by $(GG^\top)^{-1/2}$, which orthogonalizes the gradient and is derived as a non-diagonal second-order preconditioner under the assumption that the local Hessian has the Kronecker form $I \otimes \tilde H$. No running averages or other state variables are kept. On LLaMA-style models from 60M to 1.3B parameters trained on the C4 dataset, SWAN matches or beats Adam's validation perplexity while cutting total memory by roughly half, and the 350M and 1.3B runs reach Adam's perplexity in roughly half the tokens.

Load-bearing premise

The load-bearing premise is that a single mini-batch's gradient outer product $G G^\top$ is a reliable proxy for the local curvature of the loss, so whitening with its inverse square root genuinely counteracts the Hessian rather than just rescaling noise; the paper specifically assumes the Hessian has the shared block-diagonal form $I \otimes \tilde H$.

Editorial extensions

If this is right

  • LLM pre-training can run with zero optimizer states, cutting total end-to-end memory by roughly 50% compared with Adam and up to about 70% when combined with per-layer training techniques.
  • At 350M and 1.3B scale, the token budget needed to reach a fixed evaluation perplexity is roughly halved, a about 2x speedup in steps or tokens.
  • The diagonal-substitution Newton\textendash{}Schulz scheme makes SWAN's raw throughput comparable to Adam's without distributed computation, so the token-efficiency gain translates into roughly 2x effective throughput.
  • The same hyperparameters work across model sizes, and a no-warmup, Adam-like setting already outperforms Adam in perplexity, suggesting the optimizer can be dropped in without extensive tuning.
  • Ablations show that removing either GradNorm or GradWhitening degrades final perplexity, so the combination, not simply a larger effective learning rate, carries the result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If single-batch whitening is a reliable curvature proxy, the same preprocessing could be composed with momentum or with other stateless gradient transforms; the paper explicitly leaves the composition effect open.
  • The quadratic analysis implies SWAN-type updates should transfer to other ill-conditioned, non-convex problems where Adam's diagonal preconditioner is a poor fit, though only toy Rastrigin tests are reported.
  • The additive-versus-multiplicative speedup evidence suggests SWAN's token advantage over Adam may grow with model scale, so running at 7B-13B would be the natural next check of the scaling claim.
  • Because GradWhitening cost depends on the smaller dimension of each weight matrix, very wide layers may need a block-wise or low-rank whitening variant, which the paper does not develop.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SWAN, a stateless optimizer that pre-processes the instantaneous SGD gradient with row-wise normalization (GradNorm) followed by whitening (GradWhitening), optionally accelerated by a diagonal-substitution Newton-Schulz scheme (NSDS). The authors claim that SWAN matches the memory footprint of SGD while achieving comparable or better validation perplexity than Adam on LLaMA-style pretraining, with up to 2x faster convergence in tokens on 350M and 1.3B models. The paper includes a theoretical analysis of GradNorm and GradWhitening under a simplified transformer block, plus experiments on 60M-1.3B models, ablations, and throughput measurements.

Significance. If the central claims hold, SWAN would be a practically significant contribution to memory-efficient LLM training: it achieves roughly 50% total memory reduction versus Adam on the measured setups, with no optimizer states for the linear layers to which it is applied. The paper is also visibly careful in several respects: it runs ablations for both components, studies the effect of learning-rate warmup, reports throughput with and without model parallelism, and attempts to connect the design to a theoretical learning-dynamics analysis. The empirical claim is nevertheless strongly tied to the choice of Adam baseline, and the paper's own reproduced Adam numbers differ from the quoted official numbers by more than SWAN's edge over the quoted baseline at 1.3B. The stateless framing is also broader than what is actually implemented, since non-linear parameters still use Adam.

major comments (4)
  1. [Section 6.1, Table 2; Appendix J] The headline '2x speedup' comparison is not based on a single consistent Adam baseline. Table 2 reports two Adam rows, 'Adam' (33.02/24.44/19.24/16.44) and 'Adam (cited)' (34.06/25.08/18.80/15.56), while Appendix J states that the authors' own reproductions 'generally obtain slightly worse results for ... Adam for larger models.' At 1.3B, the reproduced Adam differs from the cited Adam by 0.88 perplexity, while SWAN-0's edge over the cited Adam is only 0.43 (15.13 vs 15.56). The speedup row '>2.4X' is therefore not robust to replacing quoted baselines with consistently re-run baselines. Please report speedups against a fully re-run, identically configured Adam baseline, or explicitly quantify the sensitivity of the speedup to the choice of Adam reference.
  2. [Section 6.1, Appendix J; Abstract] The claim that SWAN 'eliminates the need to store any optimizer states' is overstated as stated. Appendix J says SWAN is applied only to all linear projection weights in transformer blocks and that 'the rest of the non-linear parameters still uses Adam as the default choice.' Thus the full training run still maintains Adam optimizer states for a subset of parameters, and the end-to-end memory numbers in Figure 1(c) reflect that hybrid setup. The abstract and title should be scoped to 'stateless for linear layers' or the remaining states should be eliminated as well.
  3. [Section 5.3, Assumption 1; Proposition 1] The theoretical justification for GradWhitening as a curvature-correcting update relies on Assumption 1 (H = I⊗Htilde) and on replacing the expectation E[vec(G)vec(G)^T] with the single-mini-batch sample GG^T. The paper does not provide evidence that this single-sample estimate is accurate for the Hessian/FIM blocks of real transformers; Proposition 1 is derived under the simplified transformer block and full-batch dynamics. Given that this assumption is the main link between GradWhitening and second-order optimization, I ask the authors to either add a direct empirical check of the approximation (e.g., correlation between GG^T and Hessian blocks during a real run) or explicitly weaken the claim to 'a heuristic motivated by Kronecker structure.'
  4. [Section 4.2, Algorithm 2; Appendix B.3] The NSDS acceleration is critical to the throughput claim, but it is presented as a heuristic whose selection is not documented in a falsifiable way. Appendix B.3 shows that 1 NSDS iteration makes the loss diverge, while 2 iterations work, and the step size beta=0.4 is chosen empirically. Since SWAN‡ is the variant used for the headline speedup, please report the sensitivity of the final perplexity and throughput to the NSDS iteration count and beta, or at least state that the reported numbers hold only for the specific values used.
minor comments (4)
  1. [Equation (1)] The definition of s appears to be missing a square on the gradient entries: it should be s = sqrt(1/n * sum_j G_{:,j}^2), not s = sqrt(1/n * sum_j G_{:,j}).
  2. [Section 2] There is a typo in the related-work discussion: 'making the optimizer fully statless' should be 'stateless.'
  3. [Table 1] The memory-usage formulas for Apollo and other entries are garbled in the typeset table; please reformat them so the reader can verify the comparison.
  4. [Section 6.3, Equations (7)-(10)] The counterfactual additive curve is defined using a set of PPL thresholds P_i, but it is not specified how these thresholds are chosen and how many are used; please clarify the exact procedure and include the number of thresholds N.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims rest on external Adam benchmarks and conditional theory, and the only self-citation is a non-load-bearing afterword.

full rationale

SWAN's update rule is explicitly defined by Equations (1) and (2), and the theoretical analysis in Section 5 is presented conditionally (Assumption 1, simplified transformer block, Tian et al. assumptions) rather than as a derivation that forces the empirical outcomes. The paper itself states that 'the complete theoretical properties of SWAN is an open question which we leave for future work,' so the analysis is not a self-contained derivation that reduces to its inputs. The empirical 2x speedup claim is an external comparison against Adam and other optimizers on validation perplexity; it does not rename any fitted parameter as a prediction. The main caveat is that Table 2 mixes official and reproduced Adam numbers (Appendix J notes reproduced Adam is 'slightly worse' for larger models), but this is a baseline-consistency and reproducibility concern, not a circularity of the SWAN derivation itself. The only self-citation, Scetbon et al. 2025 in Section 7, is a follow-up remark about MNGD and is not used to justify GradNorm, GradWhitening, Assumption 1, or the empirical results, so it is not load-bearing. Consequently, no step satisfies the standard for a circularity finding: no quoted equation is identical to another by construction, and no fitted value is relabeled as a prediction. The baseline discrepancy should be weighed as a correctness risk, not as circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The empirical central claim rests on an experimental protocol in which SWAN is applied to linear layer weights while non-linear parameters still use Adam, and on comparisons against quoted baseline numbers. The theoretical justifications rely on the Kronecker Hessian assumption, the STB dynamics assumptions inherited from Tian et al. (2023), and several technical conditions, some of which are not satisfied in practice. Hyperparameters such as learning rates, NSDS step size, and iteration counts are tuned empirically. No new entities are introduced.

free parameters (5)
  • Global learning rate for SWAN-0 = 0.001 for 60M/130M/350M; effective 0.0003 for 1.3B with alpha=0.3
    SWAN-0 borrows Adam's learning rate without tuning, but at 1.3B the authors scale alpha down because Adam's optimal LR is smaller, which is a per-model fitted adjustment.
  • Global learning rate and scaling factor for SWAN-dagger/SWAN-double-dagger = eta=0.02, alpha=0.05, effective LR 0.001 across all model sizes
    Chosen by a small grid search over {0.01, 0.02, 0.05} and setting alpha so the effective LR returns to 0.001, as described in Appendix J.
  • NSDS step size beta and iteration count K for SWAN-double-dagger = beta=0.4, K=2
    Tuned for performance in the condition-number experiments and the ablation in Table 4; not derived from theory.
  • Naive Newton-Schulz step size beta and iteration count K for SWAN-0/dagger = beta=0.8, K=10
    Set heuristically as 'usually sufficient' in Appendix J; no rigorous criterion.
  • Positions of diagonal substitution in NSDS
    The placement of Diag() operations in Eq. (4) was determined by performance on synthetic datasets, as stated in Section 4.2.
assumptions (5)
  • ad hoc to paper Assumption 1: the local Hessian has shared block-diagonal Kronecker structure H = I_n ⊗ H_tilde, with H_tilde in R^{m x m}.
    Introduced in Section 5.3 to justify GradWhitening as a second-order update; only argued for a simplified transformer block, not verified for real LLMs.
  • domain assumption Assumptions inherited from Theorem 1 of Tian et al. (2023) on simplified transformer block dynamics, used by Theorem 1 and Proposition 1.
    The paper states it inherits these assumptions but does not restate or verify them in this work.
  • domain assumption Lemma B.6 of Zhao et al. (2024a) about asymptotic coordinate dominance is used to derive the shared Hessian block structure in Proposition 1.
    The lemma originates in GaLore's analysis and its applicability to the STB Hessian is not established in the paper.
  • ad hoc to paper Stationary back-propagated gradient assumption in the proof of Theorem 1: each row of V moves along a fixed direction mu_i.
    Appendix D invokes a 'stationary back-propagated gradient assumption' to conclude u_i moves along a fixed direction; this is a non-trivial condition not derived from the STB dynamics.
  • standard math Theorem 2 assumes the initialization distribution assigns zero probability to any zero-Lebesgue-measure set, and the one-step result additionally requires W to lie on the Stiefel manifold.
    These are technical conditions for the quadratic convergence theorem; the Stiefel manifold initialization is not satisfied in LLM training, limiting the theorem's practical applicability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training." pith.science (2026). https://pith.science/paper/YTJYAWIU

@misc{pith2026241213148,
  author       = {Pith},
  title        = {Pith review of: SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTJYAWIU}},
  note         = {Machine review of arXiv:2412.13148}
}
abstract

Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they often require to maintain optimizer states throughout training, which can result in memory requirements several times greater than the model footprint. This overhead imposes constraints on scalability and computational efficiency. Stochastic Gradient Descent (SGD), in contrast, is a stateless optimizer, as it does not track state variables during training. Consequently, it achieves optimal memory efficiency. However, its capability in LLM training is limited (Zhao et al., 2024b). In this work, we show that pre-processing SGD in a stateless manner can achieve the same performance as the Adam optimizer for LLM training, while drastically reducing the memory cost. Specifically, we propose to pre-process the instantaneous stochastic gradients using normalization and whitening. We show that normalization stabilizes gradient distributions, and whitening counteracts the local curvature of the loss landscape. This results in SWAN (SGD with Whitening And Normalization), a stochastic optimizer that eliminates the need to store any optimizer states. Empirically, SWAN has the same memory footprint as SGD, achieving $\approx 50\%$ reduction on total end-to-end memory compared to Adam. In language modeling tasks, SWAN demonstrates comparable or even better performance than Adam: when pre-training the LLaMA model with 350M and 1.3B parameters, SWAN achieves a 2x speedup by reaching the same evaluation perplexity using half as many tokens.

Figures

Figures reproduced from arXiv: 2412.13148 by the authors.

Figure 1
Figure 1. SWAN Performance on LLM Pretraining. (a) and (b): On both 350M and 1.3B LLama architectures, SWAN achieves over 2X speed-up compared to Adam in terms of tokens seen. (c): Memory footprint. We measure end-to-end memory usage during full-model training with a batch size of 1 sequence. SWAN achieves nearly 100% reduction in optimizer states memory, and 50% total memory reduction (up to 70% when combined with per-layer … view at source ↗
Figure 2
Figure 2. Illustration of GradNorm and GradWhitening operators. In GradNorm operator, we perform standardization across the output dimensions (columns), using statistics computed row￾wise. In GradWhitening operator (illustration adapted from Huang et al. (2019)), we treat each column of the gradient matrix G as a separate data sample. Then, GradWhitening can be seen as stretching/squeezing the data such that the covariance ma… view at source ↗
Figure 3
Figure 3. SWAN Optimizer. When diag is set to True, we cover the fast variant denoted by SWAN‡ . 4.1 SWAN UPDATE RULES In SWAN (Algorithm 1), the raw SGD gradient Gt is processed by the operations below2 :  G˜ (t) ← GradNorm(G(t) ) ∆W(t) ← GradWhitening(G˜ (t) ) (SWAN) The weight is then updated by W(t+1) = W(t) − η∆W(t) . The GradNorm operator (Equation (1)) denotes the normalization of the gradient matrix row-wise ( [PITH… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison of convergence rate of different methods on LLM pretraining tasks. The [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparative analysis of SWAN and Adam optimizers: speedup ratios and perplexity [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies on 130M model. (a) Ablation on the contribution of each components in [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Comparison between different whitening schemes. (a) Performance comparison by [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Normalized Hessian Blocks of size MC × MC along the diagonal direction of the Hessian, obtained from numerically solving the STB ODE (with n = 12, MC = 10) (1) given by the full-batch dynamics (i.e., removing noise in Equation (16)). During all training steps, we analy…
Figure 9
Figure 9. Figure 9: KL divergence comparison of gradient distributions against initial gradient distribution [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: Comparison of convergence rate of different methods on quadratic and non-convex [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Gradient Multi-Normalization for Stateless and Scalable LLM Training

    cs.LG 2025-02 conditional novelty 7.0 of 10

    SinkGD is a stateless optimizer that balances gradients via square-root Sinkhorn iterations, matching or beating Adam and memory-efficient baselines on LLaMA pretraining with SGD-level memory.

  2. Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning

    cs.MM 2026-04 conditional novelty 6.5 of 10

    Localized Muon-style spectral orthogonalization of primary and constraint momentum subspaces (SIFT) reduces objective conflicts and improves constrained LLM steering across four tasks.

  3. Muse: Representation Geometry of Muon Beyond Normalized Momentum

    cs.LG 2026-07 conditional novelty 6.0 of 10

    The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.

  4. Low-rank Momentum Factorization for Memory Efficient Training

    cs.LG 2025-07 reject novelty 6.0 of 10

    MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.

  5. Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Optimizers such as Adam, Shampoo, and SOAP are unified as structured Fisher approximations, and two new derived optimizers, RACS and Alice, achieve faster LLaMA pre-training than Adam at lower memory.

Reference graph

Works this paper leans on

77 extracted references · 24 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Scalable second order optimization for deep learning, 2021

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning, 2021. URL https://arxiv.org/abs/2002.09018

    arXiv 2021

  3. [3]

    Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.06450, 2016. URL https://api.semanticscholar.org/CorpusID:8236317

    arXiv 2016

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    Modular duality in deep learning, 2024 a

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning, 2024 a . URL https://arxiv.org/abs/2410.21265

    arXiv 2024

  6. [7]

    Old optimizer, new norm: An anthology, 2024 c

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology, 2024 c . URL https://arxiv.org/abs/2409.20325

    arXiv 2024

  7. [8]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp.\ 560--569. PMLR, 2018

    2018

  8. [9]

    DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Aixin ...

    arXiv 2024

Show all 77 references
  1. [10]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jef...

    2020

  2. [11]

    Preconditioned spectral descent for deep learning

    David E Carlson, Edo Collins, Ya-Ping Hsieh, Lawrence Carin, and Volkan Cevher. Preconditioned spectral descent for deep learning. Advances in neural information processing systems, 28, 2015

    2015

  3. [12]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho - Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms. In NeurIPS, 2023

    2023

  4. [13]

    Robustness to unbounded smoothness of generalized signsgd

    Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems, 35: 0 9955--9968, 2022

    2022

  5. [14]

    Momentum improves normalized sgd

    Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pp.\ 2260--2268. PMLR, 2020

    2020

  6. [15]

    A general system of differential equations to model first-order adaptive algorithms

    Andr \'e Belotto Da Silva and Maxime Gazeau. A general system of differential equations to model first-order adaptive algorithms. Journal of Machine Learning Research, 21 0 (129): 0 1--42, 2020

    2020

  7. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  8. [17]

    Duchi, Elad Hazan, and Yoram Singer

    John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12: 0 2121--2159, 2011. URL https://api.semanticscholar.org/CorpusID:538820

    2011

  9. [18]

    Kronecker-factored approximate curvature for modern neural network architectures

    Runa Eschenhagen, Alexander Immer, Richard Turner, Frank Schneider, and Philipp Hennig. Kronecker-factored approximate curvature for modern neural network architectures. Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [19]

    A trace-restricted kronecker-factored approximation to natural gradient

    Kaixin Gao, Xiaolei Liu, Zhenghai Huang, Min Wang, Zidong Wang, Dachuan Xu, and Fan Yu. A trace-restricted kronecker-factored approximation to natural gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 7519--7527, 2021

    2021

  11. [20]

    Eigenvalue-corrected natural gradient based on a new approximation

    Kaixin Gao, Zheng-Hai Huang, Xiaolei Liu, Min Wang, Shuangling Wang, Zidong Wang, Dachuan Xu, and Fan Yu. Eigenvalue-corrected natural gradient based on a new approximation. Asia-Pacific Journal of Operational Research, 40 0 (01): 0 2340005, 2023

    2023

  12. [21]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis

    Thomas George, C \'e sar Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018

    2018

  13. [22]

    Shampoo: Preconditioned stochastic tensor optimization, 2018

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018. URL https://arxiv.org/abs/1802.09568

    2018 arXiv

  14. [23]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. ArXiv, abs/2402.03293, 2024. URL https://api.semanticscholar.org/CorpusID:267412117

    2024 arXiv

  15. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [25]

    Decorrelated batch normalization

    Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 791--800, 2018

    2018

  17. [26]

    Iterative normalization: Beyond standardization towards efficient whitening

    Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 4874--4883, 2019

    2019

  18. [27]

    Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information

    Dongseong Hwang. Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information. arXiv preprint arXiv:2405.12807, 2024

    2024 arXiv

  19. [28]

    An isometric stochastic optimizer

    Jacob Jackson. An isometric stochastic optimizer. arXiv preprint arXiv:2307.12979, 2023

    2023 arXiv

  20. [29]

    Three factors influencing minima in sgd

    Stanis aw Jastrz e bski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017

    2017 arXiv

  21. [30]

    How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    Kaiqi Jiang, Dhruv Malik, and Yuanzhi Li. How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [31]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  23. [32]

    No train no gain: Revisiting efficient training algorithms for transformer-based language models

    Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [33]

    Exploring low rank training of deep neural networks

    Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N Gomez. Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209.13569, 2022

    2022 arXiv

  25. [34]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015

    2015

  26. [35]

    Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition

    Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Val \'e rie Gar \`e s, Mounir Haddou, and Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. arXiv preprint arXiv:2201.10285, 2022

    2022 arXiv

  27. [36]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  28. [37]

    Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be

    Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. arXiv preprint arXiv:2304.13960, 2023

    2023 arXiv

  29. [38]

    Heavy-tailed class imbalance and why adam outperforms gradient descent on language models

    Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. arXiv preprint arXiv:2402.19449, 2024

    2024 arXiv

  30. [39]

    Towards faster training of global covariance pooling networks by iterative matrix square root normalization

    Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 947--955, 2018

    2018

  31. [40]

    Preconditioned stochastic gradient descent

    Xi-Lin Li. Preconditioned stochastic gradient descent. IEEE transactions on neural networks and learning systems, 29 0 (5): 0 1454--1466, 2017

    2017

  32. [41]

    Relora: High-rank training through low-rank updates

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations, 2023. URL https://api.semanticscholar.org/CorpusID:259836974

    2023

  33. [42]

    Can we remove the square-root in adaptive gradient methods? a second-order perspective

    Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E Turner, and Alireza Makhzani. Can we remove the square-root in adaptive gradient methods? a second-order perspective. arXiv preprint arXiv:2402.03496, 2024

    2024 arXiv

  34. [43]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. CoRR, abs/2305.14342, 2023

    2023 arXiv

  35. [44]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster) . OpenReview.net, 2019

    2019

  36. [45]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  37. [46]

    Kronecker-factored curvature approximations for recurrent neural networks

    James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018

    2018

  38. [47]

    Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization

    Jonathan Mei, Alexander Moreno, and Luke Walters. Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization. In Uncertainty in Artificial Intelligence, pp.\ 1412--1422. PMLR, 2023

    2023

  39. [48]

    A theory on adam instability in large-scale machine learning

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023

    2023 arXiv

  40. [49]

    Introductory lectures on convex optimization: A basic course, volume 87

    Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013

    2013

  41. [50]

    The ademamix optimizer: Better, faster, older

    Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137, 2024

    2024 arXiv

  42. [51]

    Fishy: Layerwise fisher approximation for higher-order neural network optimization

    Abel Peirson, Ehsan Amid, Yatong Chen, Vladimir Feinberg, Manfred K Warmuth, and Rohan Anil. Fishy: Layerwise fisher approximation for higher-order neural network optimization. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022

    2022

  43. [52]

    Curvature-informed sgd via general purpose lie-group preconditioners

    Omead Pooladzandi and Xi-Lin Li. Curvature-informed sgd via general purpose lie-group preconditioners. arXiv preprint arXiv:2402.04553, 2024

    2024 arXiv

  44. [53]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020. URL...

    2020

  45. [54]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML , volume 80 of Proceedings of Machine Learning Research, pp.\ 4603--4611. PMLR , 2018

    2018

  46. [55]

    A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale

    Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. a...

    2023 arXiv

  47. [56]

    Fast differentiable matrix square root and inverse square root

    Yue Song, Nicu Sebe, and Wei Wang. Fast differentiable matrix square root and inverse square root. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (6): 0 7367--7380, 2022

    2022

  48. [57]

    Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention

    Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying multilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.00535, 2023

    2023 arXiv

  49. [58]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

    Tijmen Tieleman. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4 0 (2): 0 26, 2012

    2012

  50. [59]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  51. [60]

    Llama 2: Open foundation and fine-tuned chat models, 2023 b

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  52. [61]

    Orthogonalising gradients to speed up neural network optimisation

    Mark Tuddenham, Adam Pr \"u gel-Bennett, and Jonathan Hare. Orthogonalising gradients to speed up neural network optimisation. arXiv preprint arXiv:2202.07052, 2022

    2022 arXiv

  53. [62]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  54. [63]

    4-bit shampoo for memory-efficient network training

    Sike Wang, Pan Zhou, Jia Li, and Hua Huang. 4-bit shampoo for memory-efficient network training. arXiv preprint arXiv:2405.18144, 2024

    2024 arXiv

  55. [65]

    No more adam: Learning rate scaling at initialization is all you need

    Minghao Xu, Lichuan Xiang, Xu Cai, and Hongkai Wen. No more adam: Learning rate scaling at initialization is all you need. arXiv preprint arXiv:2412.11768, 2024 b

    2024 arXiv

  56. [66]

    Principal whitened gradient for information geometry

    Zhirong Yang and Jorma Laaksonen. Principal whitened gradient for information geometry. Neural Networks, 21 0 (2-3): 0 232--240, 2008

    2008

  57. [67]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019

    1904 arXiv

  58. [68]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  59. [69]

    Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    2020

  60. [70]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  61. [71]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402.16788, 2024 a

    2024 arXiv

  62. [72]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024 b

    2024 arXiv

  63. [73]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu (Allen) Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. ArXiv, abs/2403.03507, 2024 a . URL https://api.semanticscholar.org/CorpusID:268253596

    2024 arXiv

  64. [75]

    Deconstructing what makes a good optimizer for language models, 2024 c

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models, 2024 c . URL https://arxiv.org/abs/2407.07972

    2024 arXiv

  65. [76]

    Apollo: Sgd-like memory, adamw-level performance

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270, 2024

    2024 arXiv

  66. [77]

    The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects

    Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv:1803.00195, 2018

    2018 arXiv

  67. [78]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  68. [79]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  69. [80]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.