Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Grams: Gradient Descent with Adaptive Momentum Scaling

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new optimizer, Grams, claims to make each training step reduce loss at least as much as cautious Adam while guaranteeing global convergence.

desk verdict A real but small optimizer variant with a correct per-step lemma; the headline comparison and global convergence theorems are vacuous as stated due to an unsatisfiable step-size condition. read the letter →

arxiv 2412.17107 v3 pith:G2G67WK2 submitted 2024-12-22 cs.LG cs.AIcs.DSmath.OC

Yang Cao , Xiaoyu Li , Zhao Song This is my paper · ORCID
classification cs.LGcs.AIcs.DSmath.OC MSC 68T0790C2665K05
keywords GramsoptimizeradaptivemomentumscalingCautiousoptimizersAdamLionglobalconvergenceHamiltoniandescentlargelanguagemodeltraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Grams, an optimizer that separates update direction from update magnitude: the direction is set by the current gradient's sign, while an Adam-style momentum term is used only to scale how far each coordinate moves. The central claim is that this decoupling makes Grams descend the loss at least as fast as Cautious Adam and Cautious Lion on every step, under a stated step-size condition, and that Grams converges globally at an O(1/T) rate under standard smoothness, bounded-gradient, and PL assumptions. If correct, Grams is a drop-in optimizer that could speed up pre-training and fine-tuning of large language models without extra memory. The paper supports the claim with experiments on a 60M-parameter Llama pre-training run, WideResNet on CIFAR-10, and fine-tuning of Llama 3.2 1B/3B, where Grams matches or beats Adam, Lion, and their cautious variants.

What carries the argument

The load-bearing object is the Grams update rule itself, $w_{t+1}=w_t-\eta_t\,\mathrm{sign}(g_t)\circ|u_t|$: the gradient's sign fixes each coordinate's direction and Adam's normalized momentum $u_t$ supplies only the magnitude, so the first-order loss change is always $-\eta_t\langle|g_t|,|u_t|\rangle$, a sum of non-positive terms. The proof machinery is the L-smoothness bound of Fact 3.2 applied to this rule (Lemma B.2), the comparison to the Cautious mask via a sign-aware index split (Theorem B.3), and a Hamiltonian continuous-time formulation (Definition 4.5) in which the same sign-magnitude decoupling makes the Hamiltonian and loss decrease rates dominate the Cautious system. These pieces together carry the faster-descent and global-convergence conclusions.

What would settle it

Take any training step where the momentum and gradient disagree in sign in at least one coordinate and compute the two inner products in Theorem 4.3's condition; the second inner product is negative, so the minimum is negative and the required step size is negative, making the theorem's inequality inapplicable — a direct calculation that settles whether the stated proof supports the faster-descent claim.

Watch

Extended reading notes

Core claim

The paper's claim is that Grams' update rule, $w_{t+1}=w_t-\eta_t\,\mathrm{sign}(g_t)\circ|u_t|$, where $g_t$ is the gradient and $u_t$ is Adam's normalized momentum estimate, dominates cautious optimizers in loss descent. Lemma 4.2 bounds the one-step loss change by $-\eta_t\langle|g_t|,|u_t|\rangle + \frac{L\eta_t^2}{2}\|u_t\|_2^2$, and Theorem 4.3 asserts that whenever the step size satisfies $\eta_t \le \frac{2}{L\|u_t\|_2^2}\min\{\langle u_t\circ g_t, 1_{u_t\circ g_t\ge 0}\rangle, \langle u_t\circ g_t, 1_{u_t\circ g_t<0}\rangle\}$, Grams' loss decrease is at least that of the Cautious update and both are non-positive. Theorem 4.15 then states that under the PL condition Grams satisfies $L(w_T)-L^* \le \frac{4G}{\mu\eta T}(L(w_1)-L^*)$, the same O(1/T) global convergence rate as Adam. Empirically, Grams reports the lowest pretraining perplexity (38.60 vs 43.21 for C-Adam) on a 60M Llama run and the highest CIFAR-10 accuracy (90.55%) among the compared optimizers.

Load-bearing premise

The proof of Grams' faster-descent guarantee rests on a step-size condition that, because one of the two terms inside the minimum is non-positive, can only be met by a positive step size when the momentum and the gradient point in the same direction in every coordinate; in the typical case where they disagree in some coordinate, the theorem's premise does not hold.

Editorial extensions

If this is right

  • Grams can be dropped into existing Adam-based training loops with the same two moment buffers and default hyperparameters, since its update is a cheap sign and absolute-value modification of the Adam step.
  • Under L-smoothness and the theorem's step-size condition, Grams inherits monotone loss decrease on every step, matching the guarantee that makes Cautious optimizers attractive.
  • Under the PL condition, Grams converges from any finite initial gap at the O(1/T) rate of Adam, so its practical speed advantage is not bought by sacrificing a worst-case convergence guarantee.
  • On the reported settings, Grams reaches the lowest perplexity among compared optimizers on a 60M Llama pretraining run and the highest accuracy on WideResNet/CIFAR-10, suggesting gains compound in early-stage and full fine-tuning.
  • The continuous-time Hamiltonian analysis implies the decoupled update has a Lyapunov structure, so stability arguments used for Adam-class methods carry over in the limit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A condition implicit in Theorem 4.3 is that the step-size bound be positive; because the minimum in the bound is non-positive whenever any coordinate of momentum and gradient disagree, the theorem as stated covers only steps with no sign disagreement.
  • Since Grams' direction is the sign of the current gradient, its update differs from signSGD only by per-coordinate magnitudes; one testable extension is to analyze Grams in the nonconvex first-order-stationary-point framework used for signSGD.
  • The empirical comparisons are short-horizon (1,000 pre-training steps, 10 epochs on CIFAR-10); a natural extension is to verify whether Grams' margin persists at full-scale token budgets where AdamW's schedule and warm-up dominate.
  • The Hamiltonian formulation suggests that replacing $|u_t|$ by other magnitude statistics, such as $|u_t|^\alpha$ or a clipped norm, yields a family of algorithms with the same descent identity and tunable noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Grams, an optimizer whose update is w_{t+1} = w_t - eta_t sign(g_t) \circ |u_t|, where u_t is the Adam-style normalized momentum. The claimed contributions are (i) a per-step loss-descent comparison showing Grams decreases loss at least as fast as cautious optimizers (Theorem 4.3, formalized as Theorem B.3), (ii) Hamiltonian-dynamics descent and comparison results (Theorems 4.6-4.7, formalized as C.3-C.4), and (iii) a global O(1/T) convergence guarantee under L-smoothness, bounded gradients, and the PL condition (Theorem 4.15, formalized as D.1). The paper also reports pre-training, fine-tuning, and PEFT experiments with code released.

Significance. If the theoretical claims were valid, Grams would be a simple, drop-in optimizer with per-step domination over cautious methods and a global convergence guarantee, which would be a useful contribution. The paper also contains positive elements: the algorithm is clearly specified, the code is released, and the empirical study covers several realistic settings. However, the central comparison theorem has an unsatisfiable hypothesis as printed, and the global convergence proof relies on a comparison to Adam that no stated theorem provides. The headline theoretical guarantees are therefore not established; the empirical results alone are limited by single-run reporting and would not carry the paper's main claims.

major comments (3)
  1. [Section 4.2 / Theorem B.3] Theorem 4.3 and its formal version Theorem B.3 state the sufficient condition eta_t <= 2/(L||u_t||^2) * min{<u_t o g_t, 1_{u_t o g_t >= 0}>, <u_t o g_t, 1_{u_t o g_t < 0}>}. The second term is a sum over coordinates with u_{t,i} g_{t,i} < 0, so it is non-positive, and it is zero when no such coordinate exists. Hence the min is always non-positive, and no positive learning rate can satisfy the inequality unless the bound is vacuous in a different way. The claimed comparison Delta_L^{Grams} <= Delta_L^{Cautious} <= 0 therefore does not follow from the stated theorem in the generic case.
  2. [Appendix B, proof of Theorem B.3] The proof actually derives a different condition from the one printed in the theorem. The derivation shows that Delta_L^{Grams} - Delta_L^{Cautious} <= 0 follows from eta_t <= 2/(L||u_t||^2) * sum_{i in I^-} |u_{t,i}| |g_{t,i}|, a nonnegative quantity. This is not the min expression stated in Theorem B.3. Moreover, when I^- is empty, the proof's upper bound becomes L eta_t^2 ||u_t||^2 / 2, which does not establish the desired inequality; the cases I^- empty and g_{t,i}=0 require a separate argument. As printed, the theorem is not a consequence of its proof.
  3. [Section 4.4 / Appendix D, Theorem D.1] The global convergence proof invokes Theorem B.3 in Eq. (17) to replace per-step loss changes of Adam iterates w'_t with those of Grams iterates w_t. However, Theorem B.3 compares Grams with Cautious optimizers, not with Adam. No result in the paper bounds the loss descent of Adam relative to Grams, so this step is unjustified. Additionally, the argument implicitly requires Grams' loss values to be non-increasing so that the final iterate can be bounded by the averaged gradient bound; no such monotonicity is established under the stated assumptions. Thus Theorem 4.15 does not follow from the proof as written.
minor comments (4)
  1. [Appendix B] In the proof of Lemma B.1 Part 2, and similarly in Lemma B.2 Part 2, the smoothness lower bound is written with <g_t, w_{t+1} + w_t>; it should be <g_t, w_{t+1} - w_t>.
  2. [Appendix C] In the proof of Theorem C.3, the term <K(s_t), ...> should be <\nabla K(s_t), ...>. In the proof of Theorem C.4, the second term should be -<|\nabla L(w_t)|, |\nabla K(s_t)|> rather than -<|\nabla L(w_t)|, |\nabla L(w_t)|>.
  3. [Sections 4-5 and Appendix E] There are several presentation errors: 'CIF AR-10' should be 'CIFAR-10'; the abstract says 'Grams descents faster' and should say 'descends'; the PEFT paragraph refers to Table 7 when it should refer to Table 8; and the wording in Section 5.1 comparing C-Adam and Lion to 'their respective base optimizers, Adam and C-Lion' is confusing because C-Lion is not the base version of Lion.
  4. [Section 5 / Appendix E] The empirical evaluation would be substantially stronger with multiple seeds, standard deviations, and training curves. As reported, most tables give single final numbers without variance or significance information, which makes it hard to assess whether the observed gaps are reliable.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: Grams' central claims are derived from its definitions and external baselines; the only self-citation (SORSA in PEFT) is non-load-bearing, while the vacuous Theorem B.3 premise is a soundness gap rather than circularity.

  1. other [Section 5.2 (PEFT experiments), Table 4; reference [Cao24]]
    "We conducted parameter-efficient fine-tuning (PEFT) experiments on the Llama 3.2 3B model using the SORSA method [Cao24] and the first 100,000 rows of data from the MetaMathQA dataset [YJS+23]."

    This is the only self-citation in the paper. [Cao24] supplies the SORSA adapter used in a single PEFT experiment; it does not enter the derivation of the Grams update rule, the loss-descent comparison, or the global convergence bound. Those central results are derived from Definitions 3.7 and 4.1 together with external baselines [LRJ23, LCLL24], not from the authors' own prior work. The self-citation is therefore non-load-bearing for the main claims and is flagged only for completeness; it does not make any central result circular.

full rationale

I walked the claimed derivation chain: the Grams descent bound (Lemma 4.2/B.2) follows directly from L-smoothness, the update rule sign(g_t)∘|u_t|, and Fact A.6; the Cautious bounds are imported from the external reference [LCLL24], not from the authors' own work. The comparison in Theorem 4.3/B.3 is algebraically derived from these bounds and is not a fitted quantity renamed as a prediction. The Hamiltonian comparison (Theorem 4.7/C.4) is likewise derived from Definition 4.5 and the algebraic inequality Fact A.4. The global convergence proof (Theorem 4.15/D.1) cites [LRJ23] for Adam's rate and attempts to transfer that rate to Grams; that transfer is mathematically flawed because it invokes Theorem B.3, which compares Grams to Cautious optimizers rather than to Adam, and the chain of inequalities in Eq. (17) does not establish the claimed bound on Grams. However, a flawed proof step is a soundness gap, not circularity: no parameter is fitted to the target result, and no premise is defined in terms of the conclusion. Separately, Theorem B.3 is stated with an unsatisfiable step-size premise, since min{⟨u_t∘g_t, 1_{u_t∘g_t≥0}⟩, ⟨u_t∘g_t, 1_{u_t∘g_t<0}⟩} is non-positive whenever any coordinate of u_t and g_t disagree, so no positive η_t can satisfy the stated condition; again, this makes the theorem vacuous as printed but does not constitute circular reasoning under the rubric. The only self-citation, [Cao24], appears in one PEFT experiment and is non-load-bearing. Overall circularity score: 2.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The Grams update introduces no invented physical or mathematical entities. Its theoretical claims rest on standard smoothness, bounded-gradient, and PL assumptions, plus the Hamiltonian framework imported from [LCLL24]. The PL condition is the strongest structural assumption and is unlikely to hold for the deep learning losses the paper targets. No free parameters are fitted to data beyond standard optimizer hyperparameters.

assumptions (5)
  • domain assumption L is L-smooth (Assumption 4.12)
    Used in every loss-descent lemma (Lemma 3.9/B.1, Lemma 4.2/B.2) and in the comparison theorem B.3.
  • domain assumption Bounded gradients ||∇L(w)|| ≤ G (Assumption 4.11)
    Required for the Adam convergence lemma 4.14 quoted from [LRJ23] and used in Theorem 4.15.
  • domain assumption µ-PL condition (Assumption 4.13)
    Enables conversion of gradient-norm bounds into loss-gap bounds in Theorem 4.15. Unlikely to hold globally for deep learning losses.
  • domain assumption Hamiltonian descent framework from [LCLL24] (Definition C.1)
    Used in Theorems 4.6-4.7 to claim monotonic Hamiltonian descent and faster loss descent in continuous time. Not independently re-derived.
  • standard math Adam convergence lemma (Lemma 4.14, from [LRJ23])
    Imported as a black box. Its assumptions are stated in the paper only informally; the paper does not verify them in the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grams: Gradient Descent with Adaptive Momentum Scaling." pith.science (2026). https://pith.science/paper/G2G67WK2

@misc{pith2026241217107,
  author       = {Pith},
  title        = {Pith review of: Grams: Gradient Descent with Adaptive Momentum Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G2G67WK2}},
  note         = {Machine review of arXiv:2412.17107}
}
abstract

We introduce $\mathbf{G}$radient Descent with $\mathbf{A}$daptive $\mathbf{M}$omentum $\mathbf{S}$caling ($\mathbf{Grams}$), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning. Unlike traditional optimizers that directly integrate momentum into updates, Grams separates the update direction, derived from current gradients, from momentum, which is used solely for adaptive magnitude scaling. This approach enables Grams to achieve improved loss descent compared to state-of-the-art cautious and momentum-based optimizers. We theoretically demonstrate that Grams descents faster than other state-of-the-art optimizers and establish a global convergence guarantee for Grams. We also validate its effectiveness through extensive empirical evaluations. The results demonstrate Grams' superior performance, including faster convergence and better generalization, compared to widely-used optimizers such as Adam, Lion, and their cautious variants. Our results highlight Grams' potential as a transformative approach for efficiently training and fine-tuning large language models. Code is available at https://github.com/Gunale0926/Grams.

Figures

Figures reproduced from arXiv: 2412.17107 by the authors.

Figure 1
Figure 1. Convergence comparison on a simple convex function [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

  2. High-Order Matching for One-Step Shortcut Diffusion Models

    cs.CV 2025-02 reject novelty 4.0 of 10

    HOMO extends shortcut diffusion with acceleration and jerk supervision, but the proof of superior approximation is not supported and experiments lack error bars.

  3. Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation

    cs.CV 2025-02 reject novelty 4.0 of 10

    VLFM models video latent patches as a HiPPO-LegS polynomial flow and trains a flow matching model to generate frames, claiming bounded interpolation and extrapolation error.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    Improving adaptive moment optimization via preconditioner diagonalization

    Anonymous. Improving adaptive moment optimization via preconditioner diagonalization. In Submitted to The Thirteenth International Conference on Learning Representations , 2024. under review

    work page 2024

  4. [4]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning , pages 560--569. PMLR, 2018

    work page 2018

  5. [5]

    SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models

    Yang Cao. Sorsa: Singular values and orthonormal regularized singular vectors adaptation of large language models. arXiv preprint arXiv:2409.00055 , 2024

    work page Pith review arXiv 2024

  6. [6]

    When vision transformers outperform resnets without pre-training or strong data augmentations

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations , 2022

    work page 2022

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021

    arXiv

  8. [8]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems , 36, 2024

    work page 2024

Show all 46 references
  1. [9]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  2. [10]

    Incorporating nesterov momentum into adam, 2016

    Timothy Dozat. Incorporating nesterov momentum into adam, 2016

    2016

  3. [11]

    The road less scheduled

    Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled. arXiv preprint arXiv:2405.15682 , 2024

    2024 arXiv

  4. [12]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021

    2021

  5. [13]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning , pages 1842--1850. PMLR, 2018

    2018

  6. [14]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021

    2021 arXiv

  7. [15]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

    Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on , 14(8):2, 2012

    2012

  8. [16]

    Accelerated gradient descent escapes saddle points faster than gradient descent

    Chi Jin, Praneeth Netrapalli, and Michael I Jordan. Accelerated gradient descent escapes saddle points faster than gradient descent. In Conference On Learning Theory , pages 1042--1085. PMLR, 2018

    2018

  9. [17]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  10. [18]

    Accelerated mirror descent in continuous and discrete time

    Walid Krichene, Alexandre Bayen, and Peter L Bartlett. Accelerated mirror descent in continuous and discrete time. Advances in neural information processing systems , 28, 2015

    2015

  11. [19]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  12. [20]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009

    2009

  13. [21]

    Cautious optimizers: Improving training with one line of code

    Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. Cautious optimizers: Improving training with one line of code. arXiv preprint arXiv:2411.16085 , 2024

    2024

  14. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  15. [23]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 364, 2019

    1907 arXiv

  16. [24]

    Convergence of adam under relaxed assumptions

    Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. In Proceedings of the 37th International Conference on Neural Information Processing Systems , pages 52166--52196, 2023

    2023

  17. [25]

    Full parameter fine-tuning for large language models with limited resources

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qi jie Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. In Annual Meeting of the Association for Computational Linguistics , 2023

    2023

  18. [26]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Baker Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International Conference on Machine Learning , 2015

    2015

  19. [27]

    Hamiltonian descent methods

    Chris J Maddison, Daniel Paulin, Yee Whye Teh, Brendan O'Donoghue, and Arnaud Doucet. Hamiltonian descent methods. arXiv preprint arXiv:1809.05042 , 2018

    2018 arXiv

  20. [28]

    H-fac: Memory-efficient optimization with factorized hamiltonian descent

    Son Nguyen, Lizhang Chen, Bo Liu, and Qiang Liu. H-fac: Memory-efficient optimization with factorized hamiltonian descent. arXiv preprint arXiv:2406.09958 , 2024

    2024 arXiv

  21. [29]

    A method for solving the convex programming problem with convergence rate o (1/ \^ 2)

    Yurii Evgen’evich Nesterov. A method for solving the convex programming problem with convergence rate o (1/ \^ 2). In Dokl. akad. nauk Sssr , volume 269, pages 543--547, 1983

    1983

  22. [30]

    Le, Lukasz Kaiser, Karol Kurach, Ilya Sutskever, and James Martens

    Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Lukasz Kaiser, Karol Kurach, Ilya Sutskever, and James Martens. Adding gradient noise improves learning for very deep networks, 2017

    2017

  23. [31]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684--10695, 2022

    2022

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1--67, 2020

    2020

  25. [33]

    An overview of gradient descent optimization algorithms

    Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016

    2016 arXiv

  26. [34]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , pages 1139--1147. PMLR, 2013

    2013

  27. [35]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning , pages 4596--4604. PMLR, 2018

    2018

  28. [36]

    Adopt: Modified adam can converge with any _2 with the optimal rate

    Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong, Go Nagahara, Tomoshi Iiyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. Adopt: Modified adam can converge with any _2 with the optimal rate. arXiv preprint arXiv:2411.02853 , 2024

    2024 arXiv

  29. [37]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  30. [38]

    Variational principles for mirror descent and mirror langevin dynamics

    Belinda Tzen, Anant Raj, Maxim Raginsky, and Francis Bach. Variational principles for mirror descent and mirror langevin dynamics. IEEE Control Systems Letters , 2023

    2023

  31. [39]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321 , 2024

    2024 arXiv

  32. [40]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 2017

    2017

  33. [41]

    A lyapunov analysis of momentum methods in optimization

    Ashia C Wilson, Benjamin Recht, and Michael I Jordan. A lyapunov analysis of momentum methods in optimization. arXiv preprint arXiv:1611.02635 , 2016

    2016 arXiv

  34. [42]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  35. [43]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284 , 2023

    2023 arXiv

  36. [44]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Procedings of the British Machine Vision Conference 2016 , 2016

    2016

  37. [45]

    Adabelief optimizer: Adapting stepsizes by the belief in observed gradients

    Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795--18806, 2020

    2020

  38. [46]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu (Allen) Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. ArXiv , abs/2403.03507, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.