Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Pre-Training LLMs on a budget: A comparison of three optimizers

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AdamW tops downstream tests in three-optimizer LLM comparison

desk verdict A careful, honest empirical comparison that is worth reading for its muP validation and practical scope, but the headline AdamW-is-clearly-best claim rests on single runs and needs to be softened or replicated. read the letter →

arxiv 2507.08472 v2 pith:ORN6RXF5 submitted 2025-07-11 cs.LG cs.AI

Joel Schlotthauer , Christian Kroos , Chris Hinze , Viktor Hangya , Luzian Hahn , Fabian Küch This is my paper
classification cs.LGcs.AI
keywords optimizercomparisonLLMpre-trainingAdamWLionSophiaMaximalUpdateParametrizationhyperparametertransferlimitedcomputebudget
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pre-training a ~2.7-billion-parameter language model on 60 billion tokens is a big enough expense that the choice of optimizer should be made on evidence, not habit. This paper compares AdamW, Lion, and Sophia under exactly that budget, with each optimizer's hyperparameters tuned separately on small proxy models and transferred to the full-size model. The central finding is that the three optimizers land in roughly the same performance range but pull in different directions: AdamW gives the best downstream task accuracy, Sophia reaches the lowest training and validation loss, and Lion trains fastest. A practical consequence, if the result holds, is that minimizing loss and optimizing a benchmark are not the same objective, and compute-limited teams should pick an optimizer according to which of those they actually want.

What carries the argument

The load-bearing mechanism is the Maximal Update Parametrization (µP), a scaling scheme that keeps hyperparameter optima invariant to model width, paired with proxy-model hyperparameter search. Optimal learning rates and output multipliers are found on 50M-parameter models by grid search over 365 configurations per optimizer-architecture pair and then transferred unchanged to the 2.7B-parameter models. The paper implements µP for Lion and Sophia and reports auxiliary width-scaling experiments showing their optimal learning rates stay consistent across widths under AdamW-style scaling rules. This machinery matters because it makes the comparison fair and affordable: without it, each 2.7B run would be too expensive to tune, and untuned hyperparameters would bias the optimizer ranking.

What would settle it

Re-run the 2.7B-parameter, 60B-token configurations for each optimizer with at least three to five random seeds and check whether AdamW's downstream lead over Lion and Sophia exceeds the spread across seeds on the same benchmarks; if the distributions overlap, the central claim fails.

Watch

Extended reading notes

Core claim

Under a fixed 60-billion-token budget for roughly 2.7-billion-parameter decoder-only models, the paper claims that no optimizer wins everywhere: AdamW yields the best downstream accuracy on ARC-Easy, ARC-Challenge, Hellaswag, and MMLU; Lion is fastest in GPU hours and converges fastest early; and Sophia reaches very low training and validation losses, most clearly when data is repeated over five epochs. The downstream lead of AdamW is the paper's headline result and is consistent across single-epoch (60B unique tokens) and multi-epoch (five passes over 12B tokens) regimes and across GPT-2-style and LLaMA-style architectures, with the exception that Lion scores slightly better on the LLaMA-style model. In the single-epoch GPT run AdamW actually had the lowest final losses, and Sophia's loss advantage in the multi-epoch runs did not translate into downstream superiority. The authors also report that Maximal Update Parametrization transfers hyperparameters from 50M-parameter proxy models to the 2.7B-parameter targets for all three optimizers under AdamW-style scaling, which is what lets each optimizer be tuned fairly rather than compared with one set of default hyperparameters.

Load-bearing premise

The comparison assumes that a single training run per configuration stands in for the optimizer's typical behavior; the authors state they could not run statistical tests or confidence intervals, so if seed-to-seed variation is comparable to the observed gaps, the ranking is not established.

Editorial extensions

If this is right

  • A compute-limited team optimizing for benchmark scores should choose AdamW over Lion or Sophia at this scale and budget.
  • Sophia's lower training and validation loss should not be taken as evidence of better downstream performance; early loss-based model selection can mislead.
  • Lion remains attractive when wall-clock speed and fast initial convergence matter more than final benchmark accuracy.
  • Hyperparameter tuning via Maximal Update Parametrization works for Lion and Sophia under AdamW-style scaling, so per-optimizer tuning is affordable even on a tight budget.
  • Repeating data epochs can slightly improve AdamW's downstream results, so multi-epoch training is not automatically harmful under a fixed token budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because each configuration was run once, the reported ranking is best read as provisional; a multi-seed replication could overturn it without contradicting the paper's methodology.
  • Editorial inference: the loss-versus-benchmark divergence suggests that hyperparameter searches guided only by validation loss may systematically favor Sophia-like optimizers even when another optimizer would generalize better.
  • Editorial inference: the AdamW advantage might partly reflect its maturity, since its defaults and implementation details have been polished by years of use, so a fair test of novel optimizers may need longer tuning budgets than this study could afford.
  • Editorial inference: if the pattern generalizes to larger models, then claims that a new optimizer 'trains faster' and claims that it 'performs better' should be evaluated separately, since the paper finds these can point to different optimizers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript compares AdamW, Lion, and Sophia for pre-training decoder-only language models of roughly 2.7B parameters on 60B tokens. Hyperparameters are tuned separately for each optimizer-architecture pair using Maximal Update Parametrization on 50M-parameter proxy models; the main runs cover two architectures (GPT-2-like and LLaMA-like) and single-epoch versus five-epoch data regimes. The authors report training and validation loss, area under the loss curve, GPU hours, and zero-shot ARC, HellaSwag, and MMLU accuracy. The headline findings are that Sophia achieves the lowest losses, Lion the fastest convergence and wall-clock time, and AdamW the best downstream accuracy; the paper also claims empirical evidence that μTransfer works for Lion and Sophia under AdamW-style scaling rules.

Significance. If the findings are robust, this would be a useful practical benchmark for optimizer selection under constrained compute, and the extension of μP to Lion and Sophia is a genuine contribution. The design is transparent and methodical: separate hyperparameter tuning per optimizer, two architectures, two data regimes, standard external benchmarks, and an explicit limitations section. However, the central downstream ranking and the quantitative speed comparisons rest on single training runs per configuration, and the μTransfer validation covers only the learning rate. These gaps currently prevent the strong conclusions from being fully supported.

major comments (4)
  1. [Section 4.2 and Section 6] The claim that AdamW 'clearly outperforms' both Lion and Sophia in downstream accuracy is not supported by the evidence as presented. Each configuration was trained once with fixed seeds (Table 4), and Section 6 explicitly states that the authors 'could not employ statistical inference tests and provide confidence intervals.' For 3B-parameter models, zero-shot scores on ARC, HellaSwag, and MMLU can vary by amounts comparable to the inter-optimizer gaps, so the observed ranking may be seed noise. The abstract's own phrasing ('approximately the same range') is more cautious than the Section 4.2 wording. Please either add repeated-seed results for the main comparison or temper the claim in Section 4.2 and the abstract.
  2. [Section 3.5 and Table 3] The μTransfer validation in Section 3.5 establishes only that the optimal learning rate transfers across widths for Lion and Sophia. The tuning procedure in Section 3.4 also selects the output multiplier and, for Sophia, ρ on 50M-parameter proxies, and no evidence is given that these hyperparameters transfer to the 2.7B models or that transfer quality is optimizer-independent. Since these tuned values feed directly into the main comparison, a systematic difference in transfer quality could bias the downstream ranking. Please validate transfer for the output multiplier and ρ, or demonstrate robustness to their values.
  3. [Section 4.1 and Figure 2] The multi-epoch GPU-hour comparison is confounded by the acknowledged filesystem slowdown that inflated Sophia's training times (Figure 2 note). The subsequent statement that Sophia has only 'slightly increased (~6%) computational overhead to AdamW' is therefore not supported by the multi-epoch runs, and the claim that Lion is fastest in terms of GPU hours is only established in the single-epoch GPT setting. Please report corrected timings or restrict the speed claims to the runs where timings are reliable.
  4. [Sections 3.4 and 3.5] The proxy models are tuned on 1B tokens, and hyperparameters are selected by minimizing final training loss at that horizon, but the main models train for 60B tokens. The μTransfer validation covers width scaling, not token-horizon scaling, so there is no evidence that the optimal learning rate or output multiplier at 1B tokens remains optimal at 60B tokens. If the optimum shifts differently across optimizers, the comparison would be biased. Please discuss this limitation or provide evidence on the stability of the selected hyperparameters over training length.
minor comments (5)
  1. [Section 2] In the paragraph discussing Liu et al., 'AdanW' should be 'AdamW'.
  2. [Algorithm 2] The update for m_t omits the β2 factor: as written, m_t <- m_{t-1} + (1-β2) g_t is not the standard Lion momentum update, and β2 does not appear in the assignment. Please correct the pseudocode.
  3. [Section 4.2] No numeric downstream scores or per-task tables are provided; Figures 3-5 are the only evidence. Reporting the averaged scores with a per-task breakdown would improve reproducibility and help support the 'clearly outperforming' claim.
  4. [Section 3.4] The grid search of '365 options per system' is not documented in an appendix; a table of the searched ranges and selected values would make the tuning reproducible.
  5. [Section 4.2] The sentence 'Our best-performing models surpasses publicly available Pythia checkpoints' should read 'surpass' (or change 'models' to 'model').

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: downstream benchmark scores are measured externally, and hyperparameter tuning on proxy models is a transfer method, not an input redefinition.

full rationale

The paper's derivation chain is empirical and self-contained. Hyperparameters (learning rate, output multiplier, and Sophia's rho) are tuned on 50M proxy models trained on a 1B-token subset, selecting the configuration that minimizes final training loss, and are then transferred unchanged to 2.7B target models. Downstream accuracy on ARC, HellaSwag, and MMLU is measured with an external evaluation harness, not derived from any parameter fitted to those benchmarks. No equation in the paper defines the downstream ranking in terms of the proxy tuning objective, and no fitted quantity is renamed as a prediction. The Section 6 limitation statement, in which the authors acknowledge that they could not repeat experiments with different random seeds and therefore could not employ statistical inference tests or confidence intervals, is a genuine validity concern: it weakens the strength of the claim that AdamW 'clearly outperforms' Lion and Sophia, since the observed gaps may reflect run-to-run noise. However, this is a statistical-power and reproducibility limitation, not circularity. The self-referential note about 'preliminary results of an ongoing study at our department' pointing toward low variances is also acknowledged; it concerns training loss and is not load-bearing for the main downstream comparison, nor does it make any benchmark score equivalent to an input. Thus no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 15 free parameters · 4 assumptions · 0 invented entities

All central findings rest on 15 tuned or hand-chosen hyperparameters (learning rates, output multipliers, Sophia rho, initialization sigma) and on four domain assumptions: the validity of muP transfer to 2.7B models, the appropriateness of the Chinchilla token budget, the validity of the chosen benchmarks as downstream measures, and the representativeness of single runs. No new entities are introduced.

free parameters (15)
  • Learning rate (AdamW-GPT) = 0.0128
    Tuned on 50M proxy models via 365-point grid; transferred to 2.7B target.
  • Learning rate (AdamW-LLaMA) = 0.025
    Tuned on 50M proxy models; transferred to 2.7B target.
  • Learning rate (Lion-GPT) = 0.00076
    Tuned on 50M proxy models; transferred to 2.7B target.
  • Learning rate (Lion-LLaMA) = 0.0012
    Tuned on 50M proxy models; transferred to 2.7B target.
  • Learning rate (Sophia-GPT) = 0.001
    Tuned on 50M proxy models; transferred to 2.7B target.
  • Learning rate (Sophia-LLaMA) = 0.002
    Tuned on 50M proxy models; transferred to 2.7B target.
  • Output multiplier (AdamW-GPT) = 1.5
    Tuned output multiplier, correlated with learning rate.
  • Output multiplier (AdamW-LLaMA) = 1.0
    Tuned output multiplier.
  • Output multiplier (Lion-GPT) = 2
    Tuned output multiplier.
  • Output multiplier (Lion-LLaMA) = 1.5
    Tuned output multiplier.
  • Output multiplier (Sophia-GPT) = 1.5
    Tuned output multiplier.
  • Output multiplier (Sophia-LLaMA) = 1.5
    Tuned output multiplier.
  • Rho (Sophia-GPT) = 0.3
    Tuned parameter controlling the influence of second-order information.
  • Rho (Sophia-LLaMA) = 1.0
    Tuned parameter controlling the influence of second-order information.
  • Initialization variance = 0.073
    Chosen based on prior experiments (Dey et al., 2023), not tuned in this study.
assumptions (4)
  • domain assumption muP transfer validity: optimal hyperparameters found on 50M proxy models remain optimal for 2.7B target models for all three optimizers.
    Central to the comparison methodology; the paper validates empirically for Lion and Sophia up to width 2048 (Section 3.5, Figure 6), but transfer to the 2.7B targets is assumed.
  • domain assumption Chinchilla compute-optimal scaling recommends 60B tokens for a 3B-parameter model.
    Used to fix the 60B-token budget (Section 1); if the data budget does not match the model size, the comparison may not generalize.
  • domain assumption The benchmarks (ARC, HellaSwag, MMLU) in zero-shot cloze form are valid measures of downstream LLM quality at this scale.
    Downstream tasks are the primary success metric; the paper uses the OLMES-style cloze MMLU due to small-model constraints (Section 3.7).
  • domain assumption Run-to-run variance from random seeds is small enough that single runs per configuration are representative.
    The paper states in Section 6 that no repeated seeds were used; the comparison assumes observed differences are not noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pre-Training LLMs on a budget: A comparison of three optimizers." pith.science (2026). https://pith.science/paper/ORN6RXF5

@misc{pith2026250708472,
  author       = {Pith},
  title        = {Pith review of: Pre-Training LLMs on a budget: A comparison of three optimizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ORN6RXF5}},
  note         = {Machine review of arXiv:2507.08472}
}
read the original abstract

Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standard AdamW, the simpler Lion, developed through an evolutionary search, and the second-order optimizer Sophia. For better generalization, we train with two different base architectures and use a single- and a multiple-epoch approach while keeping the number of tokens constant. Using the Maximal Update Parametrization and smaller proxy models, we tune relevant hyperparameters separately for each combination of base architecture and optimizer. We found that while the results from all three optimizers were in approximately the same range, Sophia exhibited the lowest training and validation loss, Lion was fastest in terms of training GPU hours but AdamW led to the best downstream evaluation results.

Figures

Figures reproduced from arXiv: 2507.08472 by the authors.

Figure 1
Figure 1. Training dynamics of optimizers on GPT models trained with 60B unique tokens. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Training dynamics of optimizers across GPT and LLaMA architectures trained [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Downstream performance of AdamW, Lion and Sophia with the GPT architecture [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Downstream task performance of the three optimizers with both architectures, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Leaderboard comparison of downstream performance (averaged across tasks). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: µTransfer validation for Lion (left) and Sophia (right). Evidence of hyperparameter transferability is indicated by alignment of the optimal learning rate, i.e., minima in train loss curves occurring at similar learning rate values. Note: Learning rate ranges are not d…
Figure 7
Figure 7. Figure 7: Visualization of our tuned hyperparameters (learning rate and output multiplier) [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Average gradient L2-norm (log scale) for GPT models trained with AdamW, Lion, [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

  2. From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference

    cs.CL 2026-07 conditional novelty 4.0 of 10

    A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.

Reference graph

Works this paper leans on

40 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    sign SGD with majority vote is communication efficient and fault tolerant

    Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. sign SGD with majority vote is communication efficient and fault tolerant. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=BJxhijAcY7

    work page 2019

  2. [2]

    Pythia: A suite for analyzing large language models across training and scaling, 2023

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373

    arXiv 2023

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    1901

  4. [4]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. Symbolic discovery of optimization algorithms. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 49205--49233. Curran...

    work page 2023

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018

    arXiv 2018

  6. [6]

    Btlm-3b-8k: 7b parameter performance in a 3b parameter model, 2023

    Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, and Joel Hestness. Btlm-3b-8k: 7b parameter performance in a 3b parameter model, 2023. URL https://arxiv.org/abs/2309.11568

    arXiv 2023

  7. [7]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework...

    arXiv 2024

  8. [8]

    Olmes: A standard for language model evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08446, 2024

    arXiv 2024

Show all 40 references
  1. [9]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  2. [10]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  3. [11]

    On the parameterization of second-order optimization effective towards the infinite width, 2024

    Satoki Ishikawa and Ryo Karakida. On the parameterization of second-order optimization effective towards the infinite width, 2024. URL https://arxiv.org/abs/2312.12226

    2024 arXiv

  4. [12]

    No train no gain: Revisiting efficient training algorithms for transformer-based language models

    Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Inf...

    2023

  5. [13]

    A method for stochastic optimization

    Diederik Kinga, Jimmy Ba Adam, et al. A method for stochastic optimization. In International conference on learning representations (ICLR), volume 5. San Diego, California;, 2015

    2015

  6. [14]

    Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks

    Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 13...

    2021

  7. [15]

    ROPE : Reading order equivariant positional encoding for graph-based document information extraction

    Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, and Tomas Pfister. ROPE : Reading order equivariant positional encoding for graph-based document information extraction. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds...

    2021

  8. [16]

    Lightning AI . Litgpt. https://github.com/Lightning-AI/litgpt, 2023

    2023

  9. [17]

    An empirical study of p learning rate transfer, 2025

    Lucas Lingle. An empirical study of p learning rate transfer, 2025. URL https://arxiv.org/abs/2404.05728

    2025 arXiv

  10. [18]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=3xHDeA8Noi

    2024

  11. [19]

    Fixing weight decay regularization in adam

    Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101

    2017 arXiv

  12. [20]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://o...

    2023

  13. [21]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  14. [22]

    A modified A dam algorithm for deep neural network optimization

    Mohamed Reyad, Amany M Sarhan, and Mohammad Arafa. A modified A dam algorithm for deep neural network optimization. Neural Computing and Applications, 35 0 (23): 0 17095--17112, 2023

    2023

  15. [23]

    An overview of gradient descent optimization algorithms, 2017

    Sebastian Ruder. An overview of gradient descent optimization algorithms, 2017. URL https://arxiv.org/abs/1609.04747

    2017 arXiv

  16. [24]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 4596--460...

    2018

  17. [25]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL ht...

    2023

  18. [26]

    Spike no more: Stabilizing the pre-training of large language models, 2025

    Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models, 2025. URL https://openreview.net/forum?id=G84F1h2IiD

    2025

  19. [27]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  20. [28]

    Evolution and role of optimizers in training deep learning models, 2024

    XiaoHao Wen and MengChu Zhou. Evolution and role of optimizers in training deep learning models, 2024

    2024

  21. [29]

    Ranger21: a synergistic deep learning optimizer, 2021

    Less Wright and Nestor Demeure. Ranger21: a synergistic deep learning optimizer, 2021. URL https://arxiv.org/abs/2106.13731

    2021 arXiv

  22. [30]

    Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models

    Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 0 (12): 0 9508--9520, 2024

    2024

  23. [31]

    Tuning large neural networks via zero-shot hyperparameter transfer

    Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Va...

    2021

  24. [32]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  25. [33]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. CoRR, abs/2406.16793, 2024. URL https://doi.org/10.48550/arXiv.2406.16793

    2024 doi arXiv

  26. [34]

    Improved adam optimizer for deep neural networks

    Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp.\ 1--2. IEEE, 2018

    2018

  27. [35]

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham M. Kakade. Deconstructing what makes a good optimizer for autoregressive language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=zfeso8ceqr

    2025

  28. [36]

    Adabelief optimizer: Adapting stepsizes by the belief in observed gradients

    Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances i...

    2020

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  30. [38]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  31. [39]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  32. [40]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.