Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tuned SGD with momentum matches Adam on language models when the batch is small, and the paper argues batch size rather than architecture or data explains the optimizer gap.

desk verdict Batch size is a real and informative axis of the Adam-SGD gap in LMs, but the headline parity claim is built on best-run selection and no seed variance, so treat the quantitative claim as provisional. read the letter →

arxiv 2506.12543 v1 pith:PYZQ7EYQ submitted 2025-06-14 cs.LG math.OC

Teodora Srećković , Jonas Geiping , Antonio Orvieto This is my paper
classification cs.LGmath.OC
keywords optimizergapAdamvsSGDbatchsizelanguagemodelingstochasticdifferentialequationsmomentumgradientclippingcritical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the perceived superiority of Adam over SGD in Transformer language modeling is not a fixed property of the optimizer, the architecture, or the text data: it is largely a batch-size effect. With momentum, a tuned learning rate, and gradient clipping, SGD can match Adam's final perplexity when the batch is small, and the paper reports runs where SGD beats Adam at 410M and 1B parameters. It then argues that existing explanations, such as heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, cannot account for why SGD suddenly works at small batches. The paper proposes an SDE-based mechanism in which adaptive and signed methods gain a drift that grows with the square root of batch size, while plain SGD's early progress depends only on the number of steps. If true, this reframes the optimizer gap as a question of gradient noise scale and critical batch size.

What carries the argument

The load-bearing object is the stochastic differential equation (SDE) approximation of SGD and of sign-based updates. For signSGD the paper cites a theorem giving the drift term as $-\operatorname{erf}\big(\sqrt{B/2}\,\Sigma^{-1/2}\nabla f(x)\big)$, which is approximately linear in $\sqrt{B}$ near zero and saturates beyond a critical batch size, while SGD's drift is just $-\nabla f(x)$ with noise $\sqrt{\eta\Sigma/B}\,dW_t$. This drift-versus-batch-size separation explains both why Adam-type methods improve with larger batches and why SGD's small-batch success is really an iteration-count effect. A secondary instrument is the heterogeneous quadratic testbed with block-diagonal Hessians, used to show the batch-size effect appears even without Transformer-specific structure, and grafting experiments that isolate the update direction as the failing component of large-batch SGD.

What would settle it

Run the small-batch comparison (e.g., batch size 64, 160M model on SlimPajama) with at least five seeds per configuration, tuning the learning rate grid separately at that batch size, and compare the median final validation perplexity of SGD with momentum against Adam; if the median SGD perplexity is clearly worse than Adam's, then the parity claim holds only for best-luck runs, not for typical training.

Watch

Extended reading notes

Core claim

The central claim is that SGD with momentum can be made competitive with Adam in Transformer language modeling, provided the batch size is small and both momentum and learning rate are properly tuned: at batch size 64 and a 1.3B-token budget, a 160M model reaches final validation perplexity close to Adam's (30.76 vs 28.77), and at 410M and 1B parameters SGD can actually outperform Adam when trained with very small batches for many steps. The paper further argues that the reason is not any of the previously proposed mechanisms, all of which predict a gap that should persist at small batch sizes; instead, the gap is governed by how each optimizer responds to gradient noise. For signed and adaptive updates, the drift in the SDE limit scales as an error function of the square root of the batch size times the preconditioned gradient, giving an acceleration with batch size up to a critical point, whereas SGD's drift is batch-size independent, so its progress is set by iteration count.

Load-bearing premise

The claim that SGD matches Adam rests on reporting each optimizer's best stable run rather than its typical run, with learning-rate grids tuned at one batch size and only roughly rescaled for others; under a stricter median-seed, per-batch-size protocol, the small-batch parity may vanish.

Editorial extensions

If this is right

  • In small-batch regimes, SGD with momentum is a memory-cheaper alternative to Adam for training language models, since it does not store second-order moment estimates.
  • The Adam-SGD gap is not intrinsic to Transformers or text data; batch size and gradient noise scale are first-order moderators of the gap.
  • Increasing batch size accelerates Adam-type methods through a $\sqrt{B}$ drift effect up to a critical batch size, while SGD gains nothing from larger batches in early training.
  • Prior explanations of the gap, including heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, are incomplete because they predict a gap that should remain even where SGD is observed to work well.
  • At scale, SGD can outperform Adam under small-batch, long-step training, suggesting that optimizer choice interacts with compute budget and memory constraints rather than being fixed by architecture alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension beyond the paper: if batch size controls the gap, then compute-optimal training recipes that rely on very large batches may systematically favor Adam, while low-resource regimes that use tiny batches are precisely where simpler optimizers could win.
  • Extension beyond the paper: the SDE analysis implies that other sign-based or compressed-gradient methods should inherit Adam's batch-size acceleration; a direct test would compare SignSGD and Adam across batch sizes and check whether their gap follows the predicted $\sqrt{B}$ drift scaling.
  • Extension beyond the paper: the observation that clipping occurs most often in normalization layers at large batch sizes suggests that per-coordinate scaling of a few outlier coordinates is the active ingredient, pointing to a possible middle-ground optimizer that adaptively clips only rare large coordinates instead of maintaining full Adam states.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper revisits the commonly observed performance gap between Adam and SGD in Transformer language modeling. Through controlled sweeps on nanoGPT-style models (160M up to 1B parameters) trained on SlimPajama and FineWeb, the authors report that SGD with momentum, gradient clipping, and carefully tuned hyperparameters can match or even outperform Adam when the batch size is small, while the gap reappears at larger batch sizes. They then test several existing explanations (heavy-tailed class imbalance, directional sharpness, Hessian heterogeneity), find them insufficient to explain the small-batch result, and propose an SDE-based account in which signed-gradient methods (and by extension Adam) benefit from larger batches through an erf-shaped drift term, whereas plain SGD does not. The central empirical claim is that batch size, not just optimizer choice, is a decisive factor in the Adam-SGD gap.

Significance. If the central claim is robust, the paper provides a valuable and somewhat counterintuitive result: at small batch sizes, SGD with momentum can be competitive with Adam in Transformer language modeling at the 410M and 1B parameter scale. This would reframe the optimizer gap as a batch-size-dependent phenomenon, with practical implications for memory-constrained training and for theories of adaptive methods. The paper is also useful as a systematic empirical probe of prior explanations, and it makes an explicit, falsifiable prediction: that SGD's performance degrades with batch size while Adam's improves, up to a critical batch size. The authors are transparent about the heuristic bridge from SignSGD to Adam in the theory, which is a strength in presentation, though it leaves the theoretical contribution incomplete.

major comments (4)
  1. [Section 2.2, Table 1, Figure 5] The evaluation protocol does not support the headline parity claim as stated. The text says: "Some settings become unstable at very large learning rates, where one run may succeed, even if the median run diverges. In those settings, we report runs at the largest stable learning rate as optimal." This is best-run selection under instability, and no seed variance is reported anywhere in Section 2 or for Figure 5. Table 1's SGD result at batch size 64 (PPL 30.76, lr=0.5) and the 410M/1B trajectories in Figure 5 could therefore be draws from a distribution whose median is much worse. Please rerun the key comparisons with at least 3-5 seeds, report median and interquartile ranges or full seed curves, and define the stability criterion explicitly (e.g., the fraction of runs that diverge). Without this, the central claim that SGD can match or outperform Adam in small-batch settings is not established.
  2. [Section 2.2 and Section 2.3] The learning-rate grids are not independently tuned per batch size, despite the abstract's claim of "comprehensively tuned" baseline runs. In Section 2.2, the grid is tuned at batch size 256 and then scaled to batch sizes 64 and 1024 with only three values per batch size. In Section 2.3, fixed grids [0.25, 0.5, 1.0] for SGD and [0.001, 0.002, 0.004] for Adam are used across batch sizes. The reported Adam-SGD gap may therefore be an artifact of insufficient Adam tuning at small batch sizes or insufficient SGD tuning at large batch sizes. Please perform per-batch-size sweeps over wider grids, or at least show a learning-rate response curve for each optimizer and batch size, and justify that the scaled grids cover each optimizer's optimal region.
  3. [Section 4.3, Theorem 1 and Figure 9] The theoretical claim that large batch sizes accelerate Adam is not directly supported by the stated theorem. Theorem 1 is a result for SignSGD, and the paper says the conclusion holds "in signSGD (and likely also in closely-related algorithms, like Adam)". The quadratic experiments in Figure 9 compare SGD and SignSGD+m, not Adam, so the bridge from the theorem to the actual Adam behavior is a heuristic analogy. This is acknowledged in the text, but it is load-bearing for the paper's proposed explanation of the empirical results. Please state this limitation more prominently, or provide a formal or at least direct empirical argument that the erf drift acceleration carries over to Adam's coordinate-wise normalization rather than only to sign-based updates.
  4. [Section 2.3, Figure 5, Figure 14] The scaling experiments that support "SGD can outperform Adam even at a 410M and 1B scale" are presented using training perplexity trajectories (Figure 5), not held-out validation perplexity. The learning-rate sweeps in Figure 14 do report validation perplexity, but those appear to be single-run evaluations. Since the headline claim is about final model quality, please report final validation perplexities for the chosen runs, together with seed information, and clarify whether the outperformance holds on held-out data or only on the training set.
minor comments (4)
  1. [Appendix D] There is a typo: "Leraning rates" should be "Learning rates".
  2. [Section 2.3] The phrase "its critical batch size is close to 1" is used without a definition. Please define what is meant by "critical batch size" in this context (e.g., the threshold beyond which larger batches no longer reduce the number of steps to a given loss).
  3. [Appendix A.2] The sentence "As a sequence length of 160k, given our lack of experience with extremely low batch sizes (shorter sequence length)" is garbled; please rewrite to state the actual configuration used for the Fineweb 160M experiments.
  4. [Figure 4] The x-axis label "10 160804020" appears malformed; it should read something like "Steps (\times 10^3)" with tick labels 10, 20, 40, 80, 160.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical SGD-vs-Adam comparison is self-contained, and the cited SDE theorem for SignSGD is external support that does not encode the paper's target claim.

full rationale

The paper's central claim—that SGD with momentum can match Adam in small-batch Transformer language modeling—is established by direct hyperparameter sweeps (Section 2.2, Figure 1, Table 1) and scaling runs (Section 2.3, Figure 5), not by fitting a parameter and then presenting the same fitted quantity as a prediction. The learning-rate grids at batch sizes 64 and 1024 are scaled from the grid tuned at batch size 256, but the reported perplexities are measurements of separate runs, and Section 2.2's note that unstable settings are reported at the largest stable learning rate is a selection-robustness concern, not a circularity concern. The main theoretical input is Theorem 1, cited from Compagnoni et al. (2025b), which shares co-author Orvieto. This is a self-citation, but it is not load-bearing in a circular way: the theorem is stated with explicit assumptions (i.i.d. Gaussian noise, diagonal covariance) and is about SignSGD; it does not assume or include the paper's empirical result about Adam versus SGD. The paper only heuristically extends the theorem to Adam ('likely also in closely-related algorithms, like Adam'), and that extension is flagged as an analogy rather than derived from the target claim. No equation in the paper reduces to its own input by construction, and no fitted parameter is renamed as a prediction. A stricter evaluation protocol could weaken the empirical conclusion, but that is a correctness/robustness issue, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters; its analysis is based on existing optimization theory and empirical hyperparameter sweeps.

assumptions (3)
  • domain assumption The SDE approximation of SGD (Eq. 2) with Gaussian noise is a valid proxy for the discrete training dynamics.
    Standard in stochastic optimization theory, but relies on small learning rates and smoothness; used to derive the SignSGD drift in Theorem 1.
  • ad hoc to paper The theorem from Compagnoni et al. (2025b) for SignSGD (Theorem 1) is assumed to carry over to Adam.
    The paper states 'likely also in closely-related algorithms, like Adam' without proof; this is the bridge from SignSGD to Adam.
  • domain assumption The heterogeneous quadratic model (Appendix D) is a representative proxy for Transformer optimization.
    Borrowed from Zhang et al. (2024); the paper uses it to draw conclusions about real Transformers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling." pith.science (2026). https://pith.science/paper/PYZQ7EYQ

@misc{pith2026250612543,
  author       = {Pith},
  title        = {Pith review of: Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYZQ7EYQ}},
  note         = {Machine review of arXiv:2506.12543}
}
read the original abstract

Adam is known to perform significantly better than Stochastic Gradient Descent (SGD) in language models, a phenomenon for which a number of explanations have been proposed. In this work, we revisit this "optimizer gap" through a series of comprehensively tuned baseline training runs for language modeling with Transformers. We exhaustively study how momentum, gradient clipping, and batch size affect the gap between SGD and Adam. Our empirical findings show that SGD with momentum can actually perform similarly to Adam in small-batch settings, if tuned correctly. We revisit existing explanations for Adam's advantage, including heavy-tailed class imbalance, directional sharpness, and Hessian heterogeneity, which struggle to directly explain this phenomenon. Towards bridging this gap in our understanding, by analyzing our Transformer training runs and simple quadratic settings inspired by the literature, we provide new insights, driven by stochastic differential equation models, into the role of batch size on the training dynamics.

Figures

Figures reproduced from arXiv: 2506.12543 by the authors.

Figure 1
Figure 1. Learning rate and momentum sweep for SGD and Adam across batch sizes under a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Gradient norm after clipping (threshold 1.0) shows that clipping is more frequent in large-batch training. The setup for these runs is the same as [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. SGD (green) and Adam (purple) performance across batch sizes. Left: fixed token budget [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Perplexity during training for SGD (green) and Adam (purple) across different training [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: SGD can outperform Adam even at 410M and 1B scales in small-batch regimes. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Batch token distribution for batch sizes 64 and 1024. Lighter colors show less frequent tokens. Statistics at lower batch sizes are noisier but of similar magnitude. Prior work by Kunstner et al. [2023] attributes Adam’s advantage over SGD to heavy-tailed class imbalan…
Figure 7
Figure 7. Figure 7: (left) Perplexity during training for Adam and SGD in small- and large-batch settings, computed per frequency group. SGD shows a larger gap across groups in the large-batch setting, while the opposite holds for Adam. (right) Comparison of the Adam–SGD gap across freque…
Figure 8
Figure 8. Figure 8: Gradient correlation, directional sharpness, their sum and second-order loss approximation [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The gap between Adam and SGD relative to batch size also appears when studying [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Grafting in large-batch training: using Adam’s direction results in performance closer to Adam, while SGD direction leads to results closer to SGD. 10 1 10 2 10 3 Steps 10 2 10 3 10 4 Train PPL clip top 10% clip top 20% clip top 5% sgd adam [PITH_FULL_IMAGE:figures/f…
Figure 12
Figure 12. Figure 12: Fraction of clipped momentum coor￾dinates per layer during training, using p = 10% adaptive clipping. Only a subset of blocks is shown for clarity, as similar patterns are observed across all blocks. Clipping is present across all parame￾ters, but most pronounced in n…
Figure 13
Figure 13. Figure 13: ). 0 1000 2000 iteration 10 2 10 1 loss SGD, noise =0.1 SGD, noise =0.3 SGD, noise =1 SignSGD, noise =0.1 SignSGD, noise =0.3 SignSGD, noise =1 [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Learning rate sweep for 410M and 1B models. Trajectories for the optimal learning rate [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Perplexity during training for SGD (green) and Adam (purple) across different training [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Fineweb dataset, sequence length 2048, 12 layers Transformer. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: SlimPajama dataset, sequence length 2048, 24 layers Transformer. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: (left) Heterogeneous and (right) Homogeneous Hessian considered in [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Complement to [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling

    cs.LG 2025-10 conditional novelty 6.0 of 10

    When a cosine schedule would halve the learning rate, Seesaw cuts it by √2 and doubles the batch, matching loss curves with ~36% fewer serial steps.

Reference graph

Works this paper leans on

30 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwal, R

    N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling adaptive gradient methods from learning rates.arXiv preprint arXiv:2002.11803,

    arXiv 2002

  2. [4]

    As before, the gap decreases the longer we train, and SGD can eventually outperform Adam

    Solid lines show the rolling mean of PPL values; lighter lines show the raw values. As before, the gap decreases the longer we train, and SGD can eventually outperform Adam. 18 4 8 16 32 64 128 256 512 BS 25 30 35 40 45 50Final valid PPL SGD | 160k Adam | 160k SGD | 40k Adam | 40k SGD | 10k Adam | 10k Figure 16: Fineweb dataset, sequence length 2048, 12 l...

    work page

  3. [5]

    E. M. Compagnoni, R. Islamov, A. Orvieto, and E. Gorbunov. On the interaction of noise, compres- sion role, and adaptivity under (l_0, l_1)-smoothness: An sde-based approach.arXiv preprint arXiv:2506.00181, 2025a. E. M. Compagnoni, T. Liu, R. Islamov, F. N. Proske, A. Orvieto, and A. Lucchi. Adaptive methods through the lens of SDEs: Theoretical insights ...

    work page Pith review arXiv

  4. [9]

    org/abs/2405.18392

    URL http://arxiv. org/abs/2405.18392. arXiv:2405.18392 [cs]. K. Jiang, D. Malik, and Y . Li. How Does Adaptive Optimization Impact Local Neural Network Geometry?, Nov

    arXiv

  5. [10]

    How Does Adaptive Optimization Impact Local Neural Network Geometry?

    URLhttp://arxiv.org/abs/2211.02254. arXiv:2211.02254 [cs]. K. Jordan, Y . Jin, V . Boza, Y . Jiacheng, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks,

    work page Pith review arXiv

  6. [11]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    arXiv

  7. [13]

    arXiv:2402.19449 [cs, math, stat]

    URL http: //arxiv.org/abs/2402.19449. arXiv:2402.19449 [cs, math, stat]. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  8. [14]

    J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y . Du, Y . Qin, W. Xu, E. Lu, J. Yan, et al. Muon is scalable for LLM training.arXiv preprint arXiv:2502.16982,

    arXiv

Show all 30 references
  1. [16]

    arXiv:2205.10287 [cs]

    URLhttp://arxiv.org/abs/2205.10287. arXiv:2205.10287 [cs]. G. Mil’shtein. Weak approximation of solutions of systems of stochastic differential equations. Theory of Probability & Its Applications, 30(4):750–766,

    arXiv

  2. [18]

    Orvieto and R

    A. Orvieto and R. Gower. In search of adam’s secret sauce.arXiv preprint arXiv:2505.21829,

  3. [19]

    arXiv:2306.00204 [cs]

    URLhttp://arxiv.org/abs/2306.00204. arXiv:2306.00204 [cs]. G. Penedo, H. Kydlíˇcek, L. B. allal, A. Lozhkov, M. Mitchell, C. Raffel, L. V . Werra, and T. Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. InThe Thirty-eight Conference on Neural In...

    arXiv

  4. [22]

    arXiv:2502.00213 [cs]

    URLhttp://arxiv.org/abs/2502.00213. arXiv:2502.00213 [cs]. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

  5. [23]

    S. Xie, M. A. Mohamadi, and Z. Li. Adam exploits ℓ∞-geometry of loss landscape via coordinate- wise adaptivity.arXiv preprint arXiv:2410.08198,

    arXiv

  6. [24]

    arXiv:2410.21676 [cs]

    URL http://arxiv.org/abs/2410.21676. arXiv:2410.21676 [cs]. J. Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity, Feb. 2020a. arXiv:1905.11881 [cs, math]. J. Zhang, S. P. Karimireddy, A. Veit, S. Kim,...

    1905 arXiv

  7. [25]

    arXiv:2402.16788 [cs]

    URLhttp://arxiv.org/abs/2402.16788. arXiv:2402.16788 [cs]. R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. Kakade. Deconstructing What Makes a Good Optimizer for Language Models, July

    arXiv

  8. [26]

    arXiv:2407.07972 [cs]

    URL http://arxiv.org/abs/2407.07972. arXiv:2407.07972 [cs]. 15 Appendix A Further Experiments and Experimental Details For pre-training Transformers on Causal Language Modeling, we use a setup that builds upon the nanoGPT [Karpathy, 2022] implementation, augmenting it with Rot...

    2022 arXiv

  9. [27]

    Model settings (410 M).We use the same setting as [Biderman et al., 2023], configuration can be found here: https://github.com/EleutherAI/pythia/blob/main/models/410M/ pythia-410m-deduped.yml •Layers:24 Transformer layers •Attention heads:16 •Hidden size:1024 • Other settings ...

    2023

  10. [30]

    [2024], and uses the codebase of Orvieto and Gower [2025]

    Our setup is inspired from the results and discussions in Zhang et al. [2024], and uses the codebase of Orvieto and Gower [2025]. We consider the loss L(w) = 1 2 w⊤Hw where we construct the Homogeneous and Heterogeneous Hessians using the following procedure: • We fix the eige...

    2024

  11. [1024]

    Our learning rate grid here is the same as for SlimPajama (previous point). As a sequence length of 160k, given our lack of experience with extremely low batch sizes (shorter sequence length), we operate on a slightly larger grid:[0.0001,0.0003,0.001,0.003]for Adam and[0.03,0....

    2023

  12. [1986]

    T. Q. Nguyen and J. Salazar. Transformers without tears: Improving the normalization of self- attention.arXiv preprint arXiv:1910.05895,

    1910 arXiv

  13. [2014]

    Kumar, R

    13 A. Kumar, R. Shen, S. Bubeck, and S. Gunasekar. How to fine-tune vision models with sgd.arXiv preprint arXiv:2211.09359,

    arXiv

  14. [2017]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  15. [2018]

    arXiv:1802.04434 [cs, math]

    URL http://arxiv.org/abs/1802.04434. arXiv:1802.04434 [cs, math]. S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and sc...

    arXiv

  16. [2019]

    I. Shah, A. M. Polloreno, K. Stratos, P. Monk, A. Chaluvaraju, A. Hojel, A. Ma, A. Thomas, A. Tan- wer, D. J. Shah, et al. Practical efficiency of muon for pretraining.arXiv preprint arXiv:2505.02222,

    arXiv

  17. [2020]

    Goyal, P

    P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677,

    arXiv

  18. [2021]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983,

    arXiv

  19. [2022]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    2010 arXiv

  20. [2023]

    Black, S

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745,

    arXiv

  21. [2024]

    URL http://arxiv.org/abs/2310. 01082. arXiv:2310.01082 [cs, math]. J. Bernstein, Y .-X. Wang, K. Azizzadenesheli, and A. Anandkumar. signSGD: Compressed Opti- misation for Non-Convex Problems, Aug

    arXiv

  22. [2025]

    N. Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

    2002 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.