Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes BEMA, a bias-corrected exponential moving average that removes the lag introduced by standard EMA while keeping its variance reduction.

desk verdict Plausible and useful-sounding idea, but the submitted text is unreadable and the abstract's 'bias' language may be equivocating between initialization bias and iterate lag. read the letter →

arxiv 2508.00180 v1 pith:EVVFELZD submitted 2025-07-31 cs.LG cs.AIstat.ML

Adam Block , Cyril Zhang This is my paper
classification cs.LGcs.AIstat.ML
keywords BEMAexponentialmovingaveragebiascorrectionweightaveraginglanguagemodelfine-tuningsmall-batchstochasticityvariancereductioniterate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language-model fine-tuning with small batches is noisy, and exponential moving average (EMA) of weights is a common stabilizer. Standard EMA smooths the noise but lags behind the current solution because it keeps a stale mix of old iterates in the average. The paper proposes BEMA, a bias-corrected version of EMA that removes that stale bias while retaining variance reduction. It proves, in a simple theoretical model with noisy gradient updates, that BEMA converges faster than both standard EMA and vanilla training, and reports that BEMA improves convergence rate and final performance on standard language-model benchmarks. If the paper is right, BEMA is a low-cost drop-in replacement for EMA in LLM fine-tuning pipelines.

What carries the argument

The central object is the bias-corrected exponential moving average (BEMA), a time-dependent modification of the standard EMA recursion that removes the accumulated contribution of old iterates so that the average stays aligned with the current update direction rather than lagging behind it. This correction is what converts EMA's variance reduction into a convergence-rate advantage: it keeps the smoothing effect while preventing the average from being anchored to the initialization. The theoretical model supplies the noisy-gradient setting in which the acceleration over EMA and vanilla training is proven.

What would settle it

A controlled comparison where learning rate, batch size, and EMA decay are tuned separately and equally for BEMA, standard EMA, and vanilla training: if BEMA's final benchmark score is not above both baselines by more than run-to-run noise, the claimed bias-elimination advantage is not detectable.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bias term in EMA, not the averaging itself, is what makes EMA fall behind vanilla training; remove that bias and averaging gives the stabilization of EMA at no cost in convergence speed. The paper introduces BEMA, which modifies the EMA recursion with a correction that neutralizes the influence of early iterates, and proves that in a noisy-gradient model BEMA accelerates optimization relative to both standard EMA and plain updates. Empirically, the paper reports that BEMA yields faster convergence and better final performance than EMA and vanilla training across several standard language-model benchmarks.

Load-bearing premise

The proof of acceleration rests on a simple theoretical model of noisy gradient updates, and the claimed dominance over EMA and vanilla training transfers to real language-model fine-tuning only if that model's loss geometry and noise structure are representative of what happens with small batches.

Editorial extensions

If this is right

  • BEMA can replace EMA in existing fine-tuning pipelines with a single change to the averaging update, preserving the smoothing that practitioners rely on.
  • In the paper's theoretical model, BEMA's convergence rate strictly improves on both standard EMA and vanilla training, which means averaging no longer trades away speed for stability.
  • On standard language-model benchmarks, BEMA attains lower final loss and better downstream scores than either baseline, so bias correction is not just a theoretical nicety.
  • Because BEMA removes initialization bias, it should be most valuable in short fine-tuning runs where an EMA has little time for the stale initial iterate to wash out.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the same bias-correction principle could be applied to other averaging schemes, such as stochastic weight averaging or the momentum estimates in Adam, with similar gains whenever stale iterates dominate early in training.
  • I infer that if BEMA's advantage is driven by bias, the gap should widen with smaller batch sizes and shorter runs; this is directly testable by sweeping batch size in the reported benchmarks.
  • I infer that the theoretical model's assumptions, if they include strong convexity or quadratic loss, may not cover the non-convex loss surfaces of real language models; in those settings, bias is not necessarily the only source of lag, so BEMA's edge could shrink.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a Bias-Corrected Exponential Moving Average (BEMA) for language model fine-tuning, claiming that it removes the lag of standard EMA while retaining variance reduction. The abstract states that a simple theoretical model demonstrates provable acceleration of BEMA over both EMA and vanilla training, and that experiments on standard LM benchmarks show significantly improved convergence rates and final performance. However, the supplied full text is corrupted mojibake and cannot be read, so the algorithm definition, theoretical derivations, and experimental evidence presented in the paper are not auditable.

Significance. If the claims are correct, BEMA would be a simple, theoretically motivated modification of EMA that dominates both EMA and vanilla training in convergence rate and final performance, with direct practical value for stable LLM fine-tuning. The claimed provable acceleration, together with an extensive benchmark suite, would make this a useful contribution. However, the current submission provides no verifiable model, no proof, no experimental tables, and no numbers, so the significance cannot be assessed beyond the abstract's assertions.

major comments (3)
  1. [Full text (corrupted)] The body of the manuscript is unreadable mojibake and carries a watermark referencing arXiv:2508.00184v1 [math.GT] rather than the claimed paper. As a result, the definition of BEMA, the theoretical model, the proof of acceleration, and the experimental results cannot be checked. This is a load-bearing deficiency: the abstract's claims of 'provable acceleration' and 'significantly improved convergence rates' are unsupported by any auditable derivation or experimental reporting.
  2. [Abstract] The paper asserts that BEMA 'eliminates bias from old iterates,' but the abstract does not specify the bias-correction mechanism. Standard EMA debiasing, dividing by (1 - beta^t), corrects only initialization bias, not the smoothing lag of an exponentially weighted average when the target optimum drifts. If BEMA relies on this standard identity, the proof cannot establish the central claim as stated; if BEMA uses a different construction, the update rule must be defined and analyzed. The current text does not disambiguate these possibilities.
  3. [Abstract] The experimental claim of 'significantly improved convergence rates and final performance' is made without reporting any quantitative results, model sizes, hyperparameters, random seeds, or error bars. The abstract does not state whether the EMA and vanilla baselines were tuned with equal effort. The claimed dominance is therefore not falsifiable from the submitted text, and the empirical evidence cannot be evaluated.
minor comments (3)
  1. [Full text] The submission should be re-encoded; the corrupted text prevents any meaningful reading of the methods or results.
  2. [Abstract] The abstract should either define the BEMA update rule explicitly or refer to a numbered equation in the text, so that 'bias correction' is unambiguous.
  3. [Experimental section (unreadable)] Once the text is readable, the experimental section should include standard deviations or confidence intervals across runs, and state the hyperparameter tuning budgets for all baselines.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified; the readable abstract states a definitional augmentation plus an asserted theorem, and no reduction to inputs can be exhibited.

full rationale

The only readable portion of the manuscript is the abstract; the supplied full text is mojibake and carries a watermark for a different arXiv ID, so the equations defining BEMA, the theoretical model, and the experimental protocols cannot be inspected. On the face of the abstract, the central claim is that BEMA, an augmentation of EMA with bias correction, provably accelerates over EMA and vanilla training in a stated model. The claimed variance-reduction property is a theorem about the algorithm under a model, not a restatement of the algorithm's definition, and no fitted parameter, self-citation, or uniqueness theorem is invoked in the readable text. The abstract's wording that 'bias from old iterates' causes lag and that BEMA 'eliminates bias' could reflect an ambiguity about which bias is meant, but that is a correctness and interpretation concern rather than a demonstrated circular step: I cannot quote any equation or argument showing that the acceleration result is equivalent by construction to the input assumptions. Under the hard rules requiring quoted reductions, no circular step can be flagged, so the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

All entries are read from the abstract because the supplied full text is corrupted mojibake. No free parameter values, proof conditions, or experimental configurations could be verified from the body.

free parameters (2)
  • EMA decay beta (and any schedule) = not reported in abstract
    The smoothing strength of BEMA is set by this hyperparameter. The abstract does not state its value, whether it is scheduled, or whether it was tuned per method, so the claimed gains could depend on its choice.
  • Learning rate and schedule per baseline = not reported in abstract
    The claimed dominance over vanilla training and EMA is meaningful only under comparable tuning. The abstract gives no details on how the baselines were tuned.
assumptions (2)
  • domain assumption A simple stochastic optimization model with smoothness and bounded noise, implicit in the 'simple theoretical model' of the abstract, is representative of LLM fine-tuning dynamics.
    The abstract's provable acceleration is stated for this model, and its transfer to real LLM fine-tuning is asserted rather than demonstrated.
  • ad hoc to paper The standard EMA debiasing identity (dividing by 1 - beta^t) removes the full lag under the actual training process.
    Exact debiasing requires a known constant beta and iteration index. The abstract does not address scheduled decays or early stopping, so 'eliminating bias' is a stronger statement than the abstract verifies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes." pith.science (2026). https://pith.science/paper/EVVFELZD

@misc{pith2026250800180,
  author       = {Pith},
  title        = {Pith review of: EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVVFELZD}},
  note         = {Machine review of arXiv:2508.00180}
}
read the original abstract

Stochasticity in language model fine-tuning, often caused by the small batch sizes typically used in this regime, can destabilize training by introducing large oscillations in generation quality. A popular approach to mitigating this instability is to take an Exponential moving average (EMA) of weights throughout training. While EMA reduces stochasticity, thereby smoothing training, the introduction of bias from old iterates often creates a lag in optimization relative to vanilla training. In this work, we propose the Bias-Corrected Exponential Moving Average (BEMA), a simple and practical augmentation of EMA that retains variance-reduction benefits while eliminating bias. BEMA is motivated by a simple theoretical model wherein we demonstrate provable acceleration of BEMA over both a standard EMA and vanilla training. Through an extensive suite of experiments on Language Models, we show that BEMA leads to significantly improved convergence rates and final performance over both EMA and vanilla training in a variety of standard LM benchmarks, making BEMA a practical and theoretically motivated intervention for more stable and efficient fine-tuning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    PACE is an AdamW wrapper derived from optimal control that improves the limiting error of the returned exponential-moving-average model in both theory and LM experiments.

  2. Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    PACE is a clipped per-coordinate controller added to AdamW that improves the limiting error of the returned iterate average in both quadratic analysis and LM experiments.

Reference graph

Works this paper leans on

83 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    High-dimensional limit theorems for sgd: Effective dynamics and critical scaling

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling. Advances in neural information processing systems, 35: 0 25349--25362, 2022

    2022

  3. [3]

    Provable guarantees for generative behavior cloning: Bridging low-level stability and high-level behavior

    Adam Block, Ali Jadbabaie, Daniel Pfrommer, Max Simchowitz, and Russ Tedrake. Provable guarantees for generative behavior cloning: Bridging low-level stability and high-level behavior. Advances in Neural Information Processing Systems, 36: 0 48534--48547, 2023

    2023

  4. [4]

    Butterfly effects of sgd noise: Error amplification in behavior cloning and autoregression

    Adam Block, Dylan J Foster, Akshay Krishnamurthy, Max Simchowitz, and Cyril Zhang. Butterfly effects of sgd noise: Error amplification in behavior cloning and autoregression. In The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [5]

    Approximation methods which converge with probability one

    Julius R Blum. Approximation methods which converge with probability one. The Annals of Mathematical Statistics, pages 382--386, 1954

    work page 1954

  6. [6]

    How to scale your ema

    Dan Busbridge, Jason Ramapuram, Pierre Ablin, Tatiana Likhomanenko, Eeshan Gunesh Dhekane, Xavier Suau Cuadros, and Russell Webb. How to scale your ema. Advances in Neural Information Processing Systems, 36: 0 73122--73174, 2023

    work page 2023

  7. [7]

    Legal-bert: The muppets straight out of law school

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559, 2020

    arXiv 2010

  8. [8]

    Incorrect baseline evaluations call into question recent llm-rl claims, 2025

    Nikhil Chandak, Shashwat Goel, and Ameya Prabhu. Incorrect baseline evaluations call into question recent llm-rl claims, 2025. Notion Blog

    work page 2025

Show all 83 references
  1. [9]

    Learning to generate better than your llm

    Jonathan D Chang, Kiante Brantley, Rajkumar Ramamurthy, Dipendra Misra, and Wen Sun. Learning to generate better than your llm. arXiv preprint arXiv:2306.11816, 2023

    2023 arXiv

  2. [10]

    Bidirectional looking with a novel double exponential moving average to adaptive and non-adaptive momentum optimizers

    Yineng Chen, Zuchao Li, Lefei Zhang, Bo Du, and Hai Zhao. Bidirectional looking with a novel double exponential moving average to adaptive and non-adaptive momentum optimizers. In International Conference on Machine Learning, pages 4764--4803. PMLR, 2023

    2023

  3. [11]

    Double/debiased machine learning for treatment and structural parameters, 2018

    Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018

    2018

  4. [12]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  5. [13]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  6. [14]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [15]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021

    2021 arXiv

  8. [16]

    Saga: A fast incremental gradient method with support for non-strongly convex composite objectives

    Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014

    2014

  9. [17]

    Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions

    Alexandre D \'e fossez and Francis Bach. Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions. In Artificial Intelligence and Statistics, pages 205--213. PMLR, 2015

    2015

  10. [18]

    Harder, better, faster, stronger convergence rates for least-squares regression

    Aymeric Dieuleveut, Nicolas Flammarion, and Francis Bach. Harder, better, faster, stronger convergence rates for least-squares regression. Journal of Machine Learning Research, 18 0 (101): 0 1--51, 2017

    2017

  11. [19]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

  12. [20]

    Is behavior cloning all you need? understanding horizon in imitation learning

    Dylan J Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon in imitation learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  13. [21]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  15. [23]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842--1850. PMLR, 2018

    2018

  16. [24]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  17. [25]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  18. [26]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl \'e ment Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  19. [27]

    Accelerating stochastic gradient descent using predictive variance reduction

    Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013

    2013

  20. [28]

    Stop wasting my time! saving days of imagenet and bert training with latest weight averaging

    Jean Kaddour. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXiv:2209.14981, 2022

    2022 arXiv

  21. [29]

    No train no gain: Revisiting efficient training algorithms for transformer-based language models

    Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. Advances in Neural Information Processing Systems, 36: 0 25793--25818, 2023

    2023

  22. [30]

    Gemma 3 technical report

    Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \'e , Morgane Rivi \`e re, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  23. [31]

    Analyzing and improving the training dynamics of diffusion models

    Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. ArXiv, abs/2312.02696, 2023. URL https://api.semanticscholar.org/CorpusID:265659032

    2023 arXiv

  24. [32]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  25. [33]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1, NIPS'12, page 1097–1105, Red Hook, NY, USA, 2012. ...

    2012

  26. [34]

    Statistical inference for ergodic diffusion processes

    Yury A Kutoyants. Statistical inference for ergodic diffusion processes. Springer Science & Business Media, 2013

    2013

  27. [35]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  28. [36]

    Tulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  29. [37]

    Brownian motion, martingales, and stochastic calculus

    Jean-Fran c ois Le Gall. Brownian motion, martingales, and stochastic calculus. Springer, 2016

    2016

  30. [38]

    Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks

    Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. ArXiv, abs/2406.02596, 2024. URL https://api.semanticscholar.org/CorpusID:270258586

    2024 arXiv

  31. [39]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36 0 (4): 0 1234--1240, 2020

    2020

  32. [40]

    Theory of point estimation

    Erich L Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006

    2006

  33. [41]

    Stochastic modified equations and adaptive stochastic gradient algorithms

    Qianxiao Li, Cheng Tai, et al. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pages 2101--2110. PMLR, 2017

    2017

  34. [42]

    Siyuan Li, Zicheng Liu, Juanxi Tian, Ge Wang, Zedong Wang, Weiyang Jin, Di Wu, Cheng Tan, Tao Lin, Yang Liu, Baigui Sun, and Stan Z. Li. Switch ema: A free lunch for better flatness and sharpness. ArXiv, abs/2402.09240, 2024. URL https://api.semanticscholar.org/CorpusID:267657558

    2024 arXiv

  35. [43]

    Statistics of random processes: I

    Robert S Liptser and Albert N Shiryaev. Statistics of random processes: I. General theory, volume 5. Springer Science & Business Media, 2013 a

    2013

  36. [44]

    Statistics of random processes II: Applications, volume 6

    Robert S Liptser and Albert N Shiryaev. Statistics of random processes II: Applications, volume 6. Springer Science & Business Media, 2013 b

    2013

  37. [45]

    Improving large language model fine-tuning for solving math problems

    Yixin Liu, Avi Singh, C Daniel Freeman, John D Co-Reyes, and Peter J Liu. Improving large language model fine-tuning for solving math problems. arXiv preprint arXiv:2310.10047, 2023

    2023 arXiv

  38. [46]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  39. [47]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2022

    2022

  40. [48]

    On the sdes and scaling rules for adaptive gradient algorithms

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 0 7697--7711, 2022

    2022

  41. [49]

    A kernel-based view of language model fine-tuning

    Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610--23641. PMLR, 2023

    2023

  42. [50]

    Continuous-time limit of stochastic gradient descent revisited

    Stephan Mandt, Matthew D Hoffman, David M Blei, et al. Continuous-time limit of stochastic gradient descent revisited. NIPS-2015, 2015

    2015

  43. [51]

    Revisiting small batch training for deep neural networks

    Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. arXiv preprint arXiv:1804.07612, 2018

    2018 arXiv

  44. [52]

    Scaling data-constrained language models

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36: 0 50358--50376, 2023

    2023

  45. [53]

    Smoothing data with faster moving averages

    Patrick G Mulloy. Smoothing data with faster moving averages. Stocks & Commodities, 12 0 (1): 0 11--19, 1994

    1994

  46. [54]

    Introductory lectures on convex optimization: A basic course, volume 87

    Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013

    2013

  47. [55]

    The ademamix optimizer: Better, faster, older

    Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137, 2024

    2024 arXiv

  48. [56]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30 0 (4): 0 838--855, 1992

    1992

  49. [57]

    Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55--69

    Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade, pages 55--69. Springer, 2002

    2002

  50. [58]

    Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis

    Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory, pages 1674--1703. PMLR, 2017

    2017

  51. [59]

    Breaking the data barrier: a review of deep learning techniques for democratizing ai with small datasets

    Ishfaq Hussain Rather, Sushil Kumar, and Amir H Gandomi. Breaking the data barrier: a review of deep learning techniques for democratizing ai with small datasets. Artificial Intelligence Review, 57 0 (9): 0 226, 2024

    2024

  52. [60]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400--407, 1951

    1951

  53. [61]

    Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and imitation learning under misspecification

    Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, and Dylan J Foster. Computational-statistical tradeoffs at the next-token prediction barrier: Autoregressive and imitation learning under misspecification. arXiv preprint arXiv:2502.12465, 2025

    2025 arXiv

  54. [62]

    Efficient reductions for imitation learning

    St \'e phane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661--668. JMLR Workshop and Conference Proceedings, 2010

    2010

  55. [63]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St \'e phane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627--635. JMLR Workshop an...

    2011

  56. [64]

    Efficient estimations from a slowly convergent robbins-monro process

    David Ruppert. Efficient estimations from a slowly convergent robbins-monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988

    1988

  57. [65]

    Training trajectories, mini-batch losses and the curious role of the learning rate

    Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Nolan Miller. Training trajectories, mini-batch losses and the curious role of the learning rate. arXiv preprint arXiv:2301.02312, 2023

    2023 arXiv

  58. [66]

    Minimizing finite sums with the stochastic average gradient

    Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162: 0 83--112, 2017

    2017

  59. [67]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv. org/abs/2402.03300, 2 0 (3): 0 5, 2024

    2024 arXiv

  60. [68]

    Super-convergence: Very fast training of neural networks using large learning rates

    Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369--386. SPIE, 2019

    2019

  61. [69]

    Qwen2 technical report

    Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  62. [70]

    The calculus of variations

    B Van Brunt. The calculus of variations. Universitext. Springer, New York, NY, December 2004

    2004

  63. [71]

    Position: Will we run out of data? limits of llm scaling based on human-generated data

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning, 2024

    2024

  64. [72]

    Trl: Transformer reinforcement learning

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  65. [73]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  66. [74]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  67. [75]

    ema-pytorch: A simple way to keep track of an exponential moving average (ema) version of your pytorch model

    Phil Wang. ema-pytorch: A simple way to keep track of an exponential moving average (ema) version of your pytorch model. https://github.com/lucidrains/ema-pytorch, 2024. Accessed: 2025-06-20

    2024

  68. [76]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  69. [77]

    The large-sample distribution of the likelihood ratio for testing composite hypotheses

    Samuel S Wilks. The large-sample distribution of the likelihood ratio for testing composite hypotheses. The annals of mathematical statistics, 9 0 (1): 0 60--62, 1938

    1938

  70. [78]

    Huggingface's transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  71. [79]

    On early stopping in gradient descent learning

    Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive approximation, 26 0 (2): 0 289--315, 2007

    2007

  72. [80]

    Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model

    Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019

    2019

  73. [81]

    How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024

    Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024

    2024 arXiv

  74. [82]

    Parameter identification for fractional ornstein--uhlenbeck processes based on discrete observation

    Pu Zhang, Wei-lin Xiao, Xi-li Zhang, and Pan-qiang Niu. Parameter identification for fractional ornstein--uhlenbeck processes based on discrete observation. Economic Modelling, 36: 0 198--203, 2014

    2014

  75. [83]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.