Pith. sign in

REVIEW 4 major objections 61 references

WSqD replaces WSD's flat phase with a shifted inverse-square-root base so one peak learning rate can be reused when training is extended, with a final linear cooldown that recovers optimal last-iterate rates in convex stochastic analysis.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:03 UTC pith:NLCNZSKR

load-bearing objection Clean anytime-ish schedule with a correct last-iterate convex proof and honest small-scale continuation experiments; the theory-to-AdamW transfer is the real soft spot, not a hidden flaw in the math. the 4 major comments →

arxiv 2607.10959 v1 pith:NLCNZSKR submitted 2026-07-12 cs.LG math.OCstat.ML

WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

Jianhao Ma , Yuxin Chen This is my paper
classification cs.LG math.OCstat.ML
keywords learning rate schedulehorizon-free trainingcontinued pretrainingWSqDwarmup-stable-decaystochastic convex optimizationlast-iterate convergencelarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard learning-rate schedules either lock to a fixed training length or keep a constant peak rate that drifts out of tune when that length is extended. This paper proposes WSqD: a short warmup, a shifted inverse-square-root base that never needs the final horizon, and a short linear cooldown that is started only when a deployable endpoint is chosen. In the classical convex stochastic setting the schedule recovers the minimax-optimal last-iterate rate of order 1 over square-root of T; the scale of the base is independent of T. On SlimPajama pretraining of a 213M LLaMA-style model, a single base rate chosen on a short pilot run and reused without retuning matches or beats carefully tuned WSD and related baselines out to 60k iterations. The practical payoff is that mid-training and post-hoc horizon extension become schedule-reuse problems rather than retuning problems.

Core claim

A shifted inverse-square-root base plus a final linear cooldown (WSqD) is horizon-independent in its base parameters, attains the optimal O(1/√T) last-iterate rate for stochastic mirror descent under standard convex assumptions, and, on language-model pretraining, lets one peak learning rate selected on a short pilot be reused across substantially longer continuations while matching or outperforming WSD and related schedules.

What carries the argument

WSqD schedule: η_t = c0 / √(t + T0) until a fixed fraction of the committed horizon, then linear decay to zero. The inverse-square-root base is horizon-free; the linear cooldown is what removes the classical log-T last-iterate overhead and recovers the minimax rate.

Load-bearing premise

That the convex, nonsmooth mirror-descent theory is a reliable guide for the non-convex AdamW training of transformers, so that a base rate tuned on a short pilot remains near-optimal when the run is extended.

What would settle it

At a larger model scale or different corpus, re-tune the peak rate separately at each continuation horizon for both WSD and WSqD; if WSqD's transferred single rate falls clearly behind the re-tuned WSD (or the power schedule) at long horizons, the claimed horizon-free transfer fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper proposes WSqD, a learning-rate schedule that replaces WSD’s constant stable phase with a shifted inverse-square-root base while retaining a final linear cooldown. In the nonsmooth stochastic convex setting, Theorem 1 shows that stochastic mirror descent under WSqD attains an O(1/√T) last-iterate rate for any fixed decay fraction α∈(0,1/2), with rate-optimal scale c0 independent of the horizon T (needed only to start the cooldown). Empirically, on 213M LLaMA-style pretraining on SlimPajama (plus limited OpenWebText2 and 124M ablations), a single base learning rate chosen on a short pilot and reused without retuning matches or improves over WSD across continuation horizons up to 60k steps, and is competitive with a power-law schedule and two-stage retuned WSD. The authors carefully distinguish practical horizon-freeness of the base phase from fully anytime schedules, and acknowledge that the theory does not cover AdamW/Muon dynamics.

Significance. Post-hoc horizon extension is a genuine practical problem in modern LLM pipelines, and the paper gives a clean, first-principles schedule that is both theoretically justified and simple to implement. The main theoretical contribution is solid: combining a horizon-independent inverse-square-root base with a linear cooldown recovers minimax-optimal last-iterate rates under standard mirror-descent assumptions, with a self-contained two-part proof (suffix checkpoint + geometric block propagation). Empirically, the most useful finding is that WSqD’s preferred base scale is stable across horizons in their setting (Figure 4), whereas WSD’s optimum drifts—directly supporting the reuse protocol. Strengths include an honest continuation protocol, seed/corpus/size ablations in Appendix C, and an explicit comparison to the closest empirical competitor (power schedule). The work is a credible contribution to schedule design even if the theory–practice transfer remains stylized.

major comments (4)
  1. The load-bearing practical claim is that a single base scale selected on a short pilot can be reused without retuning when the horizon is extended (§1.2, §3.2–3.3, Figure 4). All main evidence is a 213M model, ≤60k steps, AdamW, and two corpora. Section 5 correctly flags this limit, but the title/abstract frame the result as a schedule “for large model training.” For the journal version, either add at least one larger-scale continuation check, or reframe abstract/intro claims so that the proven convex result and the small-scale transfer evidence are not oversold as established large-model practice.
  2. Theorem 1 (Assumptions 1–3, Appendix A) is for nonsmooth convex stochastic mirror descent without momentum or adaptive state; all validation-loss experiments use AdamW (§3.1). Appendix B is only an informal geometric diagnostic of omitted dual-norm scales. This gap is acknowledged, but it is still the main correctness-risk for transferring the horizon-independence of c0 to practice. Please state more sharply in §2 and the abstract what is proven versus what is hypothesized, and avoid language that presents the AdamW results as confirmation of the last-iterate theorem.
  3. §3.5 / Figure 6b: under the same short-horizon tuning protocol, the power schedule of Shen et al. slightly outperforms WSqD at the reported horizons (gap shrinks by 60k). The abstract’s “matches or outperforms … other baselines” is therefore only marginally accurate for this closest competitor. Please qualify the claim (e.g., matches power schedule while using a fixed −1/2 exponent and a simpler shift regularizer) and discuss when a practitioner should prefer WSqD over the power schedule rather than treating them as interchangeable.
  4. Figure 4 / §3.3: the claim that WSqD’s optimum is fixed at η⋆_max=0.0015 across T∈{15k,…,60k} rests on a coarse grid {0.0005,0.001,0.0015,0.002,0.0025,0.003}. Mild horizon drift could be hidden between grid points. A denser sweep around 0.001–0.002 (or reporting a fitted optimum) would make the “remarkably stable” claim more convincing and better support the no-retuning protocol.

Circularity Check

0 steps flagged

No circularity: last-iterate rate is proved from standard mirror-descent inequalities; empirical single-rate reuse is a transfer test, not a fit-to-target.

full rationale

The paper's derivation chain is self-contained and non-circular. Theorem 1 is obtained from classical stochastic mirror-descent inequalities (Lemma 1), a suffix-averaging argument adapted from Shamir and Zhang (2013) to locate a good Phase-1 checkpoint (Lemma 2), and a geometric block decomposition of the linear cooldown inspired by Jain et al. (2021) (Lemma 3). The O(1/√T) last-iterate bound and the horizon-independence of the rate-optimal scale c0 = Θ(Rψ/G) follow from these inequalities under Assumptions 1–3; they do not embed the empirical validation loss or any fitted LLM quantity. The inverse-square-root base is the classical Robbins–Monro / online-convex step-size sequence, not a quantity defined by the paper's own target. Empirically, ηmax (or c0) is chosen by a short pilot grid search and then held fixed across longer horizons—an honest transfer protocol, not a prediction of a quantity already used in the fit. There is no self-citation uniqueness theorem, no ansatz smuggled from the authors' prior work, and no renaming of a known empirical pattern as a first-principles derivation. The theory–practice gap (convex nonsmooth MD vs. nonconvex AdamW) is a correctness/transfer risk, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central theoretical claim rests on three standard stochastic-convex assumptions plus the explicit WSqD formula with free scale, shift, and decay fraction. No new physical entities are postulated; the schedule itself is the invented object. Empirical claims further depend on the usual LLM-training stack (AdamW, SlimPajama, fixed architecture) treated as given.

free parameters (4)
  • c0 (base scale / peak learning-rate scale)
    Controls overall magnitude of the inverse-square-root base; theory only gives order R_ψ/G, so the practical value is chosen by short-horizon search and reused.
  • T0 (shift)
    Hand-chosen shift that regularizes early large rates; recommended default T0 equal to pilot length; ablated but not derived.
  • α (decay fraction)
    Fixed fraction of horizon spent in linear cooldown (typically 0.2); enters both theory constants and all experiments.
  • empirical peak η_max for WSD/power baselines
    Grid-searched per protocol; for WSqD a single value is transferred, which is the point of the method but still a free hyperparameter.
axioms (4)
  • domain assumption Assumption 1: f convex and G-Lipschitz; stochastic subgradients almost surely bounded by G.
    Standard nonsmooth stochastic convex setup; invoked for all rate bounds in Section 2 and Appendix A.
  • domain assumption Assumption 2: ψ 1-strongly convex w.r.t. the chosen norm; finite Bregman diameter R_ψ².
    Defines the mirror-descent geometry used in the update and diameter terms.
  • domain assumption Assumption 3: unbiased stochastic subgradient oracle conditional on the filtration.
    Standard unbiasedness used to pass from stochastic to expected suboptimality.
  • ad hoc to paper Final decay occupies a constant fraction α ∈ (0,1/2) of the horizon; T large enough relative to T0 and α.
    Required for the O(1/√T) last-iterate claim; without a horizon-dependent cooldown the anytime lower bound of Kornowski & Shamir applies.
invented entities (1)
  • WSqD learning-rate schedule (shifted inverse-square-root base + linear cooldown) no independent evidence
    purpose: Provide a practically anytime base phase while recovering optimal last-iterate rate via a short final decay.
    The schedule is the paper's central construction; independent evidence is the theorem under stated assumptions plus the reported LLM runs, not an external physical prediction.

pith-pipeline@v1.1.0-grok45 · 29207 in / 3129 out tokens · 37574 ms · 2026-07-14T08:03:01.961244+00:00 · methodology

0 comments
read the original abstract

Standard learning rate schedules such as cosine annealing are tied to a fixed training horizon, limiting their ability to accommodate post hoc horizon extension. Warmup-stable-decay (WSD) partially addresses this issue by maintaining a long constant-rate phase before a short linear cooldown, allowing training to resume from a pre-decay checkpoint. However, its peak learning rate is still tuned based on the original training horizon and can become suboptimal when training is extended. Motivated by stochastic convex optimization, we propose WSqD (Warmup with Square-root base and linear Decay), a learning rate schedule that replaces WSD's constant stable phase with a shifted inverse-square-root base while retaining the final linear cooldown. In the stochastic convex setting, WSqD provably attains the minimax-optimal $O(1/\sqrt{T})$ last-iterate convergence rate. Importantly, its base learning rate schedule is horizon-independent, and the training horizon is needed only to determine when to begin the final cooldown. Empirically, on language-model pretraining using the SlimPajama corpus, WSqD matches or outperforms carefully tuned WSD and other baselines across multiple training horizons while reusing a single peak learning rate.

Figures

Figures reproduced from arXiv: 2607.10959 by Jianhao Ma, Yuxin Chen.

Figure 1
Figure 1. Figure 1: Illustration of cosine, WSD, and the proposed WSqD learning rate schedules. 2020), LLaMA (Touvron et al., 2023), and Llama 3 (Grattafiori et al., 2024), to name a few. Cosine annealing explicitly ties the learning rate trajectory to a pre-specified endpoint by smoothly decaying the learning rate from the peak learning rate ηmax to a small terminal learning rate ηmin over the planned horizon:1 η Cosine t = … view at source ↗
Figure 2
Figure 2. Figure 2: (a) Continued training on the SlimPajama dataset under WSD and WSqD across horizons of 15000, 30000, 45000, and 60000 iterations. (b) Fitted curves of the final validation losses after the linear decay phase. For both schedules, the base learning rate is selected by a 5000-iteration search; for WSqD, we use a shift parameter T0 = 5000. The decay fraction α is set to be 0.2 for both WSD and WSqD. increases.… view at source ↗
Figure 3
Figure 3. Figure 3: (a) Continued training on the SlimPajama dataset under WSD and WSqD across horizons of 15000, 30000, 45000, and 60000 iterations. (b) Fitted curves of the final validation losses after the linear decay phase. For both schedules, the base learning rate is selected by a 10000-iteration search; for WSqD, we use the shift parameter T0 = 10000. We set the decay fraction α as 0.2 for both WSD and WSqD. • Power s… view at source ↗
Figure 4
Figure 4. Figure 4: Final validation loss versus base learning rate for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study over the WSqD shift parameter T0. causes the inverse-square-root base to decay too aggressively during the early stages of training, leading to degraded final performance. In contrast, once T0 is moderately large, performance becomes relatively insensitive to its precise value. Based on these empirical results, we recommend the simple default T0 = T1, setting the shift equal to the length of… view at source ↗
Figure 6
Figure 6. Figure 6: (a) Validation-loss trajectories for two-stage tuned WSD with different second-phase peak learning rates. (b) Comparison of WSqD and the power schedule under continued training. to the emerging “mid-training” phase of LLM development (Mo et al., 2025). At the cooldown extreme, Bergsma et al. (2025) argued empirically that linear decay-to-zero is optimal under a tuned peak learning rate, while Tian et al. (… view at source ↗
Figure 7
Figure 7. Figure 7: Adam-style mirror-descent scale diagnostics. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Muon-style mirror-descent scale diagnostics. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Seed sensitivity of the SlimPajama continuation experiment from [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Continued training on OpenWebText2 with the 213M LLaMA model across horizons T ∈ {15000, 30000, 45000, 60000}. The base learning rate is ηmax = 0.0015 for both schedules; WSqD uses a shift parameter of T0 = 5000; and the decay fraction is α = 0.2. Apte, A., Deshpande, P., Kumar, N., Chakrabarti, S., and Kim, J. L. (2026). Anytime training with schedule-free spectral optimization. arXiv preprint arXiv:2605… view at source ↗
Figure 11
Figure 11. Figure 11: Continuation training on SlimPajama with a smaller 12-layer (124M-parameter) LLaMA model across horizons T ∈ {15000, 30000, 45000, 60000}. The base learning rate is ηmax = 0.0015 for both schedules; WSqD uses a shift parameter of T0 = 5000; and the decay fraction is α = 0.2. Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., Anthony, Q., Belilovsky, E., Rish, I., and Lesort, T. (2023). Continual pre-tra… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 22 linked inside Pith

  1. [1]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

    Deep Residual Learning for Image Recognition , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages =

  2. [2]

    2017 , publisher=

    First-order methods in optimization , author=. 2017 , publisher=

    2017

  3. [3]

    Loshchilov, Ilya and Hutter, Frank , booktitle =

  4. [4]

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , booktitle =

  5. [5]

    Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin , journal =

  6. [6]

    Hu, Shengding and Tu, Yuge and Han, Xu and He, Chaoqun and Cui, Ganqu and Long, Xiang and Zheng, Zhi and Fang, Yewei and Huang, Yuxiang and Zhao, Weilin and Zhang, Xinrong and Thai, Zheng Leng and Zhang, Kaihuo and Wang, Chongyi and Yao, Yuan and Zhao, Chenyang and Zhou, Jie and Cai, Jie and Zhai, Zhongwu and Ding, Ning and Jia, Chao and Zeng, Guoyang and...

  7. [7]

    Advances in Neural Information Processing Systems , year =

    Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations , author =. Advances in Neural Information Processing Systems , year =

  8. [8]

    arXiv preprint arXiv:2310.07831 , year =

    Optimal Linear Decay Learning Rate Schedules and Further Refinements , author =. arXiv preprint arXiv:2310.07831 , year =

    Pith/arXiv arXiv

  9. [9]

    Advances in Neural Information Processing Systems , volume=

    The road less scheduled , author=. Advances in Neural Information Processing Systems , volume=

  10. [10]

    Defazio, Aaron , journal =

  11. [11]

    International Conference on Learning Representations , year =

    Continual Pre-Training of Language Models , author =. International Conference on Learning Representations , year =

  12. [12]

    arXiv preprint arXiv:2403.08763 , year =

    Simple and Scalable Strategies to Continually Pre-train Large Language Models , author =. arXiv preprint arXiv:2403.08763 , year =

    Pith/arXiv arXiv

  13. [13]

    Advances in Neural Information Processing Systems , volume =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems , volume =

  14. [14]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and others , journal =. The

  15. [15]

    arXiv preprint arXiv:2412.19437 , year =

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2308.04014 , year =

    Continual Pre-Training of Large Language Models: How to (re)warm your model? , author =. arXiv preprint arXiv:2308.04014 , year =

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2510.06826 , year =

    Mid-Training of Large Language Models: A Survey , author =. arXiv preprint arXiv:2510.06826 , year =

    arXiv

  18. [18]

    arXiv preprint arXiv:2410.05192 , year =

    Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective , author =. arXiv preprint arXiv:2410.05192 , year =

    Pith/arXiv arXiv

  19. [19]

    The Annals of Mathematical Statistics , volume =

    A Stochastic Approximation Method , author =. The Annals of Mathematical Statistics , volume =

  20. [20]

    Foundations and Trends in Optimization , volume =

    Introduction to Online Convex Optimization , author =. Foundations and Trends in Optimization , volume =

  21. [21]

    Proceedings of the 30th International Conference on Machine Learning (ICML) , pages =

    Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes , author =. Proceedings of the 30th International Conference on Machine Learning (ICML) , pages =

  22. [22]

    and Netrapalli, Praneeth , journal =

    Jain, Prateek and Nagaraj, Dheeraj M. and Netrapalli, Praneeth , journal =. Making the Last Iterate of

  23. [23]

    Conference on Learning Theory (COLT) , pages =

    Tight Analyses for Non-Smooth Stochastic Gradient Descent , author =. Conference on Learning Theory (COLT) , pages =

  24. [24]

    SIAM Journal on Control and Optimization , volume =

    Acceleration of Stochastic Approximation by Averaging , author =. SIAM Journal on Control and Optimization , volume =

  25. [25]

    arXiv preprint arXiv:1109.5647 , year =

    Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization , author =. arXiv preprint arXiv:1109.5647 , year =

    Pith/arXiv arXiv

  26. [26]

    IEEE Transactions on Information Theory , volume =

    Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization , author =. IEEE Transactions on Information Theory , volume =

  27. [27]

    arXiv preprint arXiv:1212.2002 , year =

    A Simpler Approach to Obtaining an O(1/t) Convergence Rate for the Projected Stochastic Subgradient Method , author =. arXiv preprint arXiv:1212.2002 , year =

    Pith/arXiv arXiv 2002

  28. [28]

    SIAM Review , volume =

    Optimization Methods for Large-Scale Machine Learning , author =. SIAM Review , volume =

  29. [29]

    Last Iterate Convergence of

    Varre, Aditya Vardhan and Pillaud-Vivien, Loucas and Flammarion, Nicolas , booktitle =. Last Iterate Convergence of

  30. [30]

    arXiv preprint arXiv:2001.08361 , year =

    Scaling Laws for Neural Language Models , author =. arXiv preprint arXiv:2001.08361 , year =

    Pith/arXiv arXiv 2001

  31. [31]

    arXiv preprint arXiv:2203.15556 , year =

    Training Compute-Optimal Large Language Models , author =. arXiv preprint arXiv:2203.15556 , year =

    Pith/arXiv arXiv

  32. [32]

    Annual Meeting of the Association for Computational Linguistics , year =

    Don't Stop Pretraining: Adapt Language Models to Domains and Tasks , author =. Annual Meeting of the Association for Computational Linguistics , year =

  33. [33]

    arXiv preprint arXiv:2302.13971 , year =

    Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth. arXiv preprint arXiv:2302.13971 , year =

    Pith/arXiv arXiv

  34. [34]

    Advances in Neural Information Processing Systems , year =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems , year =

  35. [35]

    International Conference on Learning Representations , year =

    Adam: A Method for Stochastic Optimization , author =. International Conference on Learning Representations , year =

  36. [36]

    International Conference on Learning Representations , year =

    Decoupled Weight Decay Regularization , author =. International Conference on Learning Representations , year =

  37. [37]

    2024 , howpublished =

    Muon: An Optimizer for Hidden Layers in Neural Networks , author =. 2024 , howpublished =

    2024

  38. [38]

    Proceedings of the 20th International Conference on Machine Learning (ICML) , pages =

    Online Convex Programming and Generalized Infinitesimal Gradient Ascent , author =. Proceedings of the 20th International Conference on Machine Learning (ICML) , pages =

  39. [39]

    and Hestness, Joel and Dey, Nolan , year =

    Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R. and Hestness, Joel and Dey, Nolan , year =

  40. [40]

    Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for

    Bergsma, Shane and Dey, Nolan and Gosal, Gurpreet and Gray, Gavia and Soboleva, Daria and Hestness, Joel , journal =. Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for

  41. [41]

    arXiv preprint arXiv:2602.03702 , year=

    Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging , author=. arXiv preprint arXiv:2602.03702 , year=

    arXiv

  42. [42]

    International Conference on Learning Representations , year =

    Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods , author =. International Conference on Learning Representations , year =

  43. [43]

    arXiv preprint arXiv:2312.08531 , year=

    Revisiting the last-iterate convergence of stochastic gradient methods , author=. arXiv preprint arXiv:2312.08531 , year=

    arXiv

  44. [44]

    arXiv preprint arXiv:2409.20325 , year=

    Old optimizer, new norm: An anthology , author=. arXiv preprint arXiv:2409.20325 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2605.23061 , year =

    Anytime Training with Schedule-Free Spectral Optimization , author =. arXiv preprint arXiv:2605.23061 , year =

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2604.13870 , year=

    Gradient Descent's Last Iterate is Often (slightly) Suboptimal , author=. arXiv preprint arXiv:2604.13870 , year=

    Pith/arXiv arXiv

  47. [47]

    arXiv preprint arXiv:2407.07263 , year =

    Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models , author =. arXiv preprint arXiv:2407.07263 , year =

    Pith/arXiv arXiv

  48. [48]

    arXiv preprint arXiv:2408.13359 , year =

    Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler , author =. arXiv preprint arXiv:2408.13359 , year =

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2503.02844 , year=

    Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training , author=. arXiv preprint arXiv:2503.02844 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2507.17634 , year=

    WSM: decay-free learning rate schedule via checkpoint merging for LLM pre-training , author=. arXiv preprint arXiv:2507.17634 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2406.14833 , year =

    Efficient Continual Pre-training by Mitigating the Stability Gap , author =. arXiv preprint arXiv:2406.14833 , year =

    Pith/arXiv arXiv

  52. [52]

    International Conference on Learning Representations (ICLR) , year =

    Continual Evaluation for Lifelong Learning: Identifying the Stability Gap , author =. International Conference on Learning Representations (ICLR) , year =

  53. [53]

    arXiv preprint arXiv:2501.18965 , year=

    The surprising agreement between convex optimization theory and learning-rate scheduling for large model training , author=. arXiv preprint arXiv:2501.18965 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2511.07767 , year =

    Schedulers for Schedule-free: Theoretically Inspired Hyperparameters , author =. arXiv preprint arXiv:2511.07767 , year =

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2501.00656 , year=

    2 OLMo 2 Furious , author=. arXiv preprint arXiv:2501.00656 , year=

    Pith/arXiv arXiv

  56. [56]

    Scaling Optimal

    Bjorck, Johan and Benhaim, Alon and Chaudhary, Vishrav and Wei, Furu and Song, Xia , booktitle =. Scaling Optimal. 2025 , note =

    2025

  57. [57]

    International Conference on Machine Learning (ICML) , year =

    Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Time , author =. International Conference on Machine Learning (ICML) , year =

  58. [58]

    Early Weight Averaging Meets High Learning Rates for

    Sanyal, Sunny and Neerkaje, Atula Tejaswi and Kaddour, Jean and Kumar, Abhishek and Sanghavi, Sujay , booktitle =. Early Weight Averaging Meets High Learning Rates for. 2024 , note =

    2024

  59. [59]

    arXiv preprint arXiv:2507.04206 , year =

    Mpemba Effect in Large-Language Model Training Dynamics: A Minimal Analysis of the Valley-River Model , author =. arXiv preprint arXiv:2507.04206 , year =

    Pith/arXiv arXiv

  60. [60]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  61. [61]

    International Conference on Learning Representations (ICLR) , year =

    A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules , author =. International Conference on Learning Representations (ICLR) , year =