Pith. sign in

REVIEW 1 major objections 4 minor 36 references

Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles

T0 review · 1 major / 4 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Schedule-Free GD and SGD, used as-is on smooth nonconvex problems, match the optimal first-order rates and escape strict saddles after a tiny one-time nudge.

desk verdict Solid first rate-optimal nonconvex theory for unmodified Schedule-Free; the y_k rates hold cleanly, the saddle kick is a minor technical price, and x_k remains the open practical gap. read the letter →

arxiv 2607.09167 v1 pith:QDC5J55P submitted 2026-07-10 cs.LG math.OC

Jiseok Chae , Donghwan Kim This is my paper
classification cs.LGmath.OC
keywords Schedule-FreeoptimizationnonconvexsmoothLyapunovanalysiscontinuous-timeODEstrictsaddleavoidancefirst-orderratesperformanceestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Schedule-Free optimizers drop hand-tuned learning-rate schedules yet often match or beat carefully scheduled methods on deep-learning tasks. Their theory had been mostly limited to convex losses, leaving open whether the same schemes remain reliable for the nonconvex objectives that dominate modern machine learning. This paper shows that the gradient-location sequence produced by ordinary Schedule-Free gradient descent and stochastic gradient descent already attains the best known worst-case rates among first-order methods: O(1/T) for the squared gradient norm in the deterministic case and O(1/√T) in the stochastic case. The argument begins from a continuous-time ODE that the discrete method approximates, constructs a simple Lyapunov function for that ODE, then transfers the same potential to the discrete iterates under the practical choices of constant or warm-up learning rates and averaging weights. The same dynamics, viewed as a non-autonomous system, are further shown to avoid strict saddle points almost surely after an arbitrarily small one-time perturbation of a single iterate. Together the results supply a nonconvex foundation for the strong empirical behavior already observed in practice.

What carries the argument

A Lyapunov potential V_k = f(y_k) − f* + α_k ||z_k − x_k||² whose coefficients α_k are chosen to cancel the inner-product terms that appear when the continuous-time Lyapunov function of the Schedule-Free ODE is discretized; the same potential yields both the rate guarantees and the descent needed for the dynamical-system argument.

What would settle it

Construct an L-smooth nonconvex function and a constant learning rate below 1/L for which the unperturbed Schedule-Free trajectory (standard initialization, no added noise) converges to a strict saddle with positive probability, or exhibit a smooth instance on which the gradient-location sequence fails to achieve the O(1/T) rate of Corollary 4.5.

Watch

Extended reading notes

Core claim

Under standard L-smoothness (and unbiased bounded-variance stochastic gradients), the unmodified Schedule-Free gradient-location iterates y_k achieve the optimal first-order worst-case rates min_k ||∇f(y_k)||² = O(1/T) deterministically and O(1/√T) stochastically, while the same continuous-time-motivated Lyapunov analysis plus a negligible one-time absolute-continuous perturbation also guarantees almost-sure avoidance of strict saddles.

Load-bearing premise

Strict-saddle avoidance needs an arbitrarily small but nonzero random one-time push after the first averaging step; without that push the iterates start on a measure-zero diagonal and the almost-sure claim does not go through.

Editorial extensions

If this is right

  • Practitioners can run Schedule-Free GD/SGD without learning-rate schedules and still inherit the same optimal nonconvex rates that classical first-order methods enjoy.
  • The continuous-time Lyapunov construction supplies a reusable template for analyzing other averaging-plus-momentum schemes on nonconvex landscapes.
  • Under the usual strict-saddle landscape assumptions, convergent Schedule-Free trajectories are expected to reach local minimizers rather than saddles once a negligible initialization-level perturbation is allowed.
  • Worst-case PEP bounds on the evaluation sequence x_k remain weaker than those on y_k, clarifying that the theoretically rate-optimal object is the gradient-location iterate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same Lyapunov idea extends to adaptive base optimizers (Adam, etc.), Schedule-Free wrappers would inherit nonconvex rate guarantees without schedule engineering.
  • The gap between PEP-certified rates for x_k and y_k suggests that practical success of the evaluation sequence may rely on local convex-like regularity along trajectories rather than global worst-case guarantees.
  • A fully rigorous avoidance proof without any perturbation would require showing that the diagonal itself is not an attractor for strict saddles—an open dynamical-systems question left by the present argument.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper analyzes Schedule-Free GD and SGD on L-smooth nonconvex objectives in their standard form (with practical warmup and clipped averaging rates). From a continuous-time SF-ODE Lyapunov function, the authors construct discrete Lyapunov potentials and prove that the gradient-location iterates {y_k} attain the optimal first-order rates: O(1/T) for SF-GD and O(1/√T) for SF-SGD (Corollaries 4.5–4.6), matching known lower bounds. They further cast SF-GD as a nonautonomous dynamical system and prove almost-sure strict-saddle avoidance after an arbitrarily small one-time absolute-continuous perturbation of z, and use PEP to show that the evaluation iterates {x_k} can be worse in the worst case than {y_k}.

Significance. The central rate results fill a genuine gap: prior nonconvex analyses either randomized β, altered the update, or obtained suboptimal rates. The Lyapunov construction is derived from the ODE, holds under the exact practical parameter rules (Assumptions 4.1–4.2), and yields rates that match Carmon et al. and Arjevani et al. lower bounds without hidden constants. The saddle-avoidance argument (with the one-time kick shown not to degrade rates when its second moment is O(γ)) and the PEP study of evaluation iterates add useful qualitative understanding of why Schedule-Free methods work in practice. Machine-checked algebraic cancellations in the descent lemmas and fully specified PEP instances strengthen reproducibility.

major comments (1)
  1. Theorem 5.2 / G.7: Strict-saddle avoidance is proved only after a one-time absolute-continuous perturbation of z after the first averaging step. Without it the trajectory is forced onto the diagonal Δ, so the almost-sure claim fails for the unperturbed algorithm under the standard initialization. The paper correctly treats the kick as negligible for rates (Thm. G.8) and orthogonal to Corollaries 4.5–4.6, but the abstract and introduction should state more clearly that the unperturbed dynamics are not shown to avoid saddles.
minor comments (4)
  1. Section 6 / Figure 1: The PEP plots for min ||∇f(x_k)||² stop at T=100; a short remark on whether intermediate-β improvement persists for larger T would help.
  2. Assumption 4.2 and Proposition D.5: The explicit formula for k0 is useful but dense; a one-sentence intuition that large C simply delays the onset of averaging would aid readability.
  3. Section B.1: The critique of Brown et al. is technically correct; a brief forward pointer from the main text would make the comparison easier to locate.
  4. Notation: The dual use of C for the decoupling constant and for the continuous-function space in Appendix I is harmless but slightly confusing on first reading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: rate guarantees follow from algebraic Lyapunov design under standard smoothness/variance assumptions, not from fitted inputs or self-referential definitions.

full rationale

The paper's central claims (Corollaries 4.5–4.6) are worst-case rates for unmodified SF-GD/SF-SGD under L-smoothness and bounded-variance unbiased stochastic gradients. The continuous-time SF-ODE Lyapunov (Prop. 3.2) motivates the discrete ansatz V_k = f(y_k) − f* + α_k ∥z_k − x_k∥²; the coefficients α_k are then chosen algebraically so that the inner-product terms cancel (Lem. E.4 / Prop. 4.3), and the resulting descent inequalities are verified by direct expansion under Assumptions 1.1–1.3 and 4.1–4.2. No parameter is fitted to data and then re-presented as a prediction. Lower-bound optimality is imported from external references (Carmon et al., Arjevani et al.), not from author-owned uniqueness theorems. Self-citations (Defazio et al. for the algorithm definition; Song et al. for the optional C-decoupling) supply context or parameter ranges but are not load-bearing for the rate proofs. Strict-saddle avoidance (Thm. 5.2/G.7) uses an external nonautonomous dynamical-systems framework (Mușat–Boumal) plus a one-time absolute-continuous perturbation; the rates themselves are proved for the unperturbed method and shown to survive a small mean-zero perturbation (Thm. G.8). PEP numerics are independent computational checks, not circular. The derivation chain is therefore self-contained against external benchmarks.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central rate claims rest only on classical smoothness, lower-boundedness, and bounded-variance stochastic oracles, plus the explicit learning-rate and averaging-rate schedules already used by practitioners. No new physical entities or data-fitted constants are introduced; the free parameters are the usual algorithmic hyper-parameters whose dependence is tracked explicitly in the bounds.

free parameters (3)
  • β (momentum/interpolation weight)
    Fixed in [0,1); appears in every rate as a 1/(1-β) factor. Chosen by the user, not fitted to data.
  • C (decoupling constant for averaging rates)
    Appears in the definition of c_{k+1}; multiplies the logarithmic noise term. User-chosen, default recovers the original Schedule-Free rule.
  • γ (base step size) and T_w (warmup length)
    Standard algorithmic choices; the theorems give explicit dependence on both.
assumptions (4)
  • domain assumption f is L-smooth (Assumption 1.1)
    Standard first-order smoothness used for every descent lemma and for the ODE analysis.
  • domain assumption f is bounded below by f* (Assumption 1.2)
    Prevents pathological linear objectives; used to keep Lyapunov potentials non-negative.
  • domain assumption Stochastic gradients are unbiased with variance ≤σ² (Assumption 1.3)
    Classical noise model that yields the O(1/√T) rate for SF-SGD.
  • ad hoc to paper Learning rates and averaging rates obey the explicit warmup and clipped-sum rules (Assumptions 4.1–4.2)
    Matches practical Schedule-Free implementations; the proofs rely on monotonicity and the concrete upper bound c_{k+1}≤6(1-β)C(k+1)/((k+2)(2k+3)).
invented entities (2)
  • SF-ODE (continuous-time limiting system)
    purpose: Supplies the Lyapunov function that is later discretized to prove rates.
    Derived as the formal γ→0 limit of the discrete updates; existence near t=0 is proved by a contraction argument on an integral equation.
  • Discrete Lyapunov potentials V_k = f(y_k)-f* + α_k‖z_k-x_k‖²
    purpose: Monotone (in expectation) energy that yields the optimal rates.
    Coefficients α_k are chosen algebraically to cancel inner-product terms; not fitted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles." pith.science (2026). https://pith.science/paper/QDC5J55P

@misc{pith2026260709167,
  author       = {Pith},
  title        = {Pith review of: Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDC5J55P}},
  note         = {Machine review of arXiv:2607.09167}
}
read the original abstract

Schedule-Free methods have attracted growing interest for alleviating the burden of designing and tuning a learning rate scheduler, while matching and sometimes even outperforming optimizers with tuned schedulers. Despite their strong empirical results, their convergence theory in nonconvex optimization, where modern machine learning objectives typically arise, has remained largely unexplored. In this paper, we provide worst-case analyses of Schedule-Free gradient descent and Schedule-Free stochastic gradient descent, in their standard form and without auxiliary modifications or restrictive conditions, for smooth but possibly nonconvex objectives. Based on a Lyapunov analysis derived from the continuous-time limiting ordinary differential equation associated with these methods, we show that Schedule-Free gradient descent and Schedule-Free stochastic gradient descent achieve the optimal worst-case convergence rates attainable among first-order methods. We further formulate Schedule-Free gradient descent as a nonautonomous dynamical system and prove strict-saddle avoidance under an arbitrarily small one-time perturbation. These theoretical results provide a better understanding of the strong performance that Schedule-Free methods demonstrate.

Figures

Figures reproduced from arXiv: 2607.09167 by the authors.

Figure 1
Figure 1. PEP results on the convergence of the evaluation iterates [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. The convergence plots regarding the gradient location iterates [PITH_FULL_IMAGE:figures/full_fig_p040_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 3 linked inside Pith

  1. [1]

    The exact worst-case conver- gence rate of the gradient method with fixed step lengths for L-smooth functions.Optimization Letters, 16(6):1649–1661, 2022

    Hadi Abbaszadehpeivasti, Etienne de Klerk, and Moslem Zamani. The exact worst-case conver- gence rate of the gradient method with fixed step lengths for L-smooth functions.Optimization Letters, 16(6):1649–1661, 2022

    2022

  2. [2]

    General framework for online-to- nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization

    Kwangjun Ahn, Gagik Magakyan, and Ashok Cutkosky. General framework for online-to- nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization. In International Conference on Machine Learning, 2025

    2025

  3. [3]

    Duchi, Dylan J

    Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization.Mathematical Programming, 199(1):165–214, 2023

    2023

  4. [4]

    Analysis of Schedule-Free non-convex optimiza- tion

    Connor Brown, Chi Jin, and Ahmed Khaled. Analysis of Schedule-Free non-convex optimiza- tion. InOPT2025: 17th Annual Workshop on Optimization for Machine Learning, 2025

    2025

  5. [5]

    Duchi, Oliver Hinder, and Aaron Sidford

    Yair Carmon, John C. Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points II: first-order methods.Mathematical Programming, 185(1):315–355, 2021

    2021

  6. [6]

    Zico Kolter, and Ameet Talwalkar

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Conference on Learning Representations, 2021

    2021

  7. [7]

    George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badur...

    arXiv 2023

  8. [8]

    Momentum via primal averaging: Theoretical insights and learning rate schedules for non-convex optimization.arXiv preprint arXiv:2010.00406, 2020

    Aaron Defazio. Momentum via primal averaging: Theoretical insights and learning rate schedules for non-convex optimization.arXiv preprint arXiv:2010.00406, 2020

    arXiv 2010

Show all 36 references
  1. [9]

    The road less scheduled.Advances in Neural Information Processing Systems, 37, 2024

    Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled.Advances in Neural Information Processing Systems, 37, 2024

    2024

  2. [10]

    Rishabh Dixit, Mert Gürbüzbalaban, and Waheed U. Bajwa. Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima.Foundations of Computational Mathematics, 2026

    2026

  3. [11]

    The complexity of finding stationary points with stochastic gradient descent

    Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. InInternational Conference on Machine Learning, 2020

    2020

  4. [12]

    Performance of first-order methods for smooth convex mini- mization: a novel approach.Mathematical Programming, 145(1):451–482, 2014

    Yoel Drori and Marc Teboulle. Performance of first-order methods for smooth convex mini- mization: a novel approach.Mathematical Programming, 145(1):451–482, 2014

    2014

  5. [13]

    Stochastic first-and zeroth-order methods for nonconvex stochastic programming.SIAM Journal on Optimization, 23(4):2341–2368, 2013

    Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming.SIAM Journal on Optimization, 23(4):2341–2368, 2013. 10

    2013

  6. [14]

    Hendrickx, Adrien B

    Baptiste Goujaud, Céline Moucer, François Glineur, Julien M. Hendrickx, Adrien B. Taylor, and Aymeric Dieuleveut. PEPit: computer-assisted worst-case analyses of first-order optimization methods in Python.Mathematical Programming Computation, 16(3):337–367, 2024

    2024

  7. [15]

    Vetrov, and Andrew Gordon Wilson

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. InProceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 876–885, 2018

    2018

  8. [16]

    Better theory for SGD in the nonconvex world.Transactions on Machine Learning Research, 2023

    Ahmed Khaled and Peter Richtárik. Better theory for SGD in the nonconvex world.Transactions on Machine Learning Research, 2023

    2023

  9. [17]

    Kingma and Jimmy Lei Ba

    Diederik P. Kingma and Jimmy Lei Ba. A method for stochastic optimization. InInternational Conference on Learning Representations, 2015

    2015

  10. [18]

    Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I

    Jason D. Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. First-order methods almost always avoid strict saddle points.Mathematical programming, 176(1):311–337, 2019

    2019

  11. [19]

    Loss landscapes and optimization in over- parameterized non-linear systems and neural networks.Applied and Computational Harmonic Analysis, 59:85–116, 2022

    Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks.Applied and Computational Harmonic Analysis, 59:85–116, 2022

    2022

  12. [20]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  13. [21]

    Exponential moving average of weights in deep learning: Dynamics and benefits.Transactions on Machine Learning Research, 2024

    Daniel Morales-Brotons, Thijs V ogels, and Hadrien Hendrikx. Exponential moving average of weights in deep learning: Dynamics and benefits.Transactions on Machine Learning Research, 2024

    2024

  14. [22]

    Connections between Schedule- Free optimizers and accelerated SGD variants

    Depen Morwani, Nikhil Vyas, Hanlin Zhang, and Sham Kakade. Connections between Schedule- Free optimizers and accelerated SGD variants. InOPT2024: 16th Annual Workshop on Optimization for Machine Learning, 2024

    2024

  15. [23]

    A non-autonomous center-stable set theorem for saddle avoidance in optimization.arXiv preprint arXiv:2603.02782, 2026

    Andreea-Alexandra Mu¸ sat and Nicolas Boumal. A non-autonomous center-stable set theorem for saddle avoidance in optimization.arXiv preprint arXiv:2603.02782, 2026

    2026 arXiv

  16. [24]

    Quasi-monotone subgradient methods for nonsmooth convex minimization.Journal of Optimization Theory and Applications, 165(3):917–940, 2015

    Yurii Nesterov and Vladimir Shikhman. Quasi-monotone subgradient methods for nonsmooth convex minimization.Journal of Optimization Theory and Applications, 165(3):917–940, 2015

    2015

  17. [25]

    Iterate averaging as regularization for stochastic gradient descent

    Gergely Neu and Lorenzo Rosasco. Iterate averaging as regularization for stochastic gradient descent. InConference on Learning Theory, pages 3222–3242. PMLR, 2018

    2018

  18. [26]

    Michael O’Neill and Stephen J. Wright. Behavior of accelerated gradient methods near critical points of nonconvex functions.Mathematical Programming, 176(1):403–427, 2019

    2019

  19. [27]

    Boris T. Polyak. New stochastic approximation type procedures.Avtomatika i Telemekhanika, 7:98–107, 1990

    1990

  20. [28]

    Efficient estimations from a slowly convergent Robbins–Monro process

    David Ruppert. Efficient estimations from a slowly convergent Robbins–Monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988

    1988

  21. [29]

    Ryu, Adrien B

    Ernest K. Ryu, Adrien B. Taylor, Carolina Bergeling, and Pontus Giselsson. Operator splitting performance estimation: Tight contraction factors and optimal parameter selection.SIAM Journal on Optimization, 30(3):2251–2271, 2020

    2020

  22. [30]

    Through the river: Under- standing the benefit of Schedule-Free methods for language model training.Advances in Neural Information Processing Systems, 38, 2025

    Minhak Song, Beomhan Baek, Kwangjun Ahn, and Chulhee Yun. Through the river: Under- standing the benefit of Schedule-Free methods for language model training.Advances in Neural Information Processing Systems, 38, 2025

    2025

  23. [31]

    Primal averaging: A new gradient evaluation step to attain the optimal individual convergence.IEEE Transactions on Cybernetics, 50(2): 835–845, 2018

    Wei Tao, Zhisong Pan, Gaowei Wu, and Qing Tao. Primal averaging: A new gradient evaluation step to attain the optimal individual convergence.IEEE Transactions on Cybernetics, 50(2): 835–845, 2018. 11

    2018

  24. [32]

    Taylor, Julien M

    Adrien B. Taylor, Julien M. Hendrickx, and François Glineur. Exact worst-case convergence rates of the proximal gradient method for composite convex minimization.Journal of Optimiza- tion Theory and Applications, 178(2):455–476, 2018

    2018

  25. [33]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape view

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape view. InInternational Conference on Learning Representations, 2025

    2025

  26. [34]

    fast” variable zk and the “slow

    Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh. SGD converges to global minimum in deep learning via star-convex path. InInternational Conference on Learning Representations, 2019. 12 Contents 1 Introduction 1 1.1 Our Contributions . . . . . . . . . . . ...

    2019

  27. [35]

    Limitations

    Then for any (y,x)∈B r(0), by Proposition G.9, we have ∥y∥= y 0 ≤ √ 2 y 0 max < √ 2r=δ. Therefore, for any (y1,x 1) and (y2,x 2) in Br(0), whenever k≥K 0, from the three inequalities (64), (65), and (66), we obtain that hk(y1,x 1)−T k(y1,x 1)− hk(y2,x 2)−T k(y2,x 2) max ≤ γk(1...

  28. [36]

    Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.