Pith. sign in

REVIEW 5 major objections 5 minor 30 references

Extending the step-size restriction for gradient descent to avoid strict saddle points

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gradient descent with step sizes up to 2/L almost surely avoids strict saddles when a measure-zero Hessian degeneracy condition holds.

desk verdict The fixed-step-size extension to αL<2 is genuine and the degeneracy condition is the right one; the varying-step-size corollary has a real gap in the Jacobian analysis. read the letter →

arxiv 1908.01753 v1 pith:AMLNEIHZ submitted 2019-08-05 stat.ML cs.LG

Hayden Schaeffer , Scott G. McCalla This is my paper · ORCID
classification stat.MLcs.LG MSC 65K0590C2637D10
keywords gradientdescentstrictsaddlepointsstep-sizeLipschitzmeasurezerocentermanifoldtheoremnon-convexoptimizationlearningrateschedule
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proves that gradient descent on a smooth non-convex function will almost surely avoid strict saddle points even when the step size is as large as 2/L, the same upper limit that guarantees convergence in the convex case. The catch is a measure-zero condition: the set of points where 1/alpha is an eigenvalue of the Hessian must be negligible and free of saddles. Under that condition, the paper shows that the backward map of gradient descent sends null sets to null sets, so the stable manifolds of strict saddles can only be reached from a set of initializations of measure zero. This closes an open gap between the previous alpha < 1/L results and the sharp 2/L bound, and it extends to decaying and piecewise-constant learning-rate schedules.

What carries the argument

The machinery is the gradient-descent update map g(x) = x - $\alpha$ grad f(x) together with the locus A where its Jacobian I - $\alpha$ Hessian f(x) loses rank: exactly the set where $alpha^{{-1}}$ is an eigenvalue of the Hessian. The measure-zero assumption on A guarantees, via the inverse function theorem and a Lindelöf covering, that the inverse relation $g^{{-1}}$ sends Lebesgue-null sets to null sets, a Luzin-N-type property. With that in hand, the center manifold theorem gives the strict saddle's center-stable manifold W^cs_loc as a $C^{1}$ embedded disc of dimension at most d-1, hence null; any orbit that lands in a neighborhood of a saddle forever must lie on W^cs_loc, and its preimage under iterates of g is null. Thus the whole set of bad initializations is a countable union of null sets.

What would settle it

A decisive numerical test is to choose a smooth non-convex objective whose Hessian degeneracy set {x : $alpha^{{-1}}$ in $\sigma$($grad^{2}$ f(x))} can be verified to have measure zero, set $\alpha$ = 1.9/L, sample many uniform random initializations on a convex forward-invariant domain, and observe whether any converge to a strict saddle; the theorem predicts a zero fraction, while Example 2.3 shows that dropping the measure-zero condition produces a positive fraction at $\alpha$ = 1/L.

Watch

Extended reading notes

Core claim

For a twice-differentiable objective f on a convex forward-invariant set $\Omega$ with L-Lipschitz gradient, gradient descent with fixed step size $\alpha$ in (0,2/L) converges to a strict saddle only for a set of initializations of Lebesgue measure zero, provided the degeneracy set A = {x in $\Omega$ : $alpha^{{-1}}$ in $\sigma$($grad^{2}$ f(x))} has measure zero and contains no saddle points. The proof shows the update map g(x) = x - $\alpha$ grad f(x) is invertible off A, that $g^{{-1}}$ therefore maps null sets to null sets, and that the center-stable manifold of any strict saddle has dimension at most d-1; pulling these null manifolds back along the forward orbit yields a null set of starting points. This reaches the step-size frontier $\alpha$ L = 2 that is sharp for convex quadratics, and it answers the open question of whether $\alpha$ < 1/L was necessary.

Load-bearing premise

The proof relies on the set of points where 1/alpha matches an eigenvalue of the Hessian being a measure-zero set that contains no saddle points; if that set has positive measure, the update map can compress a positive-measure region onto a saddle's stable manifold, as Example 2.3 demonstrates.

Editorial extensions

If this is right

  • With alpha L exactly equal to 1, gradient descent is almost surely safe from strict saddles whenever the set where the Hessian's spectral norm attains L is measure zero and contains no saddles (Corollary 3.3).
  • For fixed alpha in the previously forbidden interval (1/L, 2/L), random initialization lands on a strict saddle's basin with probability zero, so larger learning rates do not reintroduce saddle attraction under the null-degeneracy condition.
  • Continuously decaying schedules, with alpha_{n+1} = h(alpha_n) for a strictly decreasing contraction h, and finite staircases of piecewise-constant step sizes, both preserve almost-sure avoidance under corresponding degeneracy conditions (Corollaries 3.7 and 3.8).
  • The theorem imposes no isolation assumption on strict saddles, so saddles may be non-isolated and the conclusion still holds.
  • Replacing L by the positive Lipschitz constant L+ (controlling only positive Hessian eigenvalues) gives the same avoidance for alpha L+ in (0,2) under the same null-set condition (Corollary 3.6).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The null-set-preservation mechanism is generic: for 'most' smooth functions, the set where a fixed value lies in the Hessian spectrum should be a low-dimensional submanifold, so the measure-zero assumption may often hold automatically in practice.
  • The same argument should transfer to other C^1 first-order update maps (proximal gradient, block coordinate descent), as the paper notes in its discussion, though the formal proofs are not given here.
  • A practical rule suggested by the proof: avoid step sizes alpha whose reciprocal exactly matches an eigenvalue of the Hessian on any region of positive measure; randomizing or slightly perturbing alpha may reduce the risk of saddle convergence near alpha = 2/L.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies gradient descent x_{n+1} = x_n - α∇f(x_n) for twice continuously differentiable nonconvex objective functions f on a forward-invariant convex domain Ω whose gradient has Lipschitz constant L. Prior work established almost-sure avoidance of strict saddle points for α < 1/L. The paper's main result, Theorem 3.2, claims that if αL ∈ (0,2) and the degeneracy set {x ∈ Ω : α^{-1} ∈ σ(∇²f(x))} has Lebesgue measure zero and contains no saddle points, then a uniformly random initialization in Ω converges to a strict saddle with probability zero. The proof combines the center-stable manifold theorem with a measure-preservation argument for the inverse of the gradient descent map. The paper also states a variant using the positive Lipschitz constant L+ (Proposition 3.5 and Corollary 3.6), and two corollaries for step-size schedules: a continuous decaying schedule (Corollary 3.7) and a finite staircase of step sizes (Corollary 3.8). Example 2.2 and Example 2.3 illustrate the role of the degeneracy assumption. The central theorem is plausible and the main proof strategy is sound, but several points in the manuscript need correction before the claims are fully supported.

Significance. If Theorem 3.2 is repaired, it would answer a genuine open question in the area by extending the step-size bound from the previously known α < 1/L to the sharp αL < 2 condition, under an explicit and checkable degeneracy condition. The measure-zero assumption on the degeneracy set is natural, and Example 2.3 convincingly indicates that some such condition is necessary. The paper also gives the first treatment of variable step-size schedules in this context. The use of center-stable manifolds and the Luzin-N property is appropriate and, apart from the issues below, the main argument is well motivated. The paper does not provide machine-checked proofs or code, but the theoretical contribution is potentially valuable to the optimization and machine learning theory community.

major comments (5)
  1. [Section 3, proof of Theorem 3.2] The proof contains an invalid inference in the paragraph beginning 'Thus there exists an N...' It states that because the tail g^n(x0) lies in the union ⋃_ℓ B(x_ℓ) for all n ≥ N, there exists a single ℓ such that g^n(x0) ∈ B(x_ℓ) for all n ≥ N. This does not follow: a sequence could visit different balls of a union at different times. The gap is repairable because the trajectory converges to a specific strict saddle x_*, so its tail eventually lies in the ball around that limit; the countable-subcover argument is unnecessary. The proof should be rewritten to argue directly from convergence to the limit point.
  2. [Corollary 3.7, equations (3.2)-(3.3)] The Jacobian of the augmented map g(x,α) = (x - α∇f(x), h(α)) is block upper triangular, so its eigenvalues are σ(I - α∇²f(x)) ∪ {h'(α)}. The proof silently drops the eigenvalue h'(α) and asserts that the degeneracy set is exactly {α^{-1} ∈ σ(∇²f(x))}. This is only correct if h'(α) ≠ 0 for all α in the interval. The stated hypotheses (C¹, strictly decreasing, contractive) do not imply this: a C¹ strictly decreasing contraction may have h' = 0 on a positive-measure set, e.g., h(α) = α* - ∫_{α*}^{α} w(s) ds with w ≥ 0 continuous, w ≤ 1/2, and w = 0 on a fat Cantor set. In that case Dg is singular on a positive-measure slice even when the x-spectral set is null, and the null-set preservation argument fails. Corollary 3.7 needs an explicit hypothesis such as h'(α) < 0, and the proof must include {h' = 0} in the degeneracy set.
  3. [Example 2.3] The example states 'The only critical point is at (x,y) = (0,0)'. This is false for the explicit function q given. Since q(20) = 0 and q'(20) = 0.2, the equations ∂f/∂x = -2q(y)x = 0 and ∂f/∂y = y/2 - q'(y)x² = 0 have solutions (±√50, 20). At these points ∇f = 0 and the Hessian is indefinite, so they are additional strict saddle points. The focusing mechanism of the example still works, but the statement must be corrected and the role of the example reworded accordingly.
  4. [Corollary 3.8, proof] The set Q = ⋃_i ⋃_{j≥1} g_i^{-j}(A) does not capture all initializations that eventually hit the degeneracy set under a schedule that switches between maps. For instance, a point that is mapped by g_1 for N_1 steps and then by g_2 could lie in g_1^{-N_1}(g_2^{-j}(A)), which is generally not contained in Q. Consequently the assertion 'x0 ∉ Q so x_{N1} := g_1^{N1}(x0) ∉ Q' is unjustified. The later set S accounts for some mixed compositions, but the proof needs to define Q to include all finite compositions of the maps g_i^{-1} applied to A, or otherwise justify why points outside the smaller Q cannot enter the dangerous set after a switch.
  5. [Theorem 3.2, statement] The phrase 'given one uniformly random initialization in Ω' is not well-defined if Ω is an unbounded convex set of infinite Lebesgue measure, because no uniform probability measure exists on such a set. The theorem should restrict Ω to be bounded, or specify a probability measure that is absolutely continuous with a density, and the proof should state that all null-set statements are with respect to that measure.
minor comments (5)
  1. [Corollary 3.7, equation (3.2)] The expression h'(α - α*) should be h'(α); the shift by α* appears to be a typo.
  2. [Corollary 3.7, proof] The sentence 'Since h′ is negative' is not implied by the stated assumption that h is a C¹ strictly decreasing contractive map; the paper should either state h' < 0 as a hypothesis or avoid relying on strict negativity without proof.
  3. [Theorem 3.2, proof] The sentence 'This implies that g maps sets of measure zero to sets of measure zero in Ω \ A' is imprecise: the argument concerns the local inverse g^{-1}, and the paragraph then uses g^{-1}(U). The wording should be corrected to say that the local inverse preserves null sets.
  4. [Theorem 3.2, proof] The phrase 'the spectrum of Dg(x_k) is non-zero' should read 'does not contain zero', since spectra are sets of numbers.
  5. [Example 2.2] The text says the Lipschitz constant L = 10 is achieved at y = ±√(11/3), but those points lie on the boundary of the open domain Ω. The statement should say the supremum is approached as y approaches the boundary.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the avoidance theorem is derived from spectral non-degeneracy and stable-manifold measure arguments, not from its conclusion.

full rationale

No significant circularity found. Theorem 3.2's conclusion, that the set of initializations converging to strict saddles has measure zero, is derived from the center manifold theorem, the inverse function theorem, and the explicit null-measure assumption A = {x : α^{-1} ∈ σ(∇²f(x))}. That assumption is not defined in terms of convergence to saddles; it is a spectral non-degeneracy hypothesis, and Example 2.3 shows it is independent and necessary. The proof first shows that g^{-1} maps null sets to null sets away from A, then that the set of initializations eventually entering A is a countable union of null sets, and finally that each strict saddle's center-stable manifold has measure zero because the unstable dimension is at least one. No fitted parameter is renamed as a prediction, and no load-bearing result is imported from the authors' own prior work: the cited works [14, 15, 21] concern the different step-size regime α < 1/L, and the center manifold theorem is a standard external result. Corollaries 3.3, 3.6, 3.7, and 3.8 reuse the same argument rather than assuming the conclusion. Even if Corollary 3.7's treatment of h'(α) = 0 raises a correctness question, that would be a mathematical gap, not a circular reduction of the conclusion to the assumptions. The derivation is self-contained with respect to the stated hypotheses.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The proof rests on standard theorems and on two conditions on the objective: forward invariance of Omega and the null-measure degeneracy condition. The null-measure condition is the paper's principal new assumption; Example 2.3 shows it is necessary. No free parameters are fitted to data, and no new entities are introduced.

assumptions (6)
  • domain assumption f is C^2 on Omega and its gradient is L-Lipschitz.
    Stated at the start of Theorem 3.2; defines the class of objective functions under study.
  • domain assumption Omega is a convex forward-invariant subset of R^d for the gradient descent map g.
    Needed so that iterates stay in Omega and so that the set of initializations can be discussed as a subset of Omega; stated in Theorem 3.2.
  • ad hoc to paper The set {x in Omega : alpha^{-1} in sigma(grad^2 f(x))} has measure zero and contains no saddle points.
    This is the new technical condition introduced by the paper. It is not derived and is shown necessary by Example 2.3.
  • standard math Center Manifold Theorem (Theorem 3.1, after Shub).
    Used to identify the center-stable manifold W^cs_loc of a strict saddle as a measure-zero set.
  • standard math Inverse function theorem and the Luzin N property for local C^1 diffeomorphisms.
    Used to show that g^{-1} maps null sets to null sets away from the degeneracy set A.
  • ad hoc to paper For Corollary 3.7, the step-size map h satisfies h'(alpha) != 0 where needed (unstated).
    The stated assumptions allow h' = 0, which would make the augmented Jacobian singular for every x. The proof assumes degeneracy only comes from the x-component, so this is a gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extending the step-size restriction for gradient descent to avoid strict saddle points." pith.science (2026). https://pith.science/paper/AMLNEIHZ

@misc{pith2026190801753,
  author       = {Pith},
  title        = {Pith review of: Extending the step-size restriction for gradient descent to avoid strict saddle points},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AMLNEIHZ}},
  note         = {Machine review of arXiv:1908.01753}
}
read the original abstract

We provide larger step-size restrictions for which gradient descent based algorithms (almost surely) avoid strict saddle points. In particular, consider a twice differentiable (non-convex) objective function whose gradient has Lipschitz constant L and whose Hessian is well-behaved. We prove that the probability of initial conditions for gradient descent with step-size up to 2/L converging to a strict saddle point, given one uniformly random initialization, is zero. This extends previous results up to the sharp limit imposed by the convex case. In addition, the arguments hold in the case when a learning rate schedule is given, with either a continuous decaying rate or a piece-wise constant schedule.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 23 canonical work pages

  1. [1]

    Convergence of the iterates of descent methods for analytic cost functions

    Pierre-Antoine Absil, Robert Mahony, and Benjamin Andr ews. Convergence of the iterates of descent methods for analytic cost functions. SIAM Journal on Optimization , 16(2):531–547, 2005

    work page 2005

  2. [2]

    Efficient approaches f or escaping higher order saddle points in non-convex optimization

    Animashree Anandkumar and Rong Ge. Efficient approaches f or escaping higher order saddle points in non-convex optimization. In Conference on learning theory , pages 81–102, 2016

    work page 2016

  3. [3]

    Global optimality of local search for low rank matrix recovery

    Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Sre bro. Global optimality of local search for low rank matrix recovery. In Advances in Neural Information Processing Systems , pages 3873–3881, 2016

    work page 2016

  4. [4]

    Accelerated methods for nonconvex optimization

    Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidfo rd. Accelerated methods for nonconvex optimization. SIAM Journal on Optimization , 28(2):1751–1772, 2018

    work page 2018

  5. [5]

    Andrew R Conn, Nicholas IM Gould, and Philippe L. Toint. Trust region methods, volume 1. Siam, 2000

    work page 2000

  6. [6]

    Identifying and attacking the saddle point p roblem in high-dimensional non- convex optimization

    Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyungh yun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point p roblem in high-dimensional non- convex optimization. In Advances in neural information processing systems , pages 2933–2941, 2014

    work page 2014

  7. [7]

    Gradient descent can take exponential time to escape saddle points

    Simon S Du, Chi Jin, Jason D Lee, Michael I Jordan, Aarti Si ngh, and Barnabas Poczos. Gradient descent can take exponential time to escape saddle points. In Advances in neural information processing systems , pages 1067–1077, 2017

    work page 2017

  8. [8]

    Escaping fr om saddle points–online stochas- tic gradient for tensor decomposition

    Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping fr om saddle points–online stochas- tic gradient for tensor decomposition. In Conference on Learning Theory, pages 797–842, 2015

    work page 2015

Show all 30 references
  1. [9]

    No spurious local minima in nonconvex low rank problems: A unified geometric analysis

    Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1233–1242. JMLR. org, 2017

    2017

  2. [10]

    Matrix completion ha s no spurious local minimum

    Rong Ge, Jason D Lee, and Tengyu Ma. Matrix completion ha s no spurious local minimum. In Advances in Neural Information Processing Systems , pages 2973–2981, 2016

    2016

  3. [11]

    Newton-type methods fo r unconstrained and linearly con- strained optimization

    Philip E Gill and Walter Murray. Newton-type methods fo r unconstrained and linearly con- strained optimization. Mathematical Programming, 7(1):311–350, 1974

    1974

  4. [12]

    How to escape saddle points efficiently

    Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, an d Michael I Jordan. How to escape saddle points efficiently. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1724–1732. JMLR. org, 2017. 13

    2017

  5. [13]

    On large-batch training for deep learning: G eneralization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Noce dal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: G eneralization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016

    2016 arXiv

  6. [14]

    First-order methods almost always avoid sa ddle points

    Jason D Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I Jordan, and Benjamin Recht. First-order methods almost always avoid sa ddle points. arXiv preprint arXiv:1710.07406, 2017

    2017 arXiv

  7. [15]

    Gradient descent only converges to minimizers

    Jason D Lee, Max Simchowitz, Michael I Jordan, and Benja min Recht. Gradient descent only converges to minimizers. In Conference on learning theory , pages 1246–1257, 2016

    2016

  8. [16]

    The power of normalization: Faster evasion o f saddle points

    Kfir Y Levy. The power of normalization: Faster evasion o f saddle points. arXiv preprint arXiv:1611.04831, 2016

    2016 arXiv

  9. [17]

    On the use of directio ns of negative curvature in a modified newton method

    Jorge J Mor´ e and Danny C Sorensen. On the use of directio ns of negative curvature in a modified newton method. Mathematical Programming, 16(1):1–20, 1979

    1979

  10. [18]

    Introductory lectures on convex optimization: A basic cours e, volume 87

    Yurii Nesterov. Introductory lectures on convex optimization: A basic cours e, volume 87. Springer Science & Business Media, 2013

    2013

  11. [19]

    Cubic regularizatio n of newton method and its global performance

    Yurii Nesterov and Boris T Polyak. Cubic regularizatio n of newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006

    2006

  12. [20]

    Behavior of accel erated gradient methods near critical points of nonconvex functions

    Michael O’Neill and Stephen J Wright. Behavior of accel erated gradient methods near critical points of nonconvex functions. Mathematical Programming, pages 1–25, 2017

    2017

  13. [21]

    Gradient des cent only converges to minimizers: Non-isolated critical points and invariant regions

    Ioannis Panageas and Georgios Piliouras. Gradient des cent only converges to minimizers: Non-isolated critical points and invariant regions. arXiv preprint arXiv:1605.00405 , 2016

    2016 arXiv

  14. [22]

    On the saddle point problem for non-convex optimization

    Razvan Pascanu, Yann N Dauphin, Surya Ganguli, and Yosh ua Bengio. On the saddle point problem for non-convex optimization. arXiv preprint arXiv:1405.4604 , 2014

    2014 arXiv

  15. [23]

    Nonconvergence to unstable points in u rn models and stochastic approxima- tions

    Robin Pemantle. Nonconvergence to unstable points in u rn models and stochastic approxima- tions. The Annals of Probability , 18(2):698–712, 1990

    1990

  16. [24]

    A generic approach for escapin g saddle points

    Sashank J Reddi, Manzil Zaheer, Suvrit Sra, Barnabas Po czos, Francis Bach, Ruslan Salakhut- dinov, and Alexander J Smola. A generic approach for escapin g saddle points. arXiv preprint arXiv:1709.01434, 2017

    2017 arXiv

  17. [25]

    Complexity analy sis of second-order line-search algorithms for smooth nonconvex optimization

    Cl´ ement W Royer and Stephen J Wright. Complexity analy sis of second-order line-search algorithms for smooth nonconvex optimization. SIAM Journal on Optimization , 28(2):1448– 1477, 2018

    2018

  18. [26]

    Global stability of dynamical systems

    Michael Shub. Global stability of dynamical systems . Springer Science & Business Media, 2013

    2013

  19. [27]

    Theoretical insights into the op- timization landscape of over-parameterized shallow neura l networks

    Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the op- timization landscape of over-parameterized shallow neura l networks. IEEE Transactions on Information Theory, 65(2):742–769, 2019

    2019

  20. [28]

    Complete dictionary re covery over the sphere i: Overview and the geometric picture

    Ju Sun, Qing Qu, and John Wright. Complete dictionary re covery over the sphere i: Overview and the geometric picture. IEEE Transactions on Information Theory , 63(2):853–884, 2016. 14

    2016

  21. [29]

    Complete dictionary re covery over the sphere ii: Recovery by riemannian trust-region method

    Ju Sun, Qing Qu, and John Wright. Complete dictionary re covery over the sphere ii: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory , 63(2):885– 914, 2017

    2017

  22. [30]

    A geometric analysis of phase retrieval

    Ju Sun, Qing Qu, and John Wright. A geometric analysis of phase retrieval. Foundations of Computational Mathematics , 18(5):1131–1198, 2018. 15

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.