Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Don't Be So Positive: Negative Step Sizes in Second-Order Methods

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that allowing negative step sizes in a Wolfe line search makes ascent-direction methods globally convergent, and shows that SR1 with this line search trains neural networks effectively.

desk verdict A simple, promising trick for quasi-Newton methods, with a theory gap that is easy to patch and experiments that are too thin to trust yet. read the letter →

arxiv 2411.11224 v2 pith:BRWY33ZW submitted 2024-11-18 cs.LG math.OCstat.ML

Betty Shea , Mark Schmidt This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 90C5390C26
keywords negativestepsizessecond-orderoptimizationWolfelinesearchascentdirectionsglobalconvergencesymmetricrank-one(SR1)limited-memoryquasi-Newtonneuralnetworktraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that second-order optimization methods can be made globally convergent without discarding negative curvature information, by allowing the step size to be negative. It proves that any algorithm whose search direction is not orthogonal to the gradient is globally convergent when paired with a Wolfe line search extended to both positive and negative step sizes. The paper then demonstrates experimentally that limited-memory SR1 with this extended line search, called Wolfe±, often trains neural networks more effectively than common Hessian modification methods and competitively with Adam.

What carries the argument

The key machinery is the Wolfe± line search, a simple extension of the standard Wolfe conditions that permits negative step sizes: if the search direction is an ascent direction, a standard Wolfe search is called on $-p_k$ and the returned step is negated. The proof machinery is the Zoutendijk condition, which bounds the total decrease of $f$ by $c \sum \cos^2\theta_k \|\nabla f(x_k)\|^2 < \infty$; because the direction is never orthogonal to the gradient, this forces the gradient norms to converge to zero.

What would settle it

A concrete counterexample would be an algorithm whose search direction is never orthogonal to the gradient, whose step sizes satisfy the Wolfe conditions but are allowed to shrink to zero, and which nevertheless fails to drive the gradient norm to zero—this would show the $\epsilon > 0$ condition is genuinely needed. On the empirical side, finding a suite of neural-network benchmarks where l-SR1 with damping consistently beats l-SR1 with Wolfe± under equal iteration budgets would refute the paper's central practical claim.

Watch

Extended reading notes

Core claim

The central result is Proposition 1: for a twice-differentiable, Lipschitz-smooth objective that is bounded below, any algorithm following $x_{k+1} = x_k + \alpha_k p_k$ with step sizes satisfying $|\alpha_k| \ge \epsilon > 0$ and the Wolfe conditions is globally convergent in the sense $\|\nabla f(x_k)\| \to 0$, even when $p_k$ is an ascent direction. The proof extends the classical Zoutendijk argument to ascent directions paired with negative step sizes, showing that sufficient decrease still yields the summability of $\cos^2\theta_k \|\nabla f(x_k)\|^2$, and since $|\cos\theta_k| \ge \delta > 0$, the gradient norms must vanish. The paper further claims that SR1, which can produce indefinite Hessian approximations and hence ascent directions, becomes a competitive neural-network optimizer when negative step sizes are allowed.

Load-bearing premise

The theorem assumes step sizes are bounded away from zero, $|\alpha_k| \ge \epsilon > 0$, while satisfying the Wolfe conditions, but real line searches near a stationary point typically produce step sizes that shrink to zero, so this hypothesis may not hold in practice even though the proof appears not to need it.

Editorial extensions

If this is right

  • Second-order methods that produce ascent directions, such as SR1 with non-positive-definite Hessian approximations, can be made globally convergent without Hessian modification, trust regions, or cubic regularization.
  • Adding negative step sizes is computationally inexpensive—just a sign flip in the line search—compared with the eigenvalue decomposition required by damping.
  • In the paper's experiments, l-SR1 with Wolfe± achieved lower training error than l-SR1 with damping on most datasets and often outperformed Adam and l-BFGS.
  • The benefit of negative step sizes appears to grow with network depth, making the approach more relevant for deeper architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's proof never uses the assumption $|\alpha_k| \ge \epsilon > 0$; if that observation holds, the global convergence result likely remains true even when step sizes shrink to zero, covering practical line searches that approach a stationary point.
  • A broader principle suggested by the paper is that globalization strategies preserving curvature information—such as taking a backward step along an ascent direction—may outperform strategies that modify curvature, like damping or trust-region projection.
  • Wolfe± could be combined with other ascent-producing methods, such as saddle-free Newton or conjugate gradient variants, potentially improving their saddle-escape behavior.
  • A concrete testable extension is to apply Wolfe± to stochastic or mini-batch settings, where the search direction is a noisy estimate and the curvature information is even less reliable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the use of negative step sizes in second-order and quasi-Newton methods for non-convex optimization. Its central theoretical claim (Proposition 1, Section 3) is that an algorithm using updates x_{k+1} = x_k + α_k p_k is globally convergent whenever p_k is not orthogonal to the gradient and the step sizes α_k satisfy a Wolfe-type condition, even when p_k is an ascent direction. The proof in Appendix B adapts Nocedal and Wright's Theorem 3.2 to ascent directions paired with negative step sizes. The paper also proposes a practical line search, Wolfe± (Appendix C), and reports experiments on neural network training showing that limited-memory SR1 with negative step sizes compares favorably with damping and with optimizers such as Adam.

Significance. The central idea is simple and potentially valuable: instead of modifying the Hessian or the search direction to force descent, one can take a negative step along an ascent direction and thereby retain second-order information. If the theoretical guarantee were airtight and the implementation actually matched the theorem, the paper would make a useful contribution to the optimization-for-machine-learning literature. The proof of Proposition 1 is a clean adaptation of a standard result, and the paper is honest in pointing out that the usual globalization strategies discard negative curvature. The experimental section covers many datasets and gives a concrete baseline comparison. However, as detailed below, the stated theorem does not currently cover the proposed algorithm, and the experiments lack statistical support, so the significance is not yet established to the standard required for publication.

major comments (3)
  1. [Appendix C, Algorithm 1; equation (8)] The bridge between Proposition 1 and Algorithm 1 (Wolfe±) is broken. Proposition 1 assumes the generalized curvature condition (8), which includes the lower bound 0 ≤ ∇f(x_k + α_k p_k)^T p_k. Algorithm 1, however, obtains an ascent step by running a standard Wolfe search on q = -p_k and negating the result. The standard Wolfe curvature condition for the descent direction q only yields ∇f(x_k + α_k p_k)^T p_k ≤ c_2 ∇f(x_k)^T p_k; it does not enforce the lower bound 0. For example, take f(α) = α + α² at x = 0, p = 1, c_1 = 0.1, c_2 = 0.9. The step β = 0.6 satisfies the standard Wolfe conditions on q = -1: the Armijo condition gives f(-0.6) = -0.24 ≤ f(0) + c_1 β ∇f(0)^T q = -0.06, and the curvature condition gives -0.2 ≥ c_2 ∇f(0)^T q = -0.9. The resulting step α = -0.6 has ∇f(-0.6)^T p = -0.2 < 0, violating (8). Thus the implementation as written need not satisfy the assumptions of Proposition 1. The fix is either to restate the theorem with the one-sided curvature bound that the proof actually uses, or to modify the line search so that it enforces the lower bound; without such a change, the stated global convergence guarantee does not apply to Algorithm 1.
  2. [Section 3, Proposition 1; Appendix B] The assumption |α_k| ≥ ε > 0 is never used in the proof in Appendix B. The proof uses only α_k < 0 (for ascent directions) together with (8) and (6); the ε lower bound appears neither in inequality (11) nor in the summation argument. Moreover, such an assumption is questionable in practice: near a stationary point, a Wolfe line search will typically produce step sizes that shrink to zero, so a fixed positive lower bound may be unattainable. Since the proof suggests the ε bound is unnecessary, the proposition should either be stated without it or the proof should be modified to show where it is needed. As written, the abstract's phrase 'very general conditions' overstates the result, since the statement carries an extra condition that is both unused and potentially unsatisfiable.
  3. [Section 4, Table 1] The empirical claims rest on single runs. No error bars, no number of random seeds, and no variance information are reported for the training-error comparisons in Table 1 or Figure 2. In deep learning experiments, single-run comparisons are generally not sufficient to support a claim such as 'using negative step sizes is often more effective than common Hessian modification methods.' Please provide statistics over multiple seeds, or at least state explicitly that the results are from a single run and temper the conclusions accordingly.
minor comments (4)
  1. [Section 4.1] The sentence 'With negative step sizes, SR1 becomes the an effective optimizer' contains a typo: 'the an' should be 'an'.
  2. [References, [29]] Reference [29] is listed as 'V . Ramanmurthy and N. Duffy. L-sr1: A novel second order optimization method for deep learning' but the author name is typically spelled 'Ramanurthy'. Please verify the spelling and the publication venue.
  3. [Appendix A, equation (2)] The definition of global convergence uses 'lim inf_{k→∞} ∥∇f(x_k)∥ = 0', but the proof of Proposition 1 concludes the stronger statement 'lim_{k→∞} ∥∇f(x_k)∥ = 0'. Consider aligning the definition with the proven statement, or clarify that the lim inf version is sufficient for the paper's purposes.
  4. [Appendix B] The line 'cosθ k ∥∇f(x_k)∥² → 0' is not typeset correctly; it should read 'cos² θ_k ∥∇f(x_k)∥² → 0'. Please fix the notation for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Proposition 1 is an independent extension of the standard Zoutendijk/Wolfe convergence proof, and the empirical claims are measured experiments rather than fitted predictions.

full rationale

The paper's central theoretical claim, Proposition 1, is derived from standard assumptions (bounded-below objective, Lipschitz-continuous gradient, non-orthogonal search directions, and a Wolfe-like curvature condition) using the classical Zoutendijk-condition argument from Nocedal and Wright's Theorem 3.2. The proof in Appendix B algebraically combines the curvature condition (8) with the sufficient-decrease condition (6) to obtain a per-iteration decrease proportional to cos^2(theta_k) ||grad f(x_k)||^2, which yields the Zoutendijk condition. This is a self-contained derivation from stated assumptions, not an import of the conclusion. The assumption |alpha_k| >= epsilon > 0 is mentioned in Proposition 1 but is not used in the proof; that is an overstatement or gap in the theorem statement, not circularity. The Wolfe± line search (Algorithm 1) is an implementation proposal; whether it exactly enforces the generalized curvature condition (8) is a correctness question about the match between theorem and implementation, not a circularity. The experimental sections compare l-SR1, BFGS, GD, and Adam on training error; these are direct measurements, and no fitted parameter is renamed as a prediction. The paper contains no load-bearing self-citations: references to Nocedal and Wright, Fletcher-Reeves, Byrd-Nocedal-Schnabel, and other classical works are external. Overall, the derivation chain does not reduce to its inputs by construction, and no circular step can be exhibited with specific equations.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The convergence proof relies on standard Wolfe/Zoutendijk theory plus assumptions on the objective and the algorithm that are not guaranteed by any named method. The experiments depend on several unreported implementation choices. No new physical or mathematical entities are introduced.

free parameters (4)
  • epsilon (lower bound on |alpha_k|) = unspecified > 0
    Proposition 1 assumes every step size satisfies |alpha_k| >= epsilon > 0. This constant is never used in the proof and is not guaranteed by any line search; it weakens the claim of 'very general conditions'.
  • Wolfe line-search constants c1, c2 = not reported
    Standard parameters in conditions (6)-(8), chosen by the implementation. They are conventional but their values are not stated in the experiments.
  • maximum iterations = 50
    All experiments use a budget of 50 iterations; results are training error at this budget and may not reflect asymptotic behavior.
  • network architecture (layers, hidden units, activation) = not reported
    Table 1 does not specify the architecture, initialization, or activation for the neural networks, so the experimental outcomes cannot be reconstructed from the paper.
assumptions (4)
  • domain assumption f is twice differentiable (or C^1 with Lipschitz gradient) and bounded below on R^n
    Stated in Section 3 and used in Proposition 1 and Appendix B to apply the Lipschitz inequality and bounded-decrease argument.
  • standard math The standard Wolfe/Zoutendijk convergence theorem for descent directions with positive step sizes
    The proof in Appendix B is presented as an extension of Theorem 3.2 in [25]; the descent-direction case is taken from that theorem.
  • domain assumption Every search direction p_k satisfies |cos theta_k| >= delta > 0
    Proposition 1 assumes a uniform nonzero angle with the gradient. This is not proven for SR1 or any concrete method, so the global convergence conclusion is conditional.
  • ad hoc to paper The Wolfe line search can return steps with |alpha_k| >= epsilon > 0
    Proposition 1 assumes a positive lower bound on step magnitude. The bound is not used in the proof and may fail near stationary points where Armijo step sizes tend to zero.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Don't Be So Positive: Negative Step Sizes in Second-Order Methods." pith.science (2026). https://pith.science/paper/BRWY33ZW

@misc{pith2026241111224,
  author       = {Pith},
  title        = {Pith review of: Don't Be So Positive: Negative Step Sizes in Second-Order Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BRWY33ZW}},
  note         = {Machine review of arXiv:2411.11224}
}
read the original abstract

The value of second-order methods lies in the use of curvature information. Yet, this information is costly to extract and once obtained, valuable negative curvature information is often discarded so that the method is globally convergent. This limits the effectiveness of second-order methods in modern machine learning. In this paper, we show that second-order and second-order-like methods are promising optimizers for neural networks provided that we add one ingredient: negative step sizes. We show that under very general conditions, methods that produce ascent directions are globally convergent when combined with a Wolfe line search that allows both positive and negative step sizes. We experimentally demonstrate that using negative step sizes is often more effective than common Hessian modification methods.

Figures

Figures reproduced from arXiv: 2411.11224 by the authors.

Figure 1
Figure 1. In this two-dimensional non-convex minimization problem, the negative of the Newton’s direction (green) is a better search direction than the gradient descent direction (blue), Newton’s direction (red) or any non-negative combination of both. 1.1. Contribution This paper examines the largely unexplored role of negative step sizes in optimization and shows that taking a backward step is a computationally inexpensive … view at source ↗
Figure 2
Figure 2. Using negative step sizes in training neural networks. The plots show training error by iteration for neural networks with 1-, 2- and 3- hidden layers on the heart dataset. We compare full QN methods BFGS (yellow) and SR1 (blue and green) with GD (red) and Adam (black). SR1 is non-convergent with positive-only step sizes (blue) but outperforms when step sizes are allowed to be negative (green). The outperformance ap… view at source ↗
Figure 3
Figure 3. Step sizes (top row) and cosine similarity (bottom row) of different optimizers for neural networks with 1, 2 and 3 hidden layers. SR1 often produces ascent directions and this is dealt with effectively by using negative step sizes. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Negative Stepsizes Make Gradient-Descent-Ascent Converge

    math.OC 2025-05 accept novelty 8.0 of 10

    GDA converges on bilinear, quadratic, and convex-concave min-max problems using time-varying, asymmetric, periodically negative step sizes, at rates matching optimal first-order methods.

Reference graph

Works this paper leans on

35 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    S. I. Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251–276, 1998

    work page 1998

  2. [2]

    H. Y .. Benson and D. F. Shanno. Cubic regularization in symmetric rank-1 quasi-Newton methods. Math. Prog. Comp., 10:457–486, 2018

    work page 2018

  3. [3]

    C. G. Broyden. Quasi-Newton methods and their application to function minimisation. Math- ematics of Computation, 21(99):368–381, 1967

    work page 1967

  4. [4]

    R.H. Bryd, J. Nocedal, and R.B. Schnabel. Representations of quasi-Newton matrices and their use in limited memory methods. Mathematical Programming, 63:129–156, 1994

    work page 1994

  5. [5]

    Byrd, R.B

    R.H. Byrd, R.B. Schnabel, and G.A. Schultz. Approximate solution of the trust regions prob- lem by minimization over two-dimensional subspaces. Mathematical Programming, 40:247– 263, 1988

    work page 1988

  6. [6]

    Choromanska, M

    A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y . LeCun. The loss surfaces of multilayer networks. AISTATS, 38:192–204, 2015

    work page 2015

  7. [7]

    A. Conn, N. Gould, and P. Toint. Trust-region methods. MPS/SIAM Series on Optimization. SIAM, 2000. ISBN 0-89871-460-5

    work page 2000

  8. [8]

    Conn, N.I.M

    A.R. Conn, N.I.M. Gould, and P.L. Toint. Convergence of quasi-Newton matrices generated by the symmetric rank one update. Mathematical Programming, 50:177–195, 1991

    work page 1991

Show all 35 references
  1. [9]

    Y . Dai. Further insight into the convergence of the fletcher-reeves method. Sci. China Ser. A-Math, 42:905–916, 1999

    1999

  2. [10]

    Dauphin, R

    Y . Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y . Bengio. Identify- ing and attacking the saddle point problem in high-dimensional non-convex optimization. arXiv:1406.2572, 2014

    2014 arXiv

  3. [11]

    Fletcher and C

    R. Fletcher and C. M. Reeves. Function minimization by conjugate gradients. The computer journal, 7(2):149–154, 1964

    1964

  4. [12]

    Goldfarb

    D. Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of Computation, 24(109):23–26, 1970. 6 NEGATIVE STEP SIZES

    1970

  5. [13]

    Kamzolov, K

    D. Kamzolov, K. Ziu, A. Agafonov, and Tak´aˇc. Cubic regularization is the key! the first accel- erated quasi-Newton method with a global convergence rate of o(k−1) for convex functions. arXiv:2302.04987, 2023

    2023 arXiv

  6. [14]

    H. F. Khalfan, R. H. Byrd, and R. B. Schnabel. A theoretical and experimental study of the symmetric rank-one update. SIAM Journal on Optimization, 3(1):1–24, 1993

    1993

  7. [15]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.arXiv:1412.6980, 2014

    2014 arXiv

  8. [16]

    Levenberg

    K. Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2:164–168, 1944

    1944

  9. [17]

    D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45:503–528, 1989

    1989

  10. [18]

    Malitsky and K

    Y . Malitsky and K. Mishchenko. Adaptive gradient descent without descent. In Hal Daum´e III and Aarti Singh, editors,Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research , pages 6702–6712. PMLR, 13–18 Jul 2020

    2020

  11. [19]

    Marquardt

    D.W. Marquardt. An algorithm for least squares estimation of non-linear parameters. SIAM Journal, 11:431–441, 1963

    1963

  12. [20]

    J. Martens. Deep learning via hessian-free optimization. In ICML, volume 27, 2010

    2010

  13. [21]

    Martens and R

    J. Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In ICML, 2015

    2015

  14. [22]

    J. J. Mor ´e and D. C. Sorensen. On the use of directions of negative curvature in a modified newton method. Mathematical Programming, 16:1–20, 1979

    1979

  15. [23]

    Mor ´e and D

    Jorge J. Mor ´e and D. C. Sorensen. Computing a trust region step. SIAM Journal on Scientific and Statistical Computing, 4(3):553–572, 1983

    1983

  16. [24]

    Nesterov and B

    Y . Nesterov and B. Polyak. Cubic regularization of Newton method and its global performance. Math. Program., 108:177–205, 2006

    2006

  17. [25]

    Nocedal and S

    J. Nocedal and S. J. Wright. Numerical Optimization, 2nd Ed. Springer, 2006

    2006

  18. [26]

    Pascanu, Ganguli S

    R. Pascanu, Ganguli S. Dauphin, Y . N., and Y . Bengio. On the saddle point problem for non-convex optimization. arXiv:1405.4604, 2014

    2014 arXiv

  19. [27]

    Paternain, A

    S. Paternain, A. Mokhtari, and A. Ribeiro. A Newton-based method for nonconvex optimiza- tion with fast evasion of saddle points. SIAM Journal on Optimization, 29(1):343–368, 2019

    2019

  20. [28]

    M.J.D. Powell. A new algorithm for unconstrained optimization. Nonlinear Programming, pages 31–66, 1970

    1970

  21. [29]

    Ramanmurthy and N

    V . Ramanmurthy and N. Duffy. L-sr1: A novel second order optimization method for deep learning, 2016. 7 NEGATIVE STEP SIZES

    2016

  22. [30]

    Schultz, R.B

    G.A. Schultz, R.B. Schnabel, and R.H. Byrd. A family of trust-region-based algorithms for unconstrained minimization with strong global convergence properties. SIAM Journal on Nu- merical Analysis, 22:47–67, 1985

    1985

  23. [31]

    D. F. Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of Computation, 24(111):647–656, 1970

    1970

  24. [32]

    D. C. Sorensen. Newton’s method with a model trust region modification. SIAM Journal on Numerical Analysis, 19(2):409–426, 1982

    1982

  25. [33]

    Steihaug

    T. Steihaug. The conjugate gradient method and trust regions in large scale optimization.SIAM Journal on Numerical Analysis, 20(3), 1983

    1983

  26. [34]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5 - rmsprop: Divident the gradient by a running average of its recent magnitude, 2012

    2012

  27. [35]

    P. Wolfe. Convergence conditions for ascent methods. SIAM review, 11(2):226–235, 1969. 8 NEGATIVE STEP SIZES Appendix A. Notation and further background details We say an algorithm is globally convergent if it produces a sequence of gradients that converge to zero, or lim inf ...

    1969

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.