Pith. sign in

REVIEW 1 major objections 6 minor 55 references

An Enhanced Levenberg--Marquardt Method via Gram Reduction

T0 review · 1 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that reusing the Gram matrix every m iterations lowers the Levenberg–Marquardt cost to O(d^3 ε^{-1}+d^2 ε^{-2}) while preserving global and local superlinear convergence.

desk verdict The global complexity result is a genuine improvement and mostly provable; the local superlinear theorem has a concrete proof gap and should not be cited in its current form. read the letter →

arxiv 2412.08561 v1 pith:E6Y4NQ23 submitted 2024-12-11 math.OC

Chengchang Liu , Luo Luo , John C.S. Lui This is my paper · ORCID
classification math.OC MSC 65H1090C5349M15
keywords nonlinearequationsLevenberg–MarquardtmethodGrammatrixreuseglobalconvergencelocalsuperlinearcubic-growthconditionnonconvexleastsquares
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Gram-Reduced Levenberg–Marquardt (GRLM), a method for solving nonlinear systems $F(x)=0$ that recomputes the Gram matrix $G(z_t)=J(z_t)^\top J(z_t)$ only once every $m$ iterations and reuses it in between. It claims that, under a cubic-growth condition on $F$, this reuse preserves global convergence and yields a total cost of $O(d^3\epsilon^{-1}+d^2\epsilon^{-2})$ flops to reach an $\epsilon$-stationary point, strictly better than the $O(d^3\epsilon^{-2})$ of existing Levenberg–Marquardt methods. It also claims local superlinear convergence near a solution with non-degenerate Jacobian, with no line search or subproblem solver. The paper matters because it decouples the expensive Gram computation from the iteration count, giving LM methods a per-step cost comparable to quasi-Newton methods while keeping global guarantees.

What carries the argument

The Gram-reused update $x_{t+1}=x_t-(G(z_t)+\lambda_t I)^{-1}J(x_t)^\top F(x_t)$, where $z_t=x_{m\lfloor t/m\rfloor}$ is the snapshot and $\lambda_t=\sqrt{c\|J(x_t)^\top F(x_t)\|}$. The analysis couples a two-sided step-size bound $\lambda_t^2/(c(L_1^2+\lambda_t))\le r_t\le\lambda_t/c$ with a block-wise descent lemma that accumulates $\sum r_t^2\lambda_t/6$ over each $m$-step block; the cubic-growth condition forces the residual norm to drop across blocks. The local analysis uses the non-degeneracy $\sigma_{\min}(J(x^*))=\mu>0$ to bound $(G(z_t)+\lambda_t I)^{-1}$ and feed a sequence recurrence from lazy-Hessian Newton theory.

What would settle it

Construct a smooth $F$ with known $L_1,L_2,M$ that satisfies Assumptions 2 and 7 (for instance a mildly nonlinear trigonometric map), simulate Algorithm 1 with $m=\Theta(\epsilon^{-1})$, and check whether the decrease $\|F(x_{km})\|^2-\|F(x_{(k+1)m})\|^2$ is always at least $\sum_{t=km}^{(k+1)m-1} r_t^2\lambda_t/6$; if not, Lemma 8 and the complexity claim collapse.

Watch

Extended reading notes

Core claim

The load-bearing claim is Corollary 11: running Algorithm 1 with $m=\Theta(\epsilon^{-1})$ finds a point with $\|J(x)^\top F(x)\|\le\epsilon$ in $O(d^3\epsilon^{-1}+d^2\epsilon^{-2})$ total flops, improving on the $O(d^3\epsilon^{-2})$ of prior LM analyses. The algorithm alternates between computing the SVD of the Jacobian at snapshot points $z_t = x_{m\lfloor t/m\rfloor}$ and taking cheap $O(d^2)$ steps that invert the reused matrix $(G(z_t)+\lambda_t I)^{-1}$ applied to $J(x_t)^\top F(x_t)$, with $\lambda_t=\sqrt{c\|J(x_t)^\top F(x_t)\|}$. Under the additional assumption of a non-degenerate Jacobian at the solution, the iterates converge superlinearly with an explicit per-iteration rate stated in Theorem 15.

Load-bearing premise

The global complexity result holds only under the cubic-growth condition $\|F(y)\|^2\le\|F(x)+J(x)(y-x)\|^2+M\|y-x\|^3$ for all $x,y$, which is not a consequence of a Lipschitz-continuous Jacobian; the local superlinear rate additionally depends on a sequence recurrence that the paper cites rather than proves.

Editorial extensions

If this is right

  • Choosing $m=\Theta(\epsilon^{-1})$ gives total flops $O(d^3\epsilon^{-1}+d^2\epsilon^{-2})$, strictly better than the $O(d^3\epsilon^{-2})$ bound for existing LM methods.
  • After a snapshot SVD, each iteration costs $O(d^2)$ flops, matching the per-iteration cost of quasi-Newton methods while retaining global convergence.
  • The method requires no line search and no trust-region subproblem, so the implementation reduces to one reused matrix inverse per step.
  • For $m=1$, GRLM recovers Mishchenko's LM and improves its iteration bound from $O(\epsilon^{-2.5}\log(1/\epsilon))$ to $O(\epsilon^{-2.5})$.
  • Under non-degenerate Jacobian at the solution, the iterates converge superlinearly with the explicit rate $(1/2)^{2(1+(1+m/2)\pi(t))(1+(t\bmod m)/2)}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cubic-growth condition is not implied by Lipschitz continuity of the Jacobian; for residuals whose quadratic model error grows faster than $\|y-x\|^3$, the stated global complexity has no support, and the method may still work in practice but without this guarantee.
  • The reuse idea is orthogonal to the choice of regularization; combining it with sketched or randomized Gram approximations could lower the $O(d^3)$ snapshot SVD cost and push the per-block cost toward $O(d^2)$, which would be a natural next step.
  • The local superlinear rate relies on a sequence lemma imported from lazy-Hessian theory; a self-contained proof of that recurrence would make the local result independent of that framework and clarify whether the factor $(1+m/2)$ is tight.
  • The optimal worst-case choice $m=\Theta(\epsilon^{-1})$ suggests an adaptive schedule in which $m$ grows as the iterate approaches stationarity could reduce the practical number of snapshots while preserving the asymptotic bound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes the Gram-Reduced Levenberg–Marquardt (GRLM) method for solving square nonlinear systems F(x)=0. The method reuses the Gram matrix J(x_pi(t))^T J(x_pi(t)) at snapshot points for m iterations and sets the regularization parameter lambda_t = sqrt(c ||J(x_t)^T F(x_t)||), requiring no line search or subproblem solves. Under a Lipschitz Jacobian assumption (Assumption 2) and a cubic-growth condition (Assumption 7), the authors prove an iteration complexity of O(m^2 + m^{-0.5} epsilon^{-2.5}) to reach an epsilon-stationary point, which with m = Theta(epsilon^{-1}) gives a total flop count of O(d^3 epsilon^{-1} + d^2 epsilon^{-2}), improving on the O(d^3 epsilon^{-2}) of prior LM methods. Under a non-degenerate Jacobian at the solution (Assumption 12), the paper claims an explicit local superlinear rate. Experiments on the Chandrasekhar H-equation and non-convex logistic regression compare GRLM with gradient descent and the regularized LM of Mishchenko.

Significance. If the proofs are repaired, the global complexity result is a genuine advance: it is, to my knowledge, the first LM-type method that avoids both line search and subproblem solves while achieving a total flop count better than O(d^3 epsilon^{-2}), and the flop analysis with reused Gram matrices is clean and plausible. The local superlinear rate is a desirable property that would also extend Mishchenko's method. The paper is commendable for including a complete global proof, the AM-GM-based tuning of m, and experiments on real-world datasets. However, the local convergence theorem, which is advertised in the abstract and in Table 1, is not proven as written; the recurrence used to invoke the external sequence lemma does not match what Lemma 14 actually provides, and the initial condition does not satisfy the lemma's hypothesis.

major comments (1)
  1. [Section 4.2, proof of Theorem 15 (Appendix H)] The proof of Theorem 15 does not establish the hypotheses of Lemma 18. Substituting s_t = eta e_t with eta = 2(alpha1 + alpha2^2) into inequality (18) yields s_{t+1} <= (alpha1/eta) s_t^2 + (alpha2/sqrt(eta)) s_t^{1.5} + (2 alpha1/eta) s_{pi(t)} s_t. The claimed recurrence s_{t+1} <= 1/2(s_t^2 + s_t^{1.5}) + s_{pi(t)} s_t requires alpha2/sqrt(eta) <= 1/2, i.e., alpha2^2 <= alpha1, which is not assumed and fails for admissible parameters (for example, L1 = L2 = mu = 1 and c = 1 give alpha1 = 1, alpha2 = 2, and alpha2/sqrt(eta) = 2/sqrt(10) approximately 0.632 > 1/2). Moreover, condition (19) gives s0 = eta ||x0 - x*|| <= eta/(32(alpha1 + alpha2^2)) = 1/16, whereas Lemma 18 requires s0 <= 1/24; the proof asserts s0 <= 1/24 without justification. Since the superlinear-rate conclusion is derived solely from Lemma 18, Theorem 15 is currently unsupported and needs a repaired argument.
minor comments (6)
  1. [Proof of Theorem 9, inequality (27)] In the I1 sum, the denominator should be L1^4, not L1^2: when lambda_t < L1^2, (L1^2 + lambda_t)^2 < 4 L1^4, so lambda_t^5/(6c^2(L1^2+lambda_t)^2) >= lambda_t^5/(24 c^2 L1^4). The printed L1^2 makes the inequality dimensionally inconsistent. The asymptotic iteration bound is unaffected after the constant is corrected.
  2. [Proof of Lemma 8 (Appendix E)] The inner sum in the bound for ||x_t - z_t|| should be sum_{i=0}^{t-1} r_i, not sum_{i=0}^t r_i. With the printed sum_{i=0}^t r_i, the subsequent application of Lemma 17 to t = 0,...,m-1 is invalid; for example, with m = 2 and r_0 = r_1 = 1, sum_{t=0}^1 (sum_{i=0}^t r_i)^3 = 9 > (2^3/3)(1+1) = 16/3. The intended inequality holds with the corrected upper index t-1.
  3. [Corollary 11, flop minimization] The AM-GM lower bound for the third and fourth summands should be d^{2+0.75 alpha} epsilon^{-1.25-0.75 beta}, not d^{2+1.25 alpha} epsilon^{-1.25-0.75 beta}. The final conclusion m = Theta(epsilon^{-1}) and the resulting flop count are unaffected by this typo.
  4. [Proof of Theorem 15, induction step] In the induction, z_t = x_{pi(t)} belongs to {x_0,...,x_t}, not {x_1,...,x_t} as stated; the base case x_0 is needed when t < m.
  5. [Algorithm 1] In line 4, the expression ||J(x_t)^T \nabla f(x_t)|| should be ||J(x_t)^T F(x_t)|| (or ||\nabla phi(x_t)|| with phi = 1/2||F||^2); the symbol f is otherwise undefined.
  6. [Throughout] There are several spelling errors in method names (e.g., 'Levernberg' and 'Mardquardt' in Table 1 and Figure captions) and a misreferenced equation number in the proof of Lemma 14 where the bound on lambda_t is cited as (12); these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: global and local claims are derived from stated assumptions and external lemmas, not from their own conclusions.

full rationale

Walking the derivation chain: Algorithm 1 defines lambda_t = sqrt(c ||J(x_t)^T F(x_t)||). Lemma 5 bounds the step length r_t above and below by lambda_t; Lemma 6 bounds the stale-Gram error by ||z_t - x_t||; Lemma 8, using Assumption 7 and c = max{4 L1 L2 m, M}, gives block-wise descent on ||F(x_{km})||^2; Theorem 9 partitions iterations by the size of lambda_t and converts the descent sum into min ||J(x_t)^T F(x_t)|| <= epsilon. None of these steps posits the target stationarity bound as an input; the target quantity also appears in the step-size definition, but the theorem is a genuine inequality obtained from the descent sum, not a tautology. Corollary 11 is algebra: T = O(m^2 + m^{-0.5} eps^{-2.5}), K = T/m, flops = O(d^3 K + d^2 T), and m = Theta(eps^{-1}) gives O(d^3 eps^{-1} + d^2 eps^{-2}). For the local part, Lemma 14 is an in-line contraction bound from Lipschitz continuity and non-degeneracy; Theorem 15 rescales e_t to s_t = eta ||x_t - x*|| and invokes the external sequence lemma [14, Theorem D.3]. The only author-overlap citation used in the proof chain is Proposition 13, quoted as '([31, Proposition 2.3])', a standard singular-value lower bound under Assumptions 2 and 12; it does not assume the paper's complexity or rate conclusions, so under the stated rules it is independent evidence and does not raise the circularity score. I note a non-circular rigor problem in Appendix H: the proof writes 'Since it holds s0 <= 1/24' from (19), but s0 = eta ||x0 - x*|| <= 2(alpha1 + alpha2^2)/(32(alpha1 + alpha2^2)) = 1/16, and the middle coefficient alpha2/sqrt(eta) need not satisfy <= 1/2; the local superlinear proof is therefore not established as written. This is a correctness gap, not a circular reduction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The central claim depends on two hand-chosen hyperparameters (c and m) and three assumptions on F, plus one external sequence lemma whose hypotheses are not fully checked.

free parameters (2)
  • c (regularization scaling) = tuned from {1, 10, 100, 1000} in experiments; theory requires c >= max{4 L1 L2 m, M}
    Controls the regularization magnitude lambda_t = sqrt(c ||J(x_t)^T F(x_t)||). The convergence proofs only hold for c sufficiently large, so in practice it is a tuned hyperparameter.
  • m (snapshot period) = m = 50 or 100 in main experiments; m = Theta(epsilon^{-1}) in theory
    Determines how often the Gram matrix is refreshed. The total cost is minimized at m = Theta(epsilon^{-1}), but experiments choose it by hand.
assumptions (5)
  • domain assumption Assumption 2: Jacobian J(x) is bounded by L1 and Lipschitz with constant L2.
    Used in Propositions 3 and 4 and in bounding the stale Gram error in Lemma 6. Standard for nonlinear equations analysis.
  • domain assumption Assumption 7: cubic-growth condition ||F(y)||^2 <= ||F(x)+J(x)(y-x)||^2 + M||y-x||^3 for all x,y.
    This is the key global convergence premise; it is not implied by Assumption 2 and restricts the class of admissible systems.
  • domain assumption Assumption 12: there is a solution x* with non-degenerate Jacobian, sigma_min(J(x*)) = mu > 0.
    Required for the local superlinear result, guaranteeing G(x) is positive definite near x*.
  • standard math External sequence lemma (Doikov et al. Theorem D.3): if s0 <= 1/24 and s_{t+1} <= 1/2(s_t^2 + s_t^{1.5}) + s_{pi(t)}s_t, then s_t <= (1/2)^{2(1+(1+m/2)pi(t))(1+(t%m)/2)}.
    The local rate in Theorem 15 is exactly this lemma applied to s_t = eta||x_t - x*||. As written, the hypotheses are not verified because the middle coefficient can exceed 1/2.
  • standard math Proposition 13: near a non-degenerate solution, sigma_min(J(x)) >= mu/sqrt(2) and G(x) >= (mu^2/2) I.
    Self-cited from the authors' [31] but elementary; it is used to bound (G(z_t)+lambda_t I)^{-1} in Lemma 14.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Enhanced Levenberg--Marquardt Method via Gram Reduction." pith.science (2026). https://pith.science/paper/E6Y4NQ23

@misc{pith2026241208561,
  author       = {Pith},
  title        = {Pith review of: An Enhanced Levenberg--Marquardt Method via Gram Reduction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6Y4NQ23}},
  note         = {Machine review of arXiv:2412.08561}
}
abstract

This paper studied the problem of solving the system of nonlinear equations ${\bf F}({\bf x})={\bf 0}$, where ${\bf F}:{\mathbb R}^{d}\to{\mathbb R}^d$. We propose Gram-Reduced Levenberg--Marquardt method which updates the Gram matrix ${\bf J}(\cdot)^\top{\bf J}(\cdot)$ in every $m$ iterations, where ${\bf J}(\cdot)$ is the Jacobian of ${\bf F}(\cdot)$. Our method has a global convergence guarantee without relying on any step of line-search or solving sub-problems. We prove our method takes at most $\mathcal{O}(m^2+m^{-0.5}\epsilon^{-2.5})$ iterations to find an $\epsilon$-stationary point of $\frac{1}{2}\|{\bf F}(\cdot)\|^2$, which leads to overall computation cost of $\mathcal{O}(d^3\epsilon^{-1}+d^2\epsilon^{-2})$ by taking $m=\Theta(\epsilon^{-1})$. Our results are strictly better than the cost of $\mathcal{O}(d^3\epsilon^{-2})$ for existing Levenberg--Marquardt methods. We also show the proposed method enjoys local superlinear convergence rate under the non-degenerate assumption. We provide experiments on real-world applications in scientific computing and machine learning to validate the efficiency of the proposed methods.

Figures

Figures reproduced from arXiv: 2412.08561 by the authors.

Figure 1
Figure 1. We demonstrate Jacobian-vector products computing times (#JV) and CPU time (second) vs. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. We demonstrate the iteration numbers (iteration) and CPU time (second) vs. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. We demonstrate Jacobian-vector products computing times (#JV) and CPU time (second) vs. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 49 canonical work pages

  1. [1]

    Hu, and Tianyi Lin

    Ilan Adler, Zhiyue T. Hu, and Tianyi Lin. New proximal newton-type methods for convex optimization. In CDC, 2020

    work page 2020

  2. [2]

    Penalized likelihood regression for generalized linear models with non-quadratic penalties

    Anestis Antoniadis, Ir` ene Gijbels, and Mila Nikolova. Penalized likelihood regression for generalized linear models with non-quadratic penalties. Annals of the Institute of Statistical Mathematics , 63:585–615, 2011

    work page 2011

  3. [3]

    A jacobian-free newton–krylov method for thermalhy- draulics simulations

    A Ashrafizadeh, CB Devaud, and NU Aydemir. A jacobian-free newton–krylov method for thermalhy- draulics simulations. International Journal for Numerical Methods in Fluids , 77(10):590–615, 2015

    work page 2015

  4. [4]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  5. [5]

    A Newton-Raphson method for the solution of systems of equations

    Adi Ben-Israel. A Newton-Raphson method for the solution of systems of equations. Journal of Mathematical analysis and applications , 15(2):243–252, 1966

    work page 1966

  6. [6]

    Convergence and complexity analysis of a Levenberg–Marquardt algorithm for inverse problems

    El Houcine Bergou, Youssef Diouane, and Vyacheslav Kungurtsev. Convergence and complexity analysis of a Levenberg–Marquardt algorithm for inverse problems. Journal of Optimization Theory and Applications, 185:927–944, 2020

    work page 2020

  7. [7]

    Fast and robust stability region estimation for nonlinear dynamical systems

    Eloıse Berthier, Justin Carpentier, and Francis Bach. Fast and robust stability region estimation for nonlinear dynamical systems. In 2021 European Control Conference (ECC) , pages 1412–1419. IEEE, 2021

    work page 2021

  8. [8]

    Practical Gauss-Newton optimisation for deep learning

    Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton optimisation for deep learning. In International Conference on Machine Learning , pages 557–565. PMLR, 2017

    work page 2017

Show all 55 references
  1. [9]

    Radiative transfer

    Subrahmanyan Chandrasekhar. Radiative transfer. Courier Corporation, 1960

    1960

  2. [10]

    LIBSVM: A library for support vector machines.ACM Transactions on Intelligent Systems and Technology , 2:27:1–27:27, 2011

    Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines.ACM Transactions on Intelligent Systems and Technology , 2:27:1–27:27, 2011. Software and datasets available at http: //www.csie.ntu.edu.tw/∼cjlin/libsvm

    2011

  3. [11]

    Unified convergence theory of stochastic and variance-reduced cubic newton methods

    El Mahdi Chayti, Nikita Doikov, and Martin Jaggi. Unified convergence theory of stochastic and variance-reduced cubic newton methods. arXiv preprint arXiv:2302.11962 , 2023

    2023

  4. [12]

    Second-order min-max optimization with lazy hessians

    Lesi Chen, Chengchang Liu, and Jingzhao Zhang. Second-order min-max optimization with lazy hessians. arXiv preprint arXiv:2410.09568 , 2024

    2024 arXiv

  5. [13]

    Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions

    Alexandre D´ efossez and Francis Bach. Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling distributions. In Artificial Intelligence and Statistics , pages 205–213. PMLR, 2015

    2015

  6. [14]

    Second-order optimization with lazy Hessians

    Nikita Doikov, El Mahdi Chayti, and Martin Jaggi. Second-order optimization with lazy Hessians. In International Conference on Machine Learning , pages 8138–8161. PMLR, 2023

    2023

  7. [15]

    Super-universal regularized Newton method

    Nikita Doikov, Konstantin Mishchenko, and Yurii Nesterov. Super-universal regularized Newton method. SIAM Journal on Optimization , 34(1):27–56, 2024

    2024

  8. [16]

    Fed-sophia: A communication-efficient second-order federated learning algorithm

    Ahmed Elbakary, Chaouki Ben Issaid, Mohammad Shehab, Karim Seddik, Tamer ElBatt, and Mehdi Bennis. Fed-sophia: A communication-efficient second-order federated learning algorithm. arXiv preprint arXiv:2406.06655, 2024

    2024 arXiv

  9. [17]

    A shamanskii-like levenberg-marquardt method for nonlinear equations

    Jinyan Fan. A shamanskii-like levenberg-marquardt method for nonlinear equations. Computational Optimization and Applications , 56(1):63–80, 2013

    2013

  10. [18]

    On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption

    Jinyan Fan and Ya-Xiang Yuan. On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing, 74:23–39, 2005. 10

    2005

  11. [19]

    Frehse and A

    J. Frehse and A. Bensoussan. Nonlinear elliptic systems in stochastic game theory. Journal f¨ ur die reine und angewandte Mathematik , 350:23–67, 1984

    1984

  12. [20]

    Global complexity bound of the inexact Levenberg–Marquardt method

    Jian-Chao Huang and Jin-Yan Fan. Global complexity bound of the inexact Levenberg–Marquardt method. Journal of the Operations Research Society of China , 6:417–428, 2018

    2018

  13. [21]

    Carl T. Kelley. Iterative methods for linear and nonlinear equations . SIAM, 1995

    1995

  14. [22]

    Approximate methods for the solution of the chandrasekhar h-equation

    CT Kelley. Approximate methods for the solution of the chandrasekhar h-equation. Journal of Mathematical Physics, 23(11):2097–2100, 1982

    1982

  15. [23]

    Jacobian-free newton–krylov methods: a survey of approaches and applications

    Dana A Knoll and David E Keyes. Jacobian-free newton–krylov methods: a survey of approaches and applications. Journal of Computational Physics , 193(2):357–397, 2004

    2004

  16. [24]

    Global convergence technique for the newton method with periodic hessian evaluation

    Francesco Lampariello and Marco Sciandrone. Global convergence technique for the newton method with periodic hessian evaluation. Journal of optimization theory and applications , 111:341–358, 2001

    2001

  17. [25]

    A new approach to the h-equation of chandrasekhar

    Richard W Leggett. A new approach to the h-equation of chandrasekhar. SIAM Journal on Mathematical Analysis, 7(4):542–550, 1976

    1976

  18. [26]

    A method for the solution of certain non-linear problems in least squares

    Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics , 2(2):164–168, 1944

    1944

  19. [27]

    Explicit superlinear convergence rates of Broyden’s methods in nonlinear equations

    Dachao Lin, Haishan Ye, and Zhihua Zhang. Explicit superlinear convergence rates of Broyden’s methods in nonlinear equations. arXiv preprint arXiv:2109.01974 , 2021

    2021 arXiv

  20. [28]

    Explicit convergence rates of greedy and random quasi- newton methods

    Dachao Lin, Haishan Ye, and Zhihua Zhang. Explicit convergence rates of greedy and random quasi- newton methods. Journal of Machine Learning Research , 23(162):1–40, 2022

    2022

  21. [29]

    Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

    Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59:85–116, 2022

    2022

  22. [30]

    Quasi-Newton methods for saddle point problems

    Chengchang Liu and Luo Luo. Quasi-Newton methods for saddle point problems. Advances in Neural Information Processing Systems, 35:3975–3987, 2022

    2022

  23. [31]

    Chengchang Liu, Cheng Chen, Luo Luo, and John C.S. Lui. Block Broyden’s methods for solving nonlinear equations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023

    2023

  24. [32]

    Communication efficient distributed newton method with fast convergence rates

    Chengchang Liu, Lesi Chen, Luo Luo, and John CS Lui. Communication efficient distributed newton method with fast convergence rates. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1406–1416, 2023

    2023

  25. [33]

    Sophia: A scalable stochastic second- order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second- order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342 , 2023

    2023 arXiv

  26. [34]

    Marquardt

    Donald W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics , 11(2):431–441, 1963

    1963

  27. [35]

    Regularized Newton method with global O(1/k2) convergence

    Konstantin Mishchenko. Regularized Newton method with global O(1/k2) convergence. SIAM Journal on Optimization , 33(3):1440–1462, 2023. doi: 10.1137/22M1488752

    2023 doi

  28. [36]

    Nesterov

    Yu. Nesterov. Modified Gauss–Newton scheme with worst case guarantees for global performance. Optimisation methods and software , 22(3):469–483, 2007

    2007

  29. [37]

    Cubic regularization of Newton method and its global performance

    Yurii Nesterov. Cubic regularization of Newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006

    2006

  30. [38]

    Lectures on convex optimization , volume 137

    Yurii Nesterov. Lectures on convex optimization , volume 137. Springer, 2018

    2018

  31. [39]

    Numerical optimization

    Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999. 11

    1999

  32. [40]

    Mojtaba Nourian and Peter E. Caines. ϵ-Nash mean field game theory for nonlinear stochastic dynamical systems with major and minor agents. SIAM Journal on Control and Optimization , 51(4):3302–3331, 2013

    2013

  33. [41]

    Greedy quasi-newton methods with explicit superlinear conver- gence

    Anton Rodomanov and Yurii Nesterov. Greedy quasi-newton methods with explicit superlinear conver- gence. SIAM Journal on Optimization , 31(1):785–811, 2021

    2021

  34. [42]

    A modification of newton’s method

    VE Shamanskii. A modification of newton’s method. Ukrainian Mathematical Journal , 19(1):118–122, 1967

    1967

  35. [43]

    Stochastic Gauss-Newton algorithms for nonconvex compositional optimization

    Quoc Tran-Dinh, Nhan Pham, and Lam Nguyen. Stochastic Gauss-Newton algorithms for nonconvex compositional optimization. In International Conference on Machine Learning , pages 9572–9582. PMLR, 2020

    2020

  36. [44]

    Yannick Tr´ emolet. Model-error estimation in 4D-Var.Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography , 133(626): 1267–1280, 2007

    2007

  37. [45]

    On a global complexity bound of the Levenberg-Marquardt method

    Kenji Ueda and Nobuo Yamashita. On a global complexity bound of the Levenberg-Marquardt method. Journal of optimization theory and applications , 147:443–453, 2010

    2010

  38. [46]

    Further insight into the shamanskii modification of newton method

    Chang-yu Wang, Yuan-yuan Chen, and Shou-qiang Du. Further insight into the shamanskii modification of newton method. Applied mathematics and computation , 180(1):46–52, 2006

    2006

  39. [47]

    On the rate of convergence of the Levenberg-Marquardt method

    Nobuo Yamashita and Masao Fukushima. On the rate of convergence of the Levenberg-Marquardt method. In Topics in Numerical Analysis: With Special Emphasis on Nonlinear Problems , pages 239–249. Springer, 2001

    2001

  40. [48]

    Greedy and random Broyden’s methods with explicit superlinear convergence rates in nonlinear equations

    Haishan Ye, Dachao Lin, and Zhihua Zhang. Greedy and random Broyden’s methods with explicit superlinear convergence rates in nonlinear equations. arXiv preprint arXiv:2110.08572 , 2021

    2021 arXiv

  41. [49]

    Rui Yuan, Alessandro Lazaric, and Robert M. Gower. Sketched Newton–Raphson. SIAM Journal on Optimization, 32(3):1555–1583, 2022

    2022

  42. [50]

    Trust region algorithms for nonlinear equations

    Ya-Xiang Yuan. Trust region algorithms for nonlinear equations . Citeseer, 1994

    1994

  43. [51]

    Subspace methods for large scale nonlinear equations and nonlinear least squares

    Ya-Xiang Yuan. Subspace methods for large scale nonlinear equations and nonlinear least squares. Optimization and Engineering , 10(2):207–218, 2009

    2009

  44. [52]

    Recent advances in numerical methods for nonlinear equations and nonlinear least squares

    Ya-Xiang Yuan. Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numerical algebra, control & optimization , 1(1):15, 2011

    2011

  45. [53]

    Global complexity bound of the Levenberg–Marquardt method

    Ruixue Zhao and Jinyan Fan. Global complexity bound of the Levenberg–Marquardt method. Optimiza- tion Methods and Software , 31(4):805–814, 2016

    2016

  46. [54]

    Incremental Gauss–Newton methods with superlinear convergence rates

    Zhiling Zhou, Zhuanghua Liu, Chengchang Liu, and Luo Luo. Incremental Gauss–Newton methods with superlinear convergence rates. arXiv preprint arXiv:2407.03195 , 2024. A Useful Results for Positive Sequences We provide useful results for positive sequences which are used in our...

    2024 arXiv

  47. [55]

    Since it holds s0 (19) ≤ 1/24, we achieve st ≤ 1 2 2(1+(1+m/2)π(t))(1+(t%m)/2) by using the results of Theorem D.3 of Doikov et al

    and denote st ≜ η∥xt − x∗∥, then it holds that st+1 (18) ≤ 1 2 s2 t + s1.5 t + sπ(t)st. Since it holds s0 (19) ≤ 1/24, we achieve st ≤ 1 2 2(1+(1+m/2)π(t))(1+(t%m)/2) by using the results of Theorem D.3 of Doikov et al. [14] (see Lemma 17 in Appendix A), which finishes the pro...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.