Pith. sign in

REVIEW 2 major objections 6 minor 43 references

Mixed precision Newton's method for optimization

T0 review · 2 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Newton's method can run the Hessian in low precision without hurting final accuracy; the gradient precision and working precision set the floor.

desk verdict Solid local error analysis that cleanly separates gradient, Hessian, and update errors for mixed-precision Newton, with usable precision-assignment rules and honest experiments. read the letter →

arxiv 2607.04828 v1 pith:5MSPPGJA submitted 2026-07-06 math.OC

Nicolas Brisebarre (LIP , CNRS) , Giuseppe Carrino (OCKHAM) , Theo Mary (PEQUAN) , Elisa Riccietti (OCKHAM) This is my paper · ORCID
classification math.OC MSC 90C3065K0565G5065F10
keywords Newton'smethodmixedprecisionerroranalysisfloating-pointarithmeticinexactNewtonquasi-NewtonGauss–Newtonlocalconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Newton's method is expensive because every step builds a Hessian and solves a linear system. This paper shows that those costly steps can be done in lower floating-point precision (or with deliberate approximation) without spoiling the final answer quality. A single error model covers rounding errors, inexact linear solves, and quasi-Newton Hessian approximations at once. The analysis proves that the sequence still contracts until it reaches a limiting accuracy that depends, to first order, only on the gradient error and the working precision used to store the iterates—not on the Hessian error. Hessian inexactness mainly slows the linear rate of approach. The same bounds recover classical exact-arithmetic results for inexact and Gauss–Newton methods and immediately yield mixed-precision variants of those algorithms. Experiments on synthetic problems and a CUTEst suite confirm that the predicted rates and accuracy floors appear in practice, and that many problems tolerate half-precision Hessians with only modest iteration inflation.

What carries the argument

Error Model 2.1 together with Theorems 2.3 and 2.4: the model injects three controlled perturbations into the classical Newton step; the theorems convert those perturbations into explicit α_i, β_i, γ_i (and analogous gradient-norm factors) that bound both the iterate error and the gradient norm until a precision-dependent floor is reached.

What would settle it

On a problem whose Hessian condition number is moderate, deliberately inflate only the Hessian arithmetic error (or the CG residual tolerance) while keeping gradient and working precision fixed; if the final attainable accuracy worsens by more than a constant factor while the linear rate slows as predicted, the claimed separation of γ from ε^H is false.

Watch

Extended reading notes

Core claim

Under a three-source error model (gradient absolute error, Hessian relative error, and update rounding), mixed-precision Newton satisfies a local bound whose quadratic, linear, and constant terms cleanly separate the effects of each error. The attainable accuracy is governed by the constant term γ_i ≈ ε^g_i ‖H^{-1}‖ + ε_i ‖x*‖ and is therefore independent of the Hessian perturbation to first order; Hessian error appears only in the linear rate factor β_i ≈ ε^H_i κ(H).

Load-bearing premise

Everything is local: the Hessian must be nonsingular and Lipschitz near the minimizer, the starting point must already lie inside a small enough ball, and at every step the product of the Hessian relative error and the condition number must stay strictly below one.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper develops a local error analysis of Newton’s method for unconstrained optimization under a general Error Model 2.1 that separately bounds absolute gradient error, relative Hessian (formation/solve) error, and relative update error. Theorems 2.3 and 2.4 give explicit bounds on the iterate error and gradient norm, showing that attainable accuracy is governed to first order by γ_i ≈ ε^g_i ∥H(x̂_i)^{-1}∥ + ε_i ∥x*∥ (independent of the Hessian error), while Hessian inexactness mainly enters the linear rate factor β_i ≈ ε^H_i κ(H). The framework recovers classical exact-arithmetic Newton, inexact Newton (via Rigal–Gaches), and Gauss–Newton rates as special cases and is used to propose mixed-precision variants. Extensive numerical experiments on synthetic least-squares problems and a CUTEst subset validate the bounds when the local assumptions hold and illustrate precision-assignment trade-offs.

Significance. Mixed-precision algorithms are well developed in numerical linear algebra but largely lack a unifying local theory for second-order nonlinear optimization. This paper supplies that theory with a clean separation of error sources, recovers known rates as special cases, and yields concrete guidelines (evaluate the gradient at least as accurately as the working precision; tolerate Hessian error inversely proportional to κ(H)). The analysis is deterministic and local rather than worst-case global complexity, which yields sharper, more informative bounds. Strengths include independent measurement of ε^g_i and ε^H_i (extended-precision reference and Rigal–Gaches), publicly available code, and experiments that falsifiably check when the theory is tight versus conservative. The contribution is solid and of clear interest to the optimization and high-performance computing communities.

major comments (2)
  1. The central claim of Theorem 2.3 is correctly derived under the stated local hypotheses (H(x*) nonsingular, H Lipschitz near x*, ν_i = ε^H_i κ(H(x̂_i)) < 1, and a sufficiently small starting ball). The Appendix A.1 proof follows the standard residual expansion (Lemmas A.1–A.2) and cleanly separates the three error sources; classical rates are recovered when the error terms vanish. No load-bearing algebraic error was found.
  2. Section 4.2.1 and Figure 4.2 (SINREG, n=8, κ(H)≈4×10^5) and the CUTEst discussion (MEXHAT, PENALTY1, BROWNAL) show that the assumptions of Theorems 2.3–2.4 can fail while the method still converges, and conversely that the theory correctly flags some failures. Because the paper advertises “precise guidelines for assigning a different precision,” a short, explicit discussion of how a practitioner should use (or not use) the ν_i < 1 and θ_i < θ_max tests when they are known to be conservative would strengthen the practical takeaway. This is a clarification, not a flaw in the local claim.
minor comments (6)
  1. Throughout the manuscript many cross-references appear as “??” (e.g., “?? presents the error analysis”, “See ??.”, Algorithm and theorem pointers). These must be resolved before publication.
  2. Conclusion: “sacrifing” → “sacrificing”.
  3. Section 1.3 / Definition 2.2: the Lipschitz definition is standard; a brief remark that the same constant LH is used for both the residual lemma and the rate coefficients would help readers track constants.
  4. Figures 4.1–4.3: the transparent dashed bound curves are helpful; ensuring they are distinguishable in grayscale (or adding markers) would improve accessibility.
  5. Section 3.2: the relation η_i ≈ ζ_i c_solver u_H is a useful rule of thumb; a one-sentence pointer to the corresponding experiment (Figure 4.5 / 4.6) already present later would tighten the narrative.
  6. Related-work paragraph on first-order mixed-precision methods ([28,33]) is brief; a sentence contrasting local second-order rates with the global complexity bounds typical of those works would better position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: local Newton bounds are derived forward from Error Model 2.1 and standard Lipschitz/nonsingularity lemmas; experiments validate rather than define the constants.

full rationale

The central claim (Theorem 2.3) is a standard local residual expansion of mixed-precision Newton under Error Model 2.1. The proof (Appendix A.1) starts from the perturbed update (2.3a–e), applies the classical Lipschitz residual bound (Lemma A.1 / Dennis–Schnabel) and the Neumann inverse bound (Lemma A.2), and obtains the quadratic/linear/constant terms α_i, β_i, γ_i by direct algebra. The first-order forms (2.12)–(2.13) follow by dropping higher-order terms; they are not fitted to data. Special cases (exact arithmetic, inexact Newton via Rigal–Gaches, Gauss–Newton via EH = −S) recover known rates from the literature without importing uniqueness theorems or free parameters from the authors’ prior work. Numerical sections measure ε^g_i and ε^H_i independently (extended-precision residual / Rigal–Gaches) and plot the resulting bounds against observed trajectories; the constants are therefore predictions, not post-hoc fits. Self-citations (Higham, Nocedal–Wright, Dembo–Eisenstat–Steihaug, Dennis–Schnabel, Tisseur) are ordinary background references and are not load-bearing for the derivation. No self-definitional loop, fitted-input-as-prediction, or ansatz-smuggling step appears. Score 0 is therefore the correct, proportionate finding.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central theorems rest on standard smooth-optimization and floating-point axioms plus an explicit three-term error model. No data-fitted free parameters enter the proofs; experimental tolerances (η, h for finite differences, precision triples) are design choices for validation, not constants inside the claimed bounds.

assumptions (5)
  • domain assumption f is twice continuously differentiable; H is Lipschitz continuous with constant L_H in a neighborhood of a local minimizer x* with H(x*) nonsingular (positive definite for the classical Newton step).
    Stated before Theorem 2.3; standard for local quadratic Newton theory (Nocedal–Wright style).
  • ad hoc to paper Error Model 2.1: gradient absolute error ≤ ε^g_i, Hessian relative normwise error ≤ ε^H_i, update relative error ≤ ε_i, with ν_i = ε^H_i κ(H(x̂_i)) < 1.
    The paper's modeling choice that unifies rounding, inexact solves, and Hessian approximation; all main bounds are conditional on this model.
  • standard math IEEE-style floating-point model fl(a op b) = (a op b)(1+δ), |δ|≤u, and standard backward-stability bounds for direct/iterative linear solvers.
    Section 3.1; used to instantiate ε_i, ε^H_i, ε^g_i with unit roundoffs u, u_H, u_g.
  • domain assumption Local initialization: x̂_0 sufficiently close to x* so the sequence stays in the ball where H remains invertible and θ_i < θ_max < 1.
    Second part of Theorems 2.3–2.4; no globalization strategy is analyzed.
  • domain assumption For Gauss–Newton specialization, discarded second-order residual term S can be treated as EH with ε^H = ||S||/||H|| (and comparison to Dennis–Schnabel σ-condition).
    Section 3.3; standard structural assumption for least-squares quasi-Newton analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mixed precision Newton's method for optimization." pith.science (2026). https://pith.science/paper/5MSPPGJA

@misc{pith2026260704828,
  author       = {Pith},
  title        = {Pith review of: Mixed precision Newton's method for optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MSPPGJA}},
  note         = {Machine review of arXiv:2607.04828}
}
read the original abstract

Second-order optimization methods, such as Newton's algorithm, achieve fast local convergence and high accuracy, but their practical use is often limited by high computational costs. To mitigate this issue, variants such as inexact and quasi-Newton methods are widely used. A complementary and promising approach to improve the efficiency of the method is to employ mixed precision arithmetic, using different floating-point precisions for different operations, based on their impact on the convergence and accuracy of the method. In this work, we perform an error analysis of Newton's method accounting for different sources of inexactness, including approximations and rounding errors. We present a convergence analysis for the generated sequence, establishing bounds on the convergence rate and attainable accuracy. This theoretical framework covers quasi-Newton and inexact Newton methods, and is leveraged to propose mixed precision algorithms. We present a wide set of numerical experiments to illustrate our theoretical results and the behavior of Newton's method and its approximate variants in mixed precision floating-point arithmetic.

Figures

Figures reproduced from arXiv: 2607.04828 by the authors.

Figure 4.1
Figure 4.1. Convergence of mixed precision Newton in relative error (left) and gradient [PITH_FULL_IMAGE:figures/full_fig_p013_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Convergence of mixed precision Newton in relative error (left) and gradient [PITH_FULL_IMAGE:figures/full_fig_p014_4_2.png] view at source ↗
Figure 4.3
Figure 4.3. Convergence of mixed precision Newton in relative error (left) and gra [PITH_FULL_IMAGE:figures/full_fig_p015_4_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4.4
Figure 4.4. Figure 4.4: Performance profile of uniform and mixed precision Newton’s method [PITH_FULL_IMAGE:figures/full_fig_p016_4_4.png]
Figure 4.5
Figure 4.5. Figure 4.5: Mixed precision inexact Newton on the ENGVAL1 problem. Panel (a) reports the number of nonlinear iterations required for convergence as a function of the CG stopping tolerance η for several precision sets (ug, u, uH); the purple and blue curves overlap. Panels (b)–(d…
Figure 4.6
Figure 4.6. Figure 4.6: Relative error (left) and number of linear CG iterations (right) for mixed [PITH_FULL_IMAGE:figures/full_fig_p019_4_6.png]
Figure 4.7
Figure 4.7. Figure 4.7: Convergence of mixed precision Newton and Gauss–Newton in gradient [PITH_FULL_IMAGE:figures/full_fig_p020_4_7.png]
Figure 4.8
Figure 4.8. Figure 4.8: Iterations needed to converge to limiting gradient for mixed precision [PITH_FULL_IMAGE:figures/full_fig_p020_4_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    nvidia, may 2024

    Cuda ptx isa. nvidia, may 2024. release 8.5

    2024

  2. [2]

    IEEE Std 754-2019 (Revision of IEEE 754-2008), pages 1–84, July 2019

    IEEE standard for floating-point arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), pages 1–84, July 2019. doi:10.1109/IEEESTD.2019.8766229

    doi:10.1109/ieeestd.2019.8766229 2019

  3. [3]

    Forward and backward error bounds for a mixed precision preconditioned conjugate gradient algorithm, 2025

    Thomas Bake, Erin Carson, and Yuxin Ma. Forward and backward error bounds for a mixed precision preconditioned conjugate gradient algorithm, 2025. arXiv: 2510.11379

    arXiv 2025

  4. [4]

    A. S. Bandeira, K. Scheinberg, and L. N. Vicente. Convergence of trust-region methods based on probabilistic models. SIAM J. Optim., 24(3):1238–1264, 2014. doi:10.1137/130915984

    doi:10.1137/130915984 2014

  5. [5]

    A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients

    Stefania Bellavia, Serge Gratton, and Elisa Riccietti. A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients. Numer. Math. , 140(3):791–825, November 2018. doi: 10.1007/s00211-018-0977-z

    doi:10.1007/s00211-018-0977-z 2018

  6. [6]

    Adaptive cubic regularization methods with dynamic inexact hessian information and applica- tions to finite-sum minimization

    Stefania Bellavia, Gianmarco Gurioli, and Benedetta Morini. Adaptive cubic regularization methods with dynamic inexact hessian information and applica- tions to finite-sum minimization. IMA J. Numer. Anal. , 41(1):764–799, 01 2021. doi:10.1093/imanum/drz076. 21

    work page doi:10.1093/imanum/drz076 2021

  7. [7]

    Subsampled inexact Newton methods for minimizing large sums of convex functions

    Stefania Bellavia, Nataˇ sa Kreji´ c, and Nataˇ sa Krklec Jerinki´ c. Subsampled inexact Newton methods for minimizing large sums of convex functions. IMA J. Numer. Anal., 40(4):2309–2341, 10 2020. doi:10.1093/imanum/drz027

    work page doi:10.1093/imanum/drz027 2020

  8. [8]

    Inexact Newton methods with matrix approximation by sampling for nonlinear least-squares and systems, August 2023

    Stefania Bellavia, Greta Malaspina, and Benedetta Morini. Inexact Newton methods with matrix approximation by sampling for nonlinear least-squares and systems, August 2023. URL: https://optimization-online.org/?p=24079

    2023

Show all 43 references
  1. [9]

    El Houcine Bergou, Youssef Diouane, Vyacheslav Kungurtsev, and Cl´ ement W. Royer. A stochastic Levenberg–Marquardt method using random models with complexity results. SIAM-ASA J. Uncertain. Quantif., 10(1):507–536, 2022. doi: 10.1137/20M1366253

    2022 doi

  2. [10]

    Convex Optimization

    Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge Uni- versity Press, 2004

    2004

  3. [11]

    R. H. Byrd, S. L. Hansen, Jorge Nocedal, and Y. Singer. A stochastic quasi- Newton method for large-scale optimization. SIAM J. Optim. , 26(2):1008–1031,

  4. [12]

    doi:10.1137/140954362

    doi

  5. [13]

    Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation and Perspec- tives

    Coralia Cartis, Nicholas IM Gould, and Philippe L Toint. Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation and Perspec- tives. SIAM, 2022

    2022

  6. [14]

    R. Chen, M. Menickelly, and K. Scheinberg. Stochastic optimization using a trust-region method and random models. Math. Program., 169(2):447–487, June

  7. [15]

    doi:10.1007/s10107-017-1141-8

    doi

  8. [16]

    Dembo, Stanley C

    Ron S. Dembo, Stanley C. Eisenstat, and Trond Steihaug. Inexact Newton meth- ods. SIAM J. Numer. Anal. , 19(2):400–408, 1982. doi:10.1137/0719025

    1982 doi

  9. [17]

    J. E. Dennis, Jr. and Robert B. Schnabel. Numerical methods for unconstrained optimization and nonlinear equations , volume 16 of Classics in Applied Mathe- matics. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1996. Corrected reprint of the 1983 original. ...

    1996 doi

  10. [18]

    Dolan and Jorge J

    Elizabeth D. Dolan and Jorge J. Mor´ e. Benchmarking optimization software with performance profiles. Math. Program., 91(2):201–213, January 2002. doi: 10.1007/s101070100263

    2002 doi

  11. [19]

    Gratton and Ph

    S. Gratton and Ph. L. Toint. A note on solving nonlinear optimization problems in variable precision. Comput. Optim. Appl., 76(3):917–933, 2020. doi:10.1007/ s10589-020-00190-2

    2020

  12. [20]

    Gratton and Ph

    S. Gratton and Ph. L. Toint. S2MPJ and CUTEst optimization problems for Matlab, Python and Julia. Optim. Methods Softw. , 40(4):871–903, 2025. doi: 10.1080/10556788.2025.2490640

    2025 doi

  13. [21]

    Hestenes and Eduard Stiefel

    Magnus R. Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving linear systems. J. Research Nat. Bur. Standards, 49:409–436, 1952. URL: https://nvlpubs.nist.gov/nistpubs/jres/049/jresv49n6p409_A1b.pdf

    1952

  14. [22]

    Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms . Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, second edition,

  15. [23]

    doi:10.1137/1.9780898718027. 22

    doi

  16. [24]

    Higham and Theo Mary

    Nicholas J. Higham and Theo Mary. A new approach to probabilistic rounding error analysis. SIAM J. Sci. Comput. , 41(5):A2815–A2835, 2019. doi:10.1137/ 18M1226312

    2019

  17. [25]

    Higham and Theo Mary

    Nicholas J. Higham and Theo Mary. Mixed precision algorithms in numerical lin- ear algebra. Acta Numer., 31:347–414, 2022. doi:10.1017/S0962492922000022

    2022 doi

  18. [26]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985

    1985

  19. [27]

    C. T. Kelley. Newton’s method in mixed precision. SIAM Rev., 64(1):191–211,

  20. [28]

    doi:10.1137/20M1342902

    doi

  21. [29]

    C. T. Kelley. Newton’s method in three precisions. Pac. J. Optim. , 20(3):461– 474, 2024. URL: https://arxiv.org/abs/2307.16051

    2024 arXiv

  22. [30]

    Lancaster

    P. Lancaster. Error analysis for the Newton-Raphson method. Numer. Math. , 9(1):55–68, November 1966. doi:10.1007/BF02165230

    1966 doi

  23. [31]

    Practical quasi-newton methods for solving nonlinear systems

    Jos´ e Mario Mart´ ınez. Practical quasi-newton methods for solving nonlinear systems. J. Comput. Appl. Math. , 124(1-2):97–121, December 2000. doi: 10.1016/S0377-0427(00)00434-9

    doi

  24. [32]

    A multi-precision quadratic regularization method for unconstrained optimization with rounding error analysis

    Dominique Monnet and Dominique Orban. A multi-precision quadratic regularization method for unconstrained optimization with rounding error analysis. Comput. Optim. Appl. , 91(3):997–1031, 2025. doi:10.1007/ s10589-025-00676-x

    2025

  25. [33]

    Jorge Nocedal and Stephen J. Wright. Numerical optimization. Springer Series in Operations Research and Financial Engineering. Springer Nature, 2006. doi: 10.1007/978-0-387-40065-5

    2006 doi

  26. [34]

    A structured tour of optimization with finite differences, 2025

    Marco Rando, Cesare Molinari, Lorenzo Rosasco, and Silvia Villa. A structured tour of optimization with finite differences, 2025. arXiv:2505.19720

    2025 arXiv

  27. [35]

    Farbod Roosta-Khorasani and Michael W. Mahoney. Sub-sampled Newton methods. Math. Program. , 174(1-2):293–326, March 2019. doi:10.1007/ s10107-018-1346-5

    2019

  28. [36]

    Newton’s method in floating point arithmetic and iterative refinement of generalized eigenvalue problems

    Fran¸ coise Tisseur. Newton’s method in floating point arithmetic and iterative refinement of generalized eigenvalue problems. SIAM J. Matrix Anal. Appl. , 22(4):1038–1057, 2001. doi:10.1137/S0895479899359837

    2001 doi

  29. [37]

    Lower and upper bounds of the con- vergence rate of gradient methods with composite noise in gradient, 2026

    Artem Vasin and Alexander Gasnikov. Lower and upper bounds of the con- vergence rate of gradient methods with composite noise in gradient, 2026. arXiv:2603.12376

    2026

  30. [38]

    Wo´ zniakowski

    H. Wo´ zniakowski. Numerical stability for solving nonlinear equations. Numer. Math., 27(4):373–390, December 1977. doi:10.1007/BF01399601

    1977 doi

  31. [39]

    Peng Xu, Fred Roosta, and Michael W. Mahoney. Newton-type methods for non-convex optimization under inexact Hessian information. Mathematical Pro- gramming, 184(1):35–70, November 2020. doi:10.1007/s10107-019-01405-z . 23

    2020 doi

  32. [40]

    Yao, Peng Xu, Fred Roosta, Stephen J

    Z. Yao, Peng Xu, Fred Roosta, Stephen J. Wright, and Michael W. Mahoney. Inexact Newton-CG algorithms with complexity guarantees. IMA J. Numer. Anal., 2021. doi:10.1093/imanum/drac043

    2021 doi

  33. [41]

    Inexact nonconvex Newton-type methods

    Zhewei Yao, Peng Xu, Fred Roosta, and Michael W Mahoney. Inexact nonconvex Newton-type methods. INFORMS J. Optim., 3(2):154–182, 2021. doi:10.1287/ ijoo.2019.0043

    2021

  34. [42]

    T. J. Ypma. The effect of rounding errors on Newton-like methods. IMA J. Numer. Anal., 1983. doi:10.1093/imanum/3.1.109

    1983 doi

  35. [43]

    T. J. Ypma. Local convergence of inexact newton methods. SIAM J. Numer. Anal., 21(3):583–590, 1984. doi:10.1137/0721040. A Proofs A.1 Proof of ?? For the proof of this theorem, we will need the following two lemmas. Lemma A.1 (Lem. 4.1.12, [15]) . Let g : Rn → Rm be continuous...

    1984 doi

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.