Pith. sign in

REVIEW 1 major objections 5 minor 69 references

An augmented Lagrangian method solves constrained nonlinear least-squares by keeping linear constraints explicit and approximating the penalty Hessian with a structured SR1 update.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:04 UTC pith:LI4HDULN

load-bearing objection Solid specialized AL solver for constrained NLS: careful combination of known pieces, honest numerics, one known gap between theorem and code. the 1 major comments →

arxiv 2607.11239 v1 pith:LI4HDULN submitted 2026-07-13 math.OC cs.NAmath.NA

An augmented Lagrangian algorithm for constrained nonlinear least-squares

Pierre Borie , Fabian Bastin , St\'ephane Dellacherie This is my paper
classification math.OC cs.NAmath.NA MSC 90C3065K0590C55
keywords constrained nonlinear least-squaresaugmented Lagrangianstructured quasi-NewtonSR1 updategradient projectiontrust regionhybrid Hessian approximation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many fitting and parameter-estimation problems are nonlinear least-squares with a mixture of nonlinear constraints and linear equalities or bounds. This paper builds a solver that folds only the nonlinear constraints into an augmented Lagrangian while leaving the linear set untouched, so each outer step reduces to a linearly constrained subproblem that can be attacked by gradient projection and projected conjugate gradients. The Hessian of that subproblem is never formed exactly; instead a Gauss–Newton block is corrected by a structured SR1 update that respects the sum-of-squares structure of both residuals and constraints. Under standard smoothness and rank assumptions the outer sequence is shown to produce first-order critical points. Numerical tests on seventy-nine problems show that a hybrid version of the structured update is competitive in wall-clock time with a leading interior-point solver while remaining matrix-free.

Core claim

The paper establishes that an augmented-Lagrangian outer loop combined with trust-region gradient-projection inner iterations and a hybrid structured-SR1 Hessian approximation converges globally to first-order KKT points of a nonlinear least-squares problem subject to mixed nonlinear and linear constraints, and that the resulting algorithm is practically competitive on a standard test set.

What carries the argument

The structured secant equation for the second-order remainder of the augmented Lagrangian, Bk+1 sk = (Jk+1 − Jk)⊤ rk+1 + (Ck+1 − Ck)⊤ λ̄k+1, which is realized by an SR1 update and optionally switched with pure Gauss–Newton via a residual-ratio test.

Load-bearing premise

The code stops when the reduced gradient is small, yet the global-convergence theorem is proved for the true projected gradient; the two coincide only when active-bound multipliers keep the correct sign at every outer iterate.

What would settle it

Run the published Julia implementation on a problem whose active-bound multipliers change sign near a limit point and check whether the reduced-gradient residual vanishes while the true projected-gradient residual stays bounded away from zero.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes TRAULLS, an augmented Lagrangian algorithm for nonlinear least-squares problems with mixed nonlinear and linear (equality/bound) constraints. Nonlinear constraints are penalized via the AL function (2.1), while linear constraints are retained and handled by a trust-region gradient-projection inner solver (Cauchy point along the projected path plus projected CG minor iterations). The AL Hessian is approximated by a Gauss–Newton term plus a structured SR1 correction of the second-order residual/constraint contribution (secant equation (2.26), update (2.27)), with an optional hybrid switch (2.29). Global convergence of the inner process (Theorem 5) and of the outer AL iterates (Theorem 9 / Lemma 8) is established under standard assumptions (twice continuous differentiability, full-rank active linear constraints, compact level sets, controlled Hessian growth, and rank of C(x*)N* ≥ nc). Numerical experiments on 79 instances compare Hessian variants and show the hybrid-SR1 version competitive with IPOPT in CPU time and superior to Percival.

Significance. The work cleanly combines two mature lines—AL methods with gradient-projection subproblem solves and structured quasi-Newton updates for least-squares—into a solver that treats general nonlinear equalities/inequalities together with linear equalities and bounds. The structured secant equation for the AL second-order term and the polyhedral adaptation of the Cauchy-point path are concrete technical contributions. Global-convergence proofs follow the Conn–Gould–Toint / Conn–Gould–Sartenaer–Toint templates after careful reformulation for polyhedra, and a public Julia implementation (TRAULLS) with reproducible benchmarks is provided. The hybrid-SR1 variant is shown to be competitive with a state-of-the-art interior-point solver on a standard test set while exploiting first-order structure only. These strengths make the paper a useful addition to the constrained NLS literature.

major comments (1)
  1. End of §3: the proved outer convergence (Theorem 9 / Lemma 8) is stated for the projected-gradient residual (2.6), while the implementation and all numerical claims use the reduced-gradient test (2.10). The authors correctly note that the two coincide only when bound multipliers have the correct sign at every outer iterate; without that condition only the tangential component is controlled and a limit point need not be critical. This is the sole material gap between the theorems and the code. A short clarification—either an explicit additional assumption that the sign condition holds along the generated sequence, a practical safeguard that restores the projected residual when signs are wrong, or a statement that the numerical claims are conditional on the observed sign pattern—would close the gap without altering the rest of the argument.
minor comments (5)
  1. §2.5 / Figure 1: the hierarchy of outer / inner / minor iterations is clear, but a one-sentence reminder that the outer index K and inner index k are independent (and that B0 is reset to zero at each outer iteration) would help readers who jump between sections.
  2. §4.1: the performance profiles are informative; adding the absolute number of failures (already mentioned in the text) to the figure captions would make the profiles self-contained.
  3. Appendix A, Algorithm 4: the termination test “|A(ti)| = n−m” is correct under Assumption 4, but a brief remark that the trust-region bounds guarantee eventual full activation would remove any ambiguity for readers unfamiliar with the polyhedral path.
  4. Several minor typos appear (e.g., “nonlinar”, “Compuations”, “Transcations”, “Reasearch”); a careful proof-reading pass would clean them.
  5. References [37] and [26] both concern NL2SOL-related work; ensuring the year and author list of [37] match the published version would avoid confusion.

Circularity Check

0 steps flagged

No circularity: global KKT claims follow from stated assumptions and standard AL/trust-region theory; numerical claims are empirical benchmarks, not forced predictions.

full rationale

The paper's central theoretical claim (Theorem 9 / Lemma 8) is that, under Assumptions 1–9, every limit point of the outer iterates of Algorithm 1 is a first-order KKT point of the constrained NLS problem. The derivation is self-contained: it adapts the projected-gradient path analysis of Conn–Gould–Toint (SBMIN / LANCELOT) to polyhedral sets (Lemmas 1–3, Theorem 4–5) and the AL multiplier/feasibility arguments of Conn–Gould–Sartenaer–Toint (Lemmas 6–8, Theorem 9). No quantity that is later called a “prediction” is fitted from the target KKT residual; the structured SR1 update (2.26)–(2.27) and hybrid switch (2.29) are algorithmic choices whose effect is measured empirically, not asserted as a first-principles derivation. Hyperparameters (μ0, τ, κhyb, trust-region constants) are fixed once for the 79-instance suite and reported; they do not appear as free constants inside the theorems. Self-citations are to standard background (Conn et al., Dennis–Gay–Walsh, Biggs, etc.) that is externally published and independently checkable; none is a uniqueness theorem that forbids alternatives and thereby forces the present result. The only material gap between theory and code—the use of the reduced-gradient test (2.10) instead of the projected-gradient residual (2.6)—is explicitly flagged by the authors at the end of §3 and is a correctness/implementation caveat, not a circular reduction. Consequently the derivation chain does not collapse into its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 2 invented entities

The central mathematical claim rests on standard smoothness/rank/compactness assumptions for AL and trust-region theory, plus algorithm hyperparameters fixed for experiments. No new physical entities. The structured secant equation and hybrid switch are design choices, not free parameters fitted to force the theorems.

free parameters (5)
  • Initial penalty μ0 and increase factor τ = μ0=10, τ=100
    Set to μ0=10, τ=100 for all tests; control outer feasibility enforcement and are not derived from theory.
  • Tolerance schedule constants (ω, η, κω, κη, βω, βη) = as listed in §4
    Drive ωK, ηK → 0; values κω=βω=η=ω=1, κη=0.1, βη=0.9 chosen for the benchmark.
  • Hybrid switch threshold κhyb = 0.1
    Decides when to use BGN+B vs pure Gauss–Newton; set to 0.1 following prior hybrid NLS practice.
  • Trust-region constants (δ0, α1, α2, η1, η2, γ1) = δ0=0.1, α1=0.25, α2=2.5, η1=0.25, η2=0.75, γ1=0.0625
    Control acceptance and radius updates; fixed once for all problems.
  • Stopping tolerances ω*, η* = 1e-5 / 1e-6
    Outer success criteria for reported solves; ω*=1e-5, η*=1e-6.
axioms (7)
  • domain assumption Residuals r and constraints c are twice continuously differentiable (Assumption 1).
    Needed for gradients/Hessians of Φ and for mean-value arguments in Lemma 8.
  • domain assumption Equality matrix A has full row rank; active-bound matrix à has full row rank at every x in Ω (Assumptions 2, 4).
    Guarantees well-defined projections onto T(x) and Cholesky of ÃÃᵀ.
  • domain assumption Linear feasible set Ω is nonempty (Assumption 3).
    Subproblems must be feasible.
  • domain assumption Level set of φ intersect Ω is nonempty and compact; Hessian approximations satisfy divergent 1/bk series and bk(φk+1−φk)→0 (Assumptions 5–7).
    Standard trust-region global-convergence hypotheses for the inner loop (Theorem 5).
  • domain assumption Outer iterates stay in a closed bounded domain; at every limit point, rank(C(x*)N*) ≥ nc (Assumptions 8–9).
    Ensures existence of limit points and well-defined least-squares multipliers for outer KKT analysis.
  • standard math Steps achieve a fixed fraction of Cauchy decrease (κfcd) and standard trust-region acceptance ratios.
    Classic sufficient-decrease framework from Conn–Gould–Toint.
  • ad hoc to paper When reduced gradient is used as the practical criticality measure, bound multipliers have correct sign so that reduced and projected residuals coincide.
    Stated only at the end of §3 as the bridge from Theorem 9 to the implementation; not enforced algorithmically.
invented entities (2)
  • TRAULLS algorithm (AL + polyhedral gradient projection + structured SR1 of AL Hessian + hybrid switch) independent evidence
    purpose: Concrete solver package for constrained NLS with the claimed convergence and empirical behavior.
    The named combination and Julia implementation are the paper’s product; components are assembled from prior literature.
  • Structured secant equation for AL second-order term (Eq. 2.26) with SR1 update (2.27) independent evidence
    purpose: Approximate Sk without second derivatives while preserving a standard secant equation for Hk.
    Adaptation of Biggs / NL2SOL / Li et al. ideas to the AL residual-and-multiplier sum; not a new physical object.

pith-pipeline@v1.1.0-grok45 · 32452 in / 3953 out tokens · 38431 ms · 2026-07-14T06:04:17.424877+00:00 · methodology

0 comments
read the original abstract

We present an algorithm for solving nonlinear least-squares problems subject to a mix of nonlinear and linear constraints. The nonlinear constraints are handled by reformulating the objective as the augmented Lagrangian function while linear constraints are handled directly. Each iteration consists of approximately solving a linearly constrained problem by means of a gradient projection technique. Our approach also involves a structured approximation of the augmented Lagrangian Hessian. We show global convergence of the method and assess the performance through numerical experiments.

Figures

Figures reproduced from arXiv: 2607.11239 by Fabian Bastin, Pierre Borie, St\'ephane Dellacherie.

Figure 1
Figure 1. Figure 1: Hierarchy of the three iteration levels in the AL algorithm. The outer loop updates [PITH_FULL_IMAGE:figures/full_fig_p015_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance profiles comparing five different Hessian approximation strategies with [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance profiles comparing the Gauss–Newton approximation against hybrid [PITH_FULL_IMAGE:figures/full_fig_p029_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance profiles comparing TRAULLS (Hybrid SR1), IPOPT, and Percival on [PITH_FULL_IMAGE:figures/full_fig_p030_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 32 canonical work pages · 1 internal anchor

  1. [1]

    Al-Baali and R

    M. Al-Baali and R. Fletcher. Variational methods for non-linear least-squares.The Journal of the Operational Research Society, 36(5):405–421, 1985. doi: 10.1057/jors.1985.68

    work page doi:10.1057/jors.1985.68 1985

  2. [2]

    Andreani, E.G

    R. Andreani, E.G. Birgin, J.M. Martinez, and M.L. Schuverdt. On augmented Lagrangian methods with general lower-level constraints.SIAM Journal on Optimization, 18(4):1286– 1309, 2008. doi: 10.1137/060654797

    doi:10.1137/060654797 2008

  3. [3]

    Arreckx, A

    S. Arreckx, A. Lambe, J.R.R.A. Martins, and D. Orban. A matrix-free augmented La- grangian algorithm with application to large-scale structural design optimization.Opti- mization and Engineering, 17:359–384, 2016. doi: 10.1007/s11081-015-9287-9

    work page doi:10.1007/s11081-015-9287-9 2016

  4. [4]

    Bartholomew-Biggs

    M.C. Bartholomew-Biggs. The estimation of the hessian matrix in nonlinear least-squares problems with non-zero residuals.Mathematical Programming, 12:67–80, 1977. doi: 10. 1007/BF01593770

    1977

  5. [5]

    Bellavia, M

    S. Bellavia, M. Macconi, and B. Morini. STRSCNE: A scaled trust-region solver for constrained nonlinear equations.Computational Optimization and Applications, 28:31–50, 2004

    2004

  6. [6]

    Bellavia, J

    S. Bellavia, J. Gondzio, and B. Morini. Computational experience with numerical methods for nonnegative least-squares problems.Numerical Linear Algebra with Applications, 18 (3):363–385, 2011

    2011

  7. [7]

    Bellavia, S

    S. Bellavia, S. Gratton, and E. Riccietti. A Levenberg-Marquardt method for large nonlin- ear least-squares problems with dynamic accuracy in functions and gradients.Numerische Mathematik, 140:791–825, 2018. doi: 10.1007/s00211-018-0977-z

    doi:10.1007/s00211-018-0977-z 2018

  8. [8]

    A variable dimension sketching strategy for nonlinear least-squares.SIAM Journal on Scientific Computing, 48(3):A1206– A1234, 2026

    Stefania Bellavia, Greta Malaspina, and Benedetta Morini. A variable dimension sketching strategy for nonlinear least-squares.SIAM Journal on Scientific Computing, 48(3):A1206– A1234, 2026. doi: 10.1137/25M175980X

    work page doi:10.1137/25m175980x 2026

  9. [9]

    Bergou, Y

    E.H. Bergou, Y. Diouane, V. Kungurtsev, and C.W. Royer. A nonmonotone matrix-free algorithm for nonlinear equality-constrained least-squares problems.SIAM Journal on Scientific Computing, 43(5):S743–S766, 2021. doi: 10.1137/20M1349138

    work page doi:10.1137/20m1349138 2021

  10. [10]

    J.T. Betts. Solving the nonlinar least square problem: Application of a general method.Journal of Optimization Theory and Applications, 18:469–483, 1976. doi: 10.1007/BF00932656. 31

    work page doi:10.1007/bf00932656 1976

  11. [11]

    Bezanson, A

    J. Bezanson, A. Edelman, S. Karpinski, and V.B. Shah. Julia: A fresh approach to numerical computing.SIAM Review, 59(1):65–98, 2017. doi: 10.1137/141000671

    doi:10.1137/141000671 2017

  12. [12]

    Biegler, J

    L.T. Biegler, J. Nocedal, C. Schmid, and D. Ternet. Numerical experience with a reduced hessian method for large scale constrained optimization.Computational Optimization and Applications, 15:45–67, 2000. doi: 10.1023/A:1008723031056

    work page doi:10.1023/a:1008723031056 2000

  13. [13]

    SIAM, Philadelphia, PA, USA, 2nd edition, 2024

    ˚Ake Bj¨ orck.Numerical Methods for Least Squares Problems. SIAM, Philadelphia, PA, USA, 2nd edition, 2024. doi: 10.1137/1.9781611977950

    work page doi:10.1137/1.9781611977950 2024

  14. [14]

    R.H. Byrd, J. Nocedal, and R.B. Schnabel. Representations of quasi-Newton matrices and their use in limited memory methods.Mathematical Programming, 63:129–156, 1994. doi: 10.1007/BF01582063

    doi:10.1007/bf01582063 1994

  15. [15]

    Conn, N.I.M

    A.R. Conn, N.I.M. Gould, and Ph.L. Toint. Global convergence of a class of trust region method algorithms for optimization with simple bounds.SIAM Journal on Numerical Analysis, 25(2):433–460, 1988. doi: 10.1137/0725029

    doi:10.1137/0725029 1988

  16. [16]

    Conn, N.I.M Gould, and Ph.L

    A.R. Conn, N.I.M Gould, and Ph.L. Toint. Testing a class of methods for solving mini- mization problems with simple bounds on the variables.Mathematics of Computation, 50 (182):399–430, 1988. doi: 10.1090/S0025-5718-1988-0929544-3

    work page doi:10.1090/s0025-5718-1988-0929544-3 1988

  17. [17]

    Conn, N.I.M

    A.R. Conn, N.I.M. Gould, and Ph.L. Toint. A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds.SIAM Journal on Numerical Analysis, 28(2):545–572, 1991. doi: 10.1137/0728030

    work page doi:10.1137/0728030 1991

  18. [18]

    Conn, N.I.M

    A.R. Conn, N.I.M. Gould, and Ph.L. Toint. Convergence of quasi-Newton matrices gen- erated by the symmetric rank one update.Mathematical Programming, 50:177–195, 1991. doi: 10.1007/BF01594934

    doi:10.1007/bf01594934 1991

  19. [19]

    Conn, N.I.M Gould, and Ph.L

    A.R. Conn, N.I.M Gould, and Ph.L. Toint.LANCELOT: A Fortran Package for Large- Scale Nonlinear Optimization (Release A). Springer Series in Computational Mathematics. Springer Berlin, Heidelberg, 1992. doi: 10.1007/978-3-662-12211-2

    work page doi:10.1007/978-3-662-12211-2 1992

  20. [20]

    A.R. Conn, N. Gould A. Sartenaer, and Ph.L. Toint. Global convergence of a class of trust region algorithms for optimization using inexact projections on convex constraints.SIAM Journal on Optimization, 3(1):164–221, 1993. doi: 10.1137/0803009

    work page doi:10.1137/0803009 1993

  21. [21]

    A.R. Conn, N. Gould A. Sartenaer, and Ph.L. Toint. Convergence properties of an augmented Lagrangian algorithm for optimization with a combination of general equal- ity and linear constraints.SIAM Journal on Optimization, 6(3):674–703, 1996. doi: 10.1137/S1052623493251463

    work page doi:10.1137/s1052623493251463 1996

  22. [22]

    Conn, N.I.M Gould, and Ph.L

    A.R. Conn, N.I.M Gould, and Ph.L. Toint.Trust Region Methods. SIAM, Philadelphia, PA, USA, 2000. doi: 10.1137/1.9780898719857

    doi:10.1137/1.9780898719857 2000

  23. [23]

    Curtis, H

    F.E. Curtis, H. Jiang, and D.P. Robinson. An adaptative augmented Lagrangian method for large-scale constrained optimization.Mathematical Programming, Series A, 152:201– 245, 2015. doi: 10.1007/s10107-014-0784-y

    work page doi:10.1007/s10107-014-0784-y 2015

  24. [24]

    Community-Based Service Ecosystem Evolution Analysis

    F. Delbos, J.Ch. Gilbert, R.Glowinski, and D. Sinoquet. Constrained optimization in seis- mic reflection tomography: a Gauss-Newton augmented Lagrangian approach.Geophysic Journal International, 164:670–684, 2006. doi: 10.1111/j.1365-246X.2005.02729.x. 32

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1111/j.1365-246x.2005.02729.x 2006

  25. [25]

    Dennis Jr and R.B

    J.E. Dennis Jr and R.B. Schnabel.Numerical Methods for Unconstrained optimization and Nonlinear Equations. Classics in Applied Mathematics. SIAM, Philadelphia, PA, USA,

  26. [26]

    doi: 10.1137/1.9781611971200

    doi:10.1137/1.9781611971200

  27. [27]

    Dennis Jr, D.M

    J.E. Dennis Jr, D.M. Gay, and R.E. Walsh. An adaptive nonlinear least-squares algorithm. ACM Transactions on Mathematical Software, 7(3):348–368, 1981. doi: 10.1145/355958. 355965

    work page doi:10.1145/355958 1981

  28. [28]

    Dolan and J.J

    E.D. Dolan and J.J. Mor´ e. Benchmarking optimization software with performance profiles. Mathematical Programming, Series A 91(2):201–213, 2002. doi: 10.1007/s101070100263

    doi:10.1007/s101070100263 2002

  29. [29]

    Fletcher and C

    R. Fletcher and C. Xu. Hybrid methods for nonlinear least squares.IMA Journal of Numerical Analysis, 7:373–389, 1987. doi: 10.1093/imanum/7.3.371

    work page doi:10.1093/imanum/7.3.371 1987

  30. [30]

    Gill and D.P

    P.E. Gill and D.P. Robinson. A primal-dual augmented Lagrangian.Computationnal Optimization and Applications, 51:1–25, 2012. doi: 10.1007/s10589-010-9339-1

    doi:10.1007/s10589-010-9339-1 2012

  31. [31]

    Golub and C.F

    G.H. Golub and C.F. Van Loan.Matrix Computations. JHU Press, Maryland, MD, USA, 4th edition, 2013. doi: 10.56021/9781421407944

    doi:10.56021/9781421407944 2013

  32. [32]

    Gon¸ calves and T.C

    M.L.N. Gon¸ calves and T.C. Menezes. Gauss–Newton methods with approximate projec- tions for solving constrained nonlinear least squares problems.Journal of Complexity, 58: 101459, 2020. doi: 10.1016/j.jco.2020.101459

    work page doi:10.1016/j.jco.2020.101459 2020

  33. [33]

    Gould, M.E

    N.I.M. Gould, M.E. Hribar, and J. Nocedal. On the solution of equality constrained quadratic programming problems arising in optimization.SIAM Journal on Scientific Computing, 23(4):1376–1395, 2001. doi: 10.1137/S1064827598345667

    work page doi:10.1137/s1064827598345667 2001

  34. [34]

    Gagnon, C

    Michel Grenier, J. Gagnon, C. Mercier, and J. Richard. Short-term load forecasting at Hydro-Qu´ ebec Trans´Energie. In2006 IEEE Power Engineering Society General Meeting, Montreal, QC, Canada, 2006. doi: 10.1109/PES.2006.1709029

    work page doi:10.1109/pes.2006.1709029 2006

  35. [35]

    Hestenes

    M.R. Hestenes. Multipler and gradient methods.Journal of Optimization Theory and Applications, 4:303–320, 1969. doi: 10.1007/BF00927673

    doi:10.1007/bf00927673 1969

  36. [36]

    Hock and K

    W. Hock and K. Schittkowski.Test Examples for Nonlinear Programming Codes, volume 187 ofLecture Notes in Economics and Mathematical Systems. Springer Berlin, Heidelberg, 2nd edition, 1980. doi: 10.1007/978-3-642-48320-2

    work page doi:10.1007/978-3-642-48320-2 1980

  37. [37]

    Huschens

    J. Huschens. On the use of product structure in secant methods for nonlinear least squares problems.SIAM Journal on Optimization, 4(1):108–129, 1994. doi: 10.1137/0804005

    work page doi:10.1137/0804005 1994

  38. [38]

    Martinez, and R.A

    J.E.Dennis Jr, H.J. Martinez, and R.A. Tapia. Convergence theory for the structured BFGS method with an application to nonlinear least squares.Journal of Optimization Theory and Applications, 61(2):161–178, 1999. doi: 10.1007/BF00962795

    work page doi:10.1007/bf00962795 1999

  39. [39]

    Levenberg

    K. Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2:164–168, 1944. doi: 10.1090/qam/10666

    doi:10.1090/qam/10666 1944

  40. [40]

    Z. Li, M. Osborne, and T. Prvan. Adaptative algorithm for constrained least-squares problems.Journal of Optimization Theory and Applications, 114:423–441, 2002. doi: 10.1023/A:1016043919978. 33

    work page doi:10.1023/a:1016043919978 2002

  41. [41]

    Lin and J.J Mor´ e

    C.-J. Lin and J.J Mor´ e. Newton’s method for large bound-constrained optimiza- tion problems.SIAM Journal on Optimization, 9(4):1100–1127, 1999. doi: 10.1137/ S1052623498345075

    1999

  42. [42]

    Lukˇ san and J

    L. Lukˇ san and J. Vlˇ cek. Sparse and partially separable test problems for unconstrained and equality constrained optimization. Technical Report 767, Institute of Computer Science, Academy of Sciences of the Czech Republic, 1999

    1999

  43. [43]

    Lukˇ san, C

    L. Lukˇ san, C. Matonoha, and J. Vlˇ cek. Hybrid methods for nonlinear least squares prob- lems. Technical Report 1246, Institute of Computer Science, Academy of Sciences of the Czech Republic, 2019

    2019

  44. [44]

    Mahdavi-Amiri and R.H

    N. Mahdavi-Amiri and R.H. Bartels. Constrained nonlinear least squares: An exact penalty approach with projected structured quasi-Newton update.ACM Transcations on Mathematical Software, 15(3):220–242, 1989. doi: 10.1145/66888.66891

    work page doi:10.1145/66888.66891 1989

  45. [45]

    Marquardt

    D.W. Marquardt. An algorithm for least squares estimation of non-linear parameters. SIAM Journal, 11:431–441, 1963. doi: 10.1137/0111030

    doi:10.1137/0111030 1963

  46. [46]

    Migot, D

    T. Migot, D. Monnet, D. Orban, and S. Abel Soares. JSOSolvers.jl: Unconstrained and bound-constrained optimization solvers.Journal of Open Source Software, 11(117):9467,

  47. [47]

    URLhttps://doi.org/10.21105/joss.09467

    doi: 10.21105/joss.09467. URLhttps://doi.org/10.21105/joss.09467

    doi:10.21105/joss.09467

  48. [48]

    Mor´ e and G

    J.J. Mor´ e and G. Toraldo. On the solution of large quadratic programming problems with bound constraints.SIAM Journal on Optimization, 1(1):93–113, 1991. doi: 10.1137/ 0801008

    1991

  49. [49]

    Murtagh and M.A

    B.A. Murtagh and M.A. Saunders. Large-scale linearly constrained optimization.Mathe- matical Programming, 14:41–72, 1978. doi: 10.1007/BF01588950

    doi:10.1007/bf01588950 1978

  50. [50]

    Nocedal and S.J

    J. Nocedal and S.J. Wright.Numerical Optimization. Springer series in Operation Reasearch and Financial Engineering. Springer, New York, NY, USA, 2nd edition, 2006. doi: 10.1007/978-0-387-40065-5

    doi:10.1007/978-0-387-40065-5 2006

  51. [51]

    D. Orban. BenchmarkProfiles.jl, 2019. URLhttps://doi.org/10.5281/zenodo. 4630955

    doi:10.5281/zenodo 2019

  52. [52]

    Orban and A.S

    D. Orban and A.S. Siqueira. A regularization method for constrained nonlinear least squares.Computational Optimization and Applications, 76(3):961–989, 2020. doi: 10. 1007/s10589-020-00201-2

    2020

  53. [53]

    Porcelli

    M. Porcelli. On the convergence of an inexact Gauss-Newton trust-region method for nonlinear least-squares problems with simple bounds.Optimization Letters, 7:447–465,

  54. [54]

    doi: 10.1007/s11590-011-0430-z

    work page doi:10.1007/s11590-011-0430-z

  55. [55]

    M.J.D. Powell. A method for nonlinear constraints in minimization problems. In R. Fletcher, editor,Optimization, pages 283–298. Academic Press, London, 1969

    1969

  56. [56]

    H. Pu, Z. Chen, B. Wang, and W. Xia. Constrained least squares algorithms for nonlin- ear unmixing of hyperspectral imagery.IEEE Transactions on Geoscience and Remote Sensing, 53(3):1287–1303, 2015. doi: 10.1109/TGRS.2014.2336858. 34

    work page doi:10.1109/tgrs.2014.2336858 2015

  57. [57]

    Rockafellar

    R.T. Rockafellar. The multiplier method of Hestenes and Powell applied to convex pro- gramming.Journal of Optimization Theory and Applications, 12(6):555–562, 1973. doi: 10.1007/BF00934777

    work page doi:10.1007/bf00934777 1973

  58. [58]

    Schittkowski.More Test Examples for Nonlinear Programming Codes

    K. Schittkowski.More Test Examples for Nonlinear Programming Codes. Lecture Notes in Economics and Mathematical Systems. Springer Berlin, Heidelberg, 1987. doi: 10.1007/ 978-3-642-61582-5

    1987

  59. [59]

    Soares Siqueira and D

    A. Soares Siqueira and D. Orban. NLSProblems.jl, 2019. URLhttps://doi.org/10. 5281/zenodo.4605405

    2019

  60. [60]

    Soares Siqueira and D

    A. Soares Siqueira and D. Orban. SolverBenchmark.jl, 2020. URLhttps://doi.org/10. 5281/zenodo.3948381

    2020

  61. [61]

    Soares Siqueira and D

    A. Soares Siqueira and D. Orban. NLPModelsIpopt.jl, 2026. URLhttps://doi.org/10. 5281/zenodo.19420624

    2026

  62. [62]

    R. Tapia. On secant updates for use in general constrained optimization.Mathematics of Compuations, 51(183):181–202, 1988. doi: 10.2307/2008585

    work page doi:10.2307/2008585 1988

  63. [63]

    Tjoa and L.T

    I.B. Tjoa and L.T. Biegler. Simultaneous solution and optimization strategies for pa- rameter estimation of differential-algebraic equation systems.Industrial & Engineering Chemistry Research, 30(2):376–385, 1991. doi: 10.1021/ie00050a015

    doi:10.1021/ie00050a015 1991

  64. [64]

    Toint and D

    Ph.L. Toint and D. Tuyttens. LSNNO, a FORTRAN subroutine for solving large-scale nonlinear network optimization problems.ACM Transactions on Mathematical Software, 18(3):308–328, 1992. doi: 10.1145/131766.131771

    work page doi:10.1145/131766.131771 1992

  65. [65]

    W¨ achter and L.T

    A. W¨ achter and L.T. Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming.Mathematical Programming, Series A, 106:25–57, 2006. doi: 10.1007/s10107-004-0559-y

    doi:10.1007/s10107-004-0559-y 2006

  66. [66]

    Yabe and T

    H. Yabe and T. Takahashi. Factorized quasi-Newton methods for nonlinear least squares problems.Mathematical Programming, 51:75–100, 1991. doi: 10.1007/BF01586927

    work page doi:10.1007/bf01586927 1991

  67. [67]

    Zhang, L.H

    J.Z. Zhang, L.H. Chen, and N.Y. Deng. A family of scaled factorized broydon-like methods for nonlinear least squares problems.SIAM Journal on Optimization, 10(4):1163–1179,

  68. [68]

    doi: 10.1137/S105262349834530

    work page doi:10.1137/s105262349834530

  69. [69]

    Zhou and X

    W. Zhou and X. Chen. Global convergence of a new hybrid Gauss-Newton structured BFGS method for nonlinear least squares problems.SIAM Journal on Optimization, 20 (5):2422–2441, 2010. doi: 10.1137/090748470. 35 Appendix A Cauchy point computation For the sake of clarity, we omit the iteration index during the rest of this appendix. We describe the procedur...

    work page doi:10.1137/090748470 2010