A Twin gradient method for unconstrained optimization

Anna De Magistris; Gerardo Toraldo; Michiel E. Hochstenbach

REVIEW 2 major objections 4 minor 31 references

A Twin gradient method for unconstrained optimization

T0 review · 2 major / 4 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two cooperating gradient paths that minimize mutual distance warm-start spectral steps and beat ABBmin alone.

desk verdict Clean Twin-Step transfer from Kaczmarz into gradient methods; hybrid works in practice, but the warm-start claim is not isolated from free switch thresholds. read the letter →

arxiv 2607.11575 v1 pith:337LJZ2W submitted 2026-07-13 math.NA cs.NAmath.OC

Anna De Magistris , Michiel E. Hochstenbach , Gerardo Toraldo This is my paper · ORCID

classification math.NAcs.NAmath.OC MSC 90C3065K0590C06

keywords unconstrainedoptimizationgradientmethodTwinmutualstepBarzilai-BorweinABBminhybridalgorithmspectralstepsize

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard gradient methods choose a stepsize to reduce the objective along one path. This paper instead runs two simultaneous steepest-descent sequences and, at every iteration, chooses both stepsizes so that the Euclidean distance between the two next points is minimized. That Twin-Step rule is a purely geometric coupling: the residual between the paths is forced orthogonal to the plane spanned by the two directions. Theory shows that the mutual distance contracts at least as fast as a one-parameter Minimal-Residual update, but that the contraction degrades as the directions become nearly parallel. The authors therefore embed Twin steps inside a hybrid that restarts the second path when collinearity or stagnation is detected and permanently switches to Adaptive Barzilai–Borwein once restarts fail. On hundreds of ill-conditioned quadratics and a large CUTEst suite the short Twin phase produces a rapid early drop that leaves the subsequent spectral phase with markedly better starting conditions, cutting total gradient evaluations relative to ABBmin alone.

What carries the argument

The Twin-Step: the 2-by-2 normal equations that locate the non-negative stepsizes minimizing ||(x+αp)−(z+βq)||, geometrically forcing the residual to be orthogonal to span{p,q}.

What would settle it

Run Twin-ABBmin and pure ABBmin on the same 720 random SPD quadratics (n up to 10^4, condition numbers up to 10^7) and the 164-problem CUTEst set; if the performance-profile curves for gradient evaluations coincide or reverse, the claimed warm-start advantage disappears.

Watch

Extended reading notes

Core claim

A Twin-Step that chooses both steplengths to minimize the next mutual distance between two gradient paths contracts that distance at least as fast as a one-parameter Minimal-Residual update; when the Twin phase is used only while the directions remain well-angled and then hands over to ABBmin, the hybrid systematically outperforms pure ABBmin on large suites of ill-conditioned problems.

Load-bearing premise

The two search directions must stay uniformly away from being parallel; once they line up the Twin step loses its contraction guarantee and can produce oversized or vanishing steps.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real novelty is the Twin-Step: two simultaneous steepest-descent sequences whose stepsizes solve a 2x2 mutual-distance least-squares problem rather than independent line searches or Rayleigh quotients. That geometry is a direct, well-executed transfer from Twin Kaczmarz, with new analysis (distance contraction via a 1D T2/BB2 comparison, angle-controlled damping, orthogonal z0 construction) and an honest hybrid that switches to ABBmin when collinearity or stagnation appears.

What the paper does well is clear. Quadratic global convergence (Props 2.3–2.4) and the general-function limit under the angle bound (Prop 2.5) are clean; the hybrid inherits standard nonmonotone Armijo/ABBmin theory after a finite Twin phase. Numerics are extensive—720 random SPD quadratics (n to 10^4, κ to 10^7) plus a large CUTEst/OPM suite—and Twin-ABBmin consistently beats plain ABBmin on gradient evaluations. The orthogonal auxiliary start and the geometric restart are practical and well-motivated. Circularity burden is near zero; the construction is not fitted to the profiles.

Soft spots are real but proportionate. Pure Twin still rests on Assumption 1 (uniform bound away from collinear directions); the hybrid only mitigates it with free thresholds γ=ρ=0.9 and a restart that injects a BB1 step. The stress-test note is right that switch timing is never tabulated or ablated, so the Abstract’s claim that “the Twin phase creates favorable initial conditions” is not cleanly isolated from dual-process exploration, the restart, or the particular thresholds. CUTEst exclusions (NaNs, differing solutions) need a transparent sensitivity note, and there is no public code. None of these break the math or the observed superiority; they just leave the causal story a bit under-supported.

This is for people who already work with spectral gradient methods and want a cheap geometric warm-start for ill-conditioned smooth problems or quadratic subproblems. It is not a new technology class, but it is honest progress inside a mature literature. I would send it to referees; the idea and the experiments are solid enough to deserve that time. Engage if you care about first-order hybrids; the Twin-Step construction is worth knowing even if the hybrid thresholds stay free parameters.

Referee Report

2 major / 4 minor

Summary. The paper introduces a Twin Gradient method that runs two parallel steepest-descent sequences and selects their stepsizes jointly by minimizing the Euclidean distance between the next iterates (the Twin-Step principle), rather than by independent line searches or spectral approximations. For strictly convex quadratics the mutual distance is shown to contract at least as fast as a one-dimensional Twin-T2 residual method (related to BB2/MR), and under a uniform angle bound (Assumption 1) both sequences converge to the unique minimizer; a damping construction can enforce the angle bound. Because pure Twin degrades under near-collinearity, a hybrid Twin-ABBmin is proposed that performs Twin steps with geometric restarts and switches to ABBmin (nonmonotone Armijo for general f) when collinearity or stagnation is detected. Extensive experiments on 720 random SPD quadratics (n up to 10^4, κ up to 10^7, three spectra) and a 164-problem CUTEst suite show Twin-ABBmin dominating ABBmin on gradient-evaluation performance profiles, supporting the claim that the Twin phase supplies favorable warm-starts for subsequent BB-type iterations.

Significance. The geometric Twin-Step construction is a genuine novelty for gradient stepsize selection, cleanly transferring the mutual-step idea from Twin Kaczmarz to unconstrained optimization. If the hybrid reliably accelerates spectral methods on ill-conditioned problems, it is of practical interest for large-scale applications. The quadratic contraction argument (Props. 2.3–2.4) and the hybrid’s inheritance of standard nonmonotone Armijo/ABBmin global convergence are solid; the orthogonal initialization of z0 and the explicit residual-orthogonality interpretation are elegant. The numerical suite is large and the superiority is consistent, especially for high condition numbers. These strengths make the work a worthwhile contribution to the spectral-gradient literature even though pure Twin is not recommended as a stand-alone method.

major comments (2)

[§2.2, Proposition 2.5] The statement asserts convergence to the unique global minimizer for general f∈C1, and the proof explicitly invokes “Since the problem is strictly convex, this implies that ∇f(x∞)≠0.” Problem (1) carries no convexity hypothesis. The geometric argument actually shows only that a common limit under Assumption 1 and dk→0 must be a critical point. The proposition, its proof, and the surrounding claims of “global convergence for general functions” must be corrected (or strict convexity added) before the general theory can be accepted.
[Abstract, §4–5, Algorithms 2–3, Figs. 2–5] The central claim that “the Twin phase creates favorable initial conditions for subsequent BB-type iterations” rests solely on aggregate performance profiles of Twin-ABBmin versus ABBmin. No statistics are reported on switch iteration, number of Twin steps/restarts, or the distribution of |γk|; no ablations isolate the Twin-Step geometry from dual-process exploration or the BB1 restart that is injected on every restart. Without such isolation the causal attribution emphasized in the Abstract and §5 remains under-supported, even though the numerical superiority itself is clear.

minor comments (4)

[§2.1] Notation for unnormalized (epk) versus normalized (pk) directions is occasionally inconsistent in the Twin-T2 comparison (Eqs. (13)–(14)); a short clarifying sentence would help.
[§3, §4] Fixed thresholds γ̄=ρ̄=0.9 are used without sensitivity study; a brief remark on robustness would strengthen the hybrid presentation.
[throughout] Several minor typos appear (“poiz0”, “the the distance”, “contributionshaveproposed”, etc.); a careful proof-reading pass is needed.
[§2.2–2.3] Tables 1–2 illustrate damping and stepsize synchronization for a tiny quadratic (n=5); their relevance to the large-scale claims could be stated more explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Twin-Step is a self-contained geometric least-squares construction; convergence follows from the 2 imes2 system under an explicit angle bound; benchmarks are external.

full rationale

The core construction (Eqs. 5–11) defines steplengths by minimizing the Euclidean distance between two simultaneous steepest-descent trajectories; the resulting 2 imes2 normal equations are solved exactly and are not fitted to any performance data. Global-convergence statements (Props. 2.3–2.5, 2.7) rest on the explicit Assumption 1 (uniform bound |γ_k|≤γ̄<1) together with standard Lipschitz or positive-definiteness hypotheses; the hybrid Twin-ABBmin merely monitors the same geometric quantities and switches to a classical spectral method when they degrade. The only external reference that is adapted (Cor. 4.2 of the Twin-Kaczmarz paper) supplies a geometric residual-orthogonality argument that is re-proved in the present setting; it is not used as an unverified uniqueness theorem that forces the present claims. Numerical superiority is demonstrated on independently generated SPD ensembles and the public CUTEst/OPM suite; no parameter is fitted to those suites and then re-presented as a prediction. Consequently the derivation chain never reduces to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

Central claims rest on standard smooth-optimization hypotheses (C1, Lipschitz gradient, SPD quadratic), two explicit paper assumptions (uniform angle bound; divergent sum of stepsizes), and a handful of hand-chosen hybrid thresholds. No new physical entities; the Twin-Step and Twin-ABBmin are algorithmic constructs with independent numerical handles.

free parameters (5)

γ̄ (collinearity threshold) = 0.9
Hybrid switch when |γ_k|>γ̄; experiments fix γ̄=0.9. Controls when Twin is abandoned.
ρ̄ (distance-contraction threshold) = 0.9
Hybrid switch when δ_k/δ_{k-1}>ρ̄; experiments fix ρ̄=0.9.
ABB/ABBmin τ and memory M_α = as in Frassoldati et al. [13]
Inherited spectral parameters; paper follows [13] without re-tuning report.
nonmonotone Armijo memory M and ν, backtrack factor υ
Line-search parameters for general (nonquadratic) Twin-ABBmin; standard but free.
damping sequence η_k (pure Twin) = formula (19) or η=0.9 in Example 2.1
Used in theory (Prop. 2.7) and Table 1; hybrid for n>2 omits explicit damping.

assumptions (5)

domain assumption f ∈ C¹; for several results ∇f is L-Lipschitz
Stated in §1 and used for damping/angle control (Prop. 2.7) and hybrid convergence.
domain assumption For quadratic theory, A symmetric positive definite
Problem (2); enables T2/MR contraction and eigenvalue bounds on stepsizes.
ad hoc to paper Assumption 1: |γ_k| ≤ γ̄ < 1 for all k
Load-bearing for pure Twin convergence and stepsize bounds; hybrid tries to enforce via restart/switch.
ad hoc to paper Assumption 2: ∑ α_k = ∑ β_k = ∞
Sufficient condition used in Prop. 2.4 to invoke Bertsekas-type convergence to the minimizer.
standard math Standard nonmonotone Armijo / Zoutendijk theory for ABBmin phase
Cited [15,20]; underpins global convergence claim after switch (§3.2).

invented entities (2)

Twin-Step (mutual-distance stepsize pair) independent evidence
purpose: Couples two gradient processes by minimizing ||(x+αp)−(z+βq)|| rather than f alone.
Core algorithmic object; falsifiable via the reported performance profiles and distance-contraction plots.
Twin-ABBmin hybrid with geometric restart/switch independent evidence
purpose: Uses Twin while |γ_k| and contraction are healthy, then falls back to ABBmin.
Practical globalization of Twin; evaluated on large quadratic and CUTEst suites.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Twin gradient method for unconstrained optimization." pith.science (2026). https://pith.science/paper/337LJZ2W

@misc{pith2026260711575,
  author       = {Pith},
  title        = {Pith review of: A Twin gradient method for unconstrained optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/337LJZ2W}},
  note         = {Machine review of arXiv:2607.11575}
}

abstract

We propose a new strategy for gradient-based unconstrained optimization, involving two parallel sequences of iterates that cooperate to determine their stepsizes via a \textit{Twin-Step} principle. Rather than minimizing the objective function individually, the algorithm selects steplengths that minimize the Euclidean distance between the two gradient based processes occurring simultaneously at each iteration. The theoretical analysis shows that the convergence of the mutual distance is governed by the angle between the search directions. In particular the effectiveness of the overall process degrades as the directions approach parallelism. To ensure robustness against collinearity, we introduce a hybrid framework, Twin-ABB$_{\min}$, which switches to the Adaptive Barzilai--Borwein method when the geometric cooperation becomes ineffective. Extensive and very promising numerical results evidence that the Twin phase creates favorable initial conditions for subsequent BB-type iterations.

Figures

Figures reproduced from arXiv: 2607.11575 by the authors.

**Figure 2.** Performance profile of ABBmin and Twin-ABBmin on 720 problems, in terms of number of gradient evaluations [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

**Figure 3.** ABBmin VS Twin-ABBmin on a problem with n = 1000 and κ(A) = 104 for the figure on the left, and κ(A) = 107 for the figure on the right. The black dot represents the iterates in witch the method switches to ABBmin, (fk = min{f(xk), f(zk)}). The computational results are summarized in [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance profile on the CUTEst test set. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

**Figure 5.** Figure 5: Convergence history on the Problem 11 from CUTEst test set. The black dot rep [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 2 linked inside Pith

[1]

Andrei.Unconstrained optimization test functions

N. Andrei.Unconstrained optimization test functions. Bolyai Soc. Math. Stud., 2008

2008
[2]

Barzilai and J

J. Barzilai and J. M. Borwein. Two-point step size gradient methods.IMA J. Numer. Anal., 8(1):141–148, 1988

1988
[3]

D. P. Bertsekas.Nonlinear Programming. Athena Scientific, Belmont, MA, USA, 1999

1999
[4]

Bottou, F

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning.SIAM Rev., 60(2):223–311, 2018

2018
[5]

A.L. Cauchy. Méthode générale pour la résolution des systèmes d’équations simultanées.C. R. Acad. Sci. Paris, 25:536–538, 1847
[6]

Crisci, V

S. Crisci, V. De Simone, M. Pragliola, and G. Toraldo. Bilevel robust optimization approach for multi-period sparse portfolio selection.J. Comput. Appl. Math., 470:116729, 2025

2025
[7]

Y. H. Dai. An alternate minimization gradient method.IMA J. Numer. Anal., 23(3):377–393, 2003

2003
[8]

Y. H. Dai and L. Z. Liao. R-linear convergence of the Barzilai and Borwein gradient method.IMA J. Numer. Anal., 22(1):1–10, 2002

2002

Show all 31 references

[9]

On spectral properties of steepest descent methods.IMA J

Roberta De Asmundis, Daniela di Serafino, Filippo Riccio, and Gerardo Toraldo. On spectral properties of steepest descent methods.IMA J. Numer. Anal., 33, 09 2013

2013
[10]

De Magistris, S

A. De Magistris, S. Crisci, V. De Simone, and G. Toraldo. A speed up strategy for gradient methods. Comput. Optim. Appl., pages 1–33, 2026

2026
[11]

di Serafino, V

D. di Serafino, V. Ruggiero, G. Toraldo, and L. Zanni. On the steplength selection in gradient methods for unconstrained optimization.Appl. Math. Comput., 318:176–195, 2018

2018
[12]

E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles.Math. Program., 91(2):201–213, 2002

2002
[13]

Frassoldati, L

G. Frassoldati, L. Zanni, and G. Zanghirati. New adaptive stepsize selections in gradient methods.J. Ind. Manag. Optim., 4(2):299–312, 2008

2008
[14]

Gratton and P

S. Gratton and P. L. Toint. OPM: A collection of optimization problems in MATLAB.arXiv:2112.05636, 2021

2021 arXiv
[15]

Grippo, F

L. Grippo, F. Lampariello, and S. Lucidi. A nonmonotone line search technique for Newton’s method.SIAM J. Numer. Anal, 23:707–716, 1986

1986
[16]

Huang, Y

Y. Huang, Y. H. Dai, and X. W. Liu. A mechanism of three-dimensional quadratic termination for the gradient method with applications.arXiv:2212.07255, 2024

2024 arXiv
[17]

Loris, M

I. Loris, M. Bertero, C. De Mol, R. Zanella, and L. Zanni. Accelerating gradient projection methods for ℓ1-constrained signal recovery by steplength selection rules.Appl. Comput. Harmon. Anal., 27(2):247–254, 2009. 21

2009
[18]

Maligranda

L. Maligranda. Simple norm inequalities.Amer. Math. Monthly, 113(3):256–260, 2006

2006
[19]

Y. E. Nesterov. A method for solving the convex programming problem with convergence rateO(1/k2).Sov. Math. Dokl., 27(2):372–376, 1983

1983
[20]

Nocedal and S

J. Nocedal and S. J. Wright.Numerical Optimization. Springer, New York, 2nd edition, 2006

2006
[21]

B. T. Polyak. Some methods of speeding up the convergence of iteration methods.USSR Comput. Math. Math. Phys., 4(5):1–17, 1964

1964
[22]

M. Raydan. The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem.SIAM J. Optim., 7(1):26–33, 1997

1997
[23]

SIAM, 2003

Yousef Saad.Iterative Methods for Sparse Linear Systems. SIAM, 2003

2003
[24]

Serafini, G

T. Serafini, G. Zanghirati, and L. Zanni. Gradient projection methods for quadratic programs and applica- tions in training support vector machines.Optim. Methods Softw., 20(2-3):353–378, 2005

2005
[25]

Sun and J

C. Sun and J. P. Liu. New stepsizes for the gradient method.Optim. Lett., 14:1943–1955, 2020

1943
[26]

B. S. Van Lith, P. C. Hansen, and M. E. Hochstenbach. A twin error gauge for Kaczmarz’s iterations.SIAM J. Sci. Comput., 43(5):S173–S199, 2021

2021
[27]

S. J. Wright, R. D. Nowak, and M. A. T. Figueiredo. Sparse reconstruction by separable approximation. IEEE Trans. Signal Process., 57(7):2479–2493, 2009

2009
[28]

Y. X. Yuan. A new stepsize for the steepest descent method.J. Comput. Math., 24(2):149–156, 2006

2006
[29]

Zanella, G

R. Zanella, G. Zanghirati, R. Cavicchioli, L. Zanni, P. Boccacci, M. Bertero, and G. Vicidomini. Towards real-time image deconvolution: application to confocal and STED microscopy.Sci. Rep., 3(1):2523, 2013

2013
[30]

Zhang and C

Y. Zhang and C. Sun. Cyclic gradient methods for unconstrained optimization.J. Oper. Res. Soc. China, 12:809–828, 2024

2024
[31]

B. Zhou, L. Gao, and Y. H. Dai. Gradient methods with adaptive step-sizes.Comput. Optim. Appl., 35(1):69–86, 2006. 22

2006

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

Andrei.Unconstrained optimization test functions

N. Andrei.Unconstrained optimization test functions. Bolyai Soc. Math. Stud., 2008

2008

[2] [2]

Barzilai and J

J. Barzilai and J. M. Borwein. Two-point step size gradient methods.IMA J. Numer. Anal., 8(1):141–148, 1988

1988

[3] [3]

D. P. Bertsekas.Nonlinear Programming. Athena Scientific, Belmont, MA, USA, 1999

1999

[4] [4]

Bottou, F

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning.SIAM Rev., 60(2):223–311, 2018

2018

[5] [5]

A.L. Cauchy. Méthode générale pour la résolution des systèmes d’équations simultanées.C. R. Acad. Sci. Paris, 25:536–538, 1847

[6] [6]

Crisci, V

S. Crisci, V. De Simone, M. Pragliola, and G. Toraldo. Bilevel robust optimization approach for multi-period sparse portfolio selection.J. Comput. Appl. Math., 470:116729, 2025

2025

[7] [7]

Y. H. Dai. An alternate minimization gradient method.IMA J. Numer. Anal., 23(3):377–393, 2003

2003

[8] [8]

Y. H. Dai and L. Z. Liao. R-linear convergence of the Barzilai and Borwein gradient method.IMA J. Numer. Anal., 22(1):1–10, 2002

2002