REVIEW 4 major objections 3 minor 18 references
Kahan's Automatic Step-Size Control for Unconstrained Optimization
T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Kahan's step-size-updating gradient rule is shown to be equivalent to the long Barzilai-Borwein step in the quadratic model, and an adaptive version is proven to converge globally with local R-linear rate on general smooth unconstrained pro
desk verdict Plausible and potentially useful formalization of Kahan's step-size, but the supplied text is unreadable and the general-case convergence hinges on safeguards I cannot see; worth a real referee, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The KGD step-size recurrence, which updates the step size from its previous value and the observed gradient change instead of recomputing it each iteration. In the quadratic model this recurrence is mathematically identical to the long BB step $\alpha_k^{\mathrm{BB1}}=\|s_{k-1}\|^2/(s_{k-1}^Ty_{k-1})$, with $s_{k-1}=x_k-x_{k-1}$ and $y_{k-1}=g_k-g_{k-1}$; the short version corresponds to $s_{k-1}^Ty_{k-1}/\|y_{k-1}\|^2$. This equivalence lets BB convergence machinery be imported into KGD, and the adaptive framework's proof controls the step-size recurrence for non-quadratic objectives.
What would settle it
For the quadratic rate bound, simulate the KGD/BB recurrence on $f(x)=\frac12\sum_{i=1}^n d_i x_i^2$ with $d_1=1$, $d_n=\kappa$, and measure the worst-case asymptotic contraction $\limsup_{k\to\infty}\|x_k-x^*\|^{1/k}$ over starting points; exceeding $1-1/\kappa$ would disprove the rate theorem. For the general claim, run the adaptive KGD on a smooth strongly convex non-quadratic function with a known optimum and check whether the safeguarded step-size sequence stays within the interval the proof requires; divergence or stagnation while the paper's assumptions hold would refute global converge
Extended reading notes
Core claim
The central claim is that Kahan's step-size iteration is not merely a heuristic cousin of BB. For any strongly convex quadratic with Hessian $H$, the KGD step size equals the long BB step, so the step-size recurrence inherits the entire quadratic convergence theory of BB; the paper also constructs a short KGD step, and both variants have worst-case error contraction at least $1-1/\mathrm{cond}(H)$. Outside quadratics, the paper's adaptive framework—with safeguards on the step-size recurrence—is proved to be globally convergent and locally R-linearly convergent for general unconstrained smooth minimization.
Load-bearing premise
The general-case proof assumes that a step-size recurrence whose exact equivalence to the long BB step is proved only for quadratic models remains well-behaved for arbitrary smooth non-quadratic objectives once the proposed adaptive safeguards are added; if some allowed objective defeats those safeguards, the global convergence claim falls.
Editorial extensions
If this is right
- In quadratic models, KGD and BB step-size choices are interchangeable: analyses, worst-case rates, and any future improvements for one transfer to the other.
- The derived short KGD step provides a second step-size strategy with the same $1-1/\mathrm{cond}(H)$ worst-case rate, giving practitioners a parameter-free alternative to long-step choices.
- The adaptive framework extends Kahan's recurrence to general smooth unconstrained problems with global and local R-linear convergence guarantees, so the method can be used outside the quadratic setting without relying on heuristic rationale.
- The numerical comparisons on CUTEst and logistic regression suggest the safeguarded KGD variants remain competitive with established BB and adaptive gradient methods on real problems.
Reading between the lines
- Editorial inference: because the quadratic rate $1-1/\mathrm{cond}(H)$ is the same as plain steepest descent's worst-case rate, KGD's advantage is unlikely to be a better tail rate on ill-conditioned quadratics; the practical gain must come from avoidance of line-search cost or better early behavior, which the paper's experiments address.
- Editorial inference: the proof's dependence on the quadratic-model equivalence suggests a direct test of the safeguards—run the raw (unsafeguarded) KGD recurrence on a smooth non-quadratic strongly convex function; if it diverges while the safeguarded version converges, the safeguards are the load-bearing ingredient.
- Editorial inference: the long/short BB equivalence opens a two-way street with known BB results, so existing special-case facts about BB could be re-derived for KGD, giving quick testable predictions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to provide a rigorous analysis of Kahan's gradient descent (KGD) step-size strategies. It states that, for strongly convex quadratic objectives, the KGD step-size is equivalent to the long Barzilai-Borwein (BB) step, and that both a newly derived short KGD variant and the long variant (hence BB) converge R-linearly with rate 1 - 1/cond(H). For general unconstrained minimization, it proposes an adaptive framework that uses KGD step-sizes and proves global convergence and local R-linear convergence. Numerical experiments on CUTEst and logistic-regression problems are said to compare the proposed methods against BB variants and other adaptive gradient methods.
Significance. If the theorems can be established, the paper would contribute a useful theoretical bridge between Kahan's KGD step-size and the classical BB step, including a short-step variant with a clean R-linear rate in the quadratic case. The adaptive framework for general objectives could also be practically relevant, provided the convergence proof is valid and the safeguards are explicit. The topic is of interest to the unconstrained-optimization community. However, the submitted full text is unreadable, so none of these contributions can actually be verified from the manuscript as supplied. No code, data, or machine-checked proofs are supplied; the only concrete content is the abstract.
major comments (4)
- [Full Text (all sections)] The supplied body text is unreadable mojibake throughout, with a stray line 'arXiv:2508.06013v2 [physics.acc-ph] 14 Aug 2025' from an unrelated paper embedded in the first page. No equation, algorithm, theorem statement, or proof is legible. This is not a local formatting issue: the paper's central claims—the KGD/long-BB equivalence, the R-linear rate, and the adaptive-framework convergence—cannot be checked in any way. A clean, complete manuscript is a prerequisite for review.
- [Abstract, 'For the general unconstrained minimization...'] The central contribution is the adaptive framework for general unconstrained minimization, but the abstract does not specify the safeguards or parameter choices on which the convergence proof rests. Because the KGD step-size is equivalent to the long BB step only in the quadratic model, and unsafeguarded long-BB steps are known to fail on smooth non-quadratic problems (Dai-Fletcher examples), the global-convergence theorem must depend on a step-size bound, a descent/backtracking condition, or another safeguard. No such condition is stated in the abstract, and the full text provides no readable proof. This is a load-bearing gap for the paper's main claim.
- [Abstract, 'both the long and short KGD ... rate 1 - 1/cond(H)'] The claimed R-linear rate is stated without proof or even definitions of the step-size recursions, the iteration matrices, and the norm/metric in which the rate holds. The asserted mathematical equivalence between KGD and the long BB step is also not demonstrated in any legible form. To verify this theorem, the authors need to present the exact recursions, the error evolution, and the argument leading to the stated rate. As submitted, this is unverifiable.
- [Abstract, numerical experiments] The abstract promises extensive comparisons on CUTEst and logistic-regression problems, but the supplied text contains no tables, performance profiles, or reproducibility details. Since the paper also claims to demonstrate efficiency and robustness, the numerical evidence is part of the support for those claims; its absence in the submitted file prevents assessment of the experimental component.
minor comments (3)
- [Abstract] The notation cond(H) should be defined as the spectral condition number (or explicitly as the ratio of extreme eigenvalues) when the full text is restored.
- [Abstract] The reference to Kahan's 2019 technical report is indicated, but the report's availability should be made explicit (e.g., a stable URL) so readers can consult the original KGD step-size strategy.
- [Full Text] When the manuscript is restored, the authors should ensure that section headings, theorem environments, and equation numbering are visible and consistent, since none can currently be identified.
Circularity Check
No circularity identified; the derivation is self-contained and the corrupted full text prevents any specific reduction from being exhibited.
full rationale
The paper's claimed chain is: Kahan's KGD step-size is shown to be mathematically equivalent to the long BB step in the quadratic model; a short KGD version is derived from that equivalence; for strongly convex quadratics both long and short KGD (and hence BB) converge R-linearly with rate 1 - 1/cond(H); and an adaptive framework is then proposed for general unconstrained minimization with global and local R-linear convergence theorems. None of these steps is circular on the available evidence. The equivalence with the long BB step is a mathematical statement, not a definition that presupposes convergence; the convergence theorem is proved as a consequence of the quadratic model rather than assumed. There is no fitted parameter later renamed as a prediction, and no load-bearing self-citation: the only citation is to Kahan's external technical report. The supplied full text is largely unreadable mojibake and contains the unrelated line 'arXiv:2508.06013v2 [physics.acc-ph] 14 Aug 2025', so the proof of the adaptive-framework convergence cannot be inspected. That is a correctness/verifiability limitation, not a circularity. Without the ability to quote a specific equation that reduces to its own input, the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption The objective is strongly convex with bounded Hessian condition number.
- domain assumption Smoothness and sufficient decrease conditions for global convergence in general optimization.
Cite this review
Pith. "Pith review of Kahan's Automatic Step-Size Control for Unconstrained Optimization." pith.science (2026). https://pith.science/paper/VYH3PXJI
@misc{pith2026250806002,
author = {Pith},
title = {Pith review of: Kahan's Automatic Step-Size Control for Unconstrained Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/VYH3PXJI}},
note = {Machine review of arXiv:2508.06002}
}
abstract
The Barzilai and Borwein (BB) gradient method is one of the most widely-used line-search gradient methods. It computes the step-size for the current iterate by using the information carried in the previous iteration. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed new Gradient Descent (KGD) step-size strategies which iterate the step-size itself by effectively utilizing the information in the previous iteration. In the quadratic model, such a new step-size is shown to be mathematically equivalent to the long BB step, but no rigorous mathematical proof of its efficiency and effectiveness for the general unconstrained minimization is available. In this paper, by this equivalence with the long BB step, we first derive a short version of KGD step-size and show that, for the strongly convex quadratic model with a Hessian matrix $H$, both the long and short KGD step-size (and hence BB step-sizes) gradient methods converge at least R-linearly with a rate $1-\frac{1}{{\rm cond}(H)}$. For the general unconstrained minimization, we further propose an adaptive framework to effectively use the KGD step-sizes; global convergence and local R-linear convergence rate are proved. Numerical experiments are conducted on the CUTEst collection as well as the practical logistic regression problems, and we compare the performance of the proposed methods with various BB step-size approaches and other recently proposed adaptive gradient methods to demonstrate the efficiency and robustness.
Reference graph
Works this paper leans on
-
[1]
J. Barzilai and J. M. Borwein, Two-point step size gradient methods, IMA J. Numer. Anal., 8, ://doi.org/10.1093/imanum/8.1.141
-
[2]
O. Burdakov, Y. H. Dai and N. Huang, Stabilized B arzilai- B orwein method, J. Comput. Math., 37 (2019), 916--936
work page 2019
-
[3]
Y. H. Dai and R. Fletcher, On the asymptotic behaviour of some new gradient methods, Math. Program., Ser. A, 103 (2005), 541--559
work page 2005
-
[4]
Y. H. Dai, W. W. Hager, K. Schittkowski and H. C. Zhang, The cyclic Barzilai-Borwein method for unconstrained optimization, IMA J. Numer. Anal., 26 (2006), 604--627
work page 2006
-
[5]
Y. H. Dai and L. Z. Liao, R -linear convergence of the B arzilai and B orwein gradient method, IMA J. Numer. Anal., 22 (2002), 1--10
work page 2002
-
[6]
E. D. Dolan and J. Mor\'e, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), 201--213
work page 2002
-
[7]
R. Fletcher, Optimization and control with applications, Springer, Boston, MA, 2005, chapter On the Barzilai-Borwein method, 235--256
work page 2005
-
[8]
N. I. M. Gould, D. Orban and P. L. Toint, CUTEst : a constrained and unconstrained testing environment with safe threads for mathematical optimization, Comput. Optim. Appl., 60 (2015), 545--557
work page 2015
Show all 18 references
-
[9]
Grippo, F
L. Grippo, F. Lampariello and S. Lucidi, A nonmonotone line search technique for N ewton's method, SIAM J. Numer. Anal., 23 (1986), 707--716
1986
-
[10]
Z. Gu, K. Jin, Y. Meng, L. Xue and L.-H. Zhang, On K ahan's automatic step-size control and an anadromic gradient descent iteration, Optimization, 74:1 (2025), 189-218
2025
-
[11]
Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019, ://people.eecs.berkeley.edu/ wkahan/26Sept19.pdf
W. Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019, ://people.eecs.berkeley.edu/ wkahan/26Sept19.pdf
2019
-
[12]
W. Kahan, A Numerical Analyst thinks about ``Deep Learning" and Artificial Intelligence, Technical report, University of California, Berkeley CA, USA, 2019, ://people.eecs.berkeley.edu/ wkahan/7Nov18.pdf
2019
-
[13]
Latafat, A
P. Latafat, A. Themelis, L. Stella and P. Patrinos, Adaptive proximal algorithms for convex optimization under local L ipschitz continuity of the gradient , Technical report, arXiv:2301.04431, 2023
2023 arXiv
-
[14]
Li and R
D. Li and R. Sun, On a Faster R -Linear Convergence Rate of the B arzilai- B orwein Method , arxiv:2101.00205, 2021
2021 arXiv
-
[15]
Malitsky and K
Y. Malitsky and K. Mishchenko, Adaptive gradient descent without descent, in Proceedings of the 37th International Conference on Machine Learning , 2020, 6702--6712
2020
-
[16]
Nocedal and S
J. Nocedal and S. Wright, Numerical Optimization, 2nd edition, Springer, New York, 2006
2006
-
[17]
Raydan, On the Barzilai and Borwein choice of steplength for the gradient method, IMA J
M. Raydan, On the Barzilai and Borwein choice of steplength for the gradient method, IMA J. Numer. Anal., 13 (1993), 321--326
1993
-
[18]
Raydan, The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem, SIAM J
M. Raydan, The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem, SIAM J. Optim., 7 (1997), 26--33
1997
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.