REVIEW 5 major objections 4 minor 1 cited by
Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization
T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Dividing every step size by 1/(1+δ) makes long-step and accelerated gradient methods robust to relative gradient error.
desk verdict Useful, honest paper on relative inexactness; the shortening factor is plausible and partly supported by PEP curves, but the headline claim that FGM is robust rests on one specific corruption model and needs stronger empirical support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the performance estimation problem (PEP), a reformulation of worst-case analysis as a semidefinite program that computes tight convergence rates for a given algorithm and function class; the relative inexactness condition ‖d_k−∇f(x_k)‖≤δ‖∇f(x_k)‖ is added as an extra constraint. The second key object is the shortening factor 1/(1+δ), which rescales every step size of a schedule. It is motivated by the observation that constant-step gradient descent with step h>2/(1+δ) diverges on the one-dimensional quadratic f(x)=Lx²/2, while the standard proof of convergence on smooth convex functions survives as long as h≤2/(1+δ). Applying this uniform rescaling to the dynamic, Silver, and FGM schedules is the semi-heuristic repair whose worst-case and empirical effects the paper measures.
What would settle it
Run the shortened Silver and dynamic schedules on a different smooth convex problem with an adversarial relative error at δ=0.5 and compare accuracy against the unshortened versions; if shortening no longer improves accuracy, or if PEP worst-case rates for shortened FGM exceed those of the unshortened method, the uniform-repair claim loses support.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the factor 1/(1+δ) acts as a uniform repair for step-size schedules under relative gradient inexactness, and that accelerated methods are much more robust in practice than their worst-case guarantees suggest. The performance-estimation analysis shows that without shortening, Silver and dynamic schedules degrade sharply as δ increases, and even FGM's rate worsens; with every step size divided by 1+δ, worst-case rates improve substantially. In the compressed-gradient experiment, no method is significantly harmed by mantissa truncation, and the exact-case ordering of methods persists. In the adversarial-corruption experiment, FGM keeps high accuracy even at large δ, and shortened Silver steps rival FGM for δ≤0.3, while the shortening factor slightly hurts constant-step and FGM. The paper concludes with a robustness ranking that recommends FGM, and shortened Silver steps when a memoryless method is wanted.
Load-bearing premise
The shortening factor is derived from a divergence threshold for constant-step gradient descent, and the paper assumes that dividing every step size of the Silver, dynamic, and FGM schedules by 1+δ preserves or restores their convergence; this is validated experimentally only on one logistic-regression dataset with six starting points.
Editorial extensions
If this is right
- Shortening by 1/(1+δ) turns the theoretically fragile long-step schedules into methods with usable worst-case rates, making them candidates for memory-limited inexact settings.
- Compressed gradient descent with mantissas truncated to a few bits can reduce gradient storage by more than a factor of three without harming empirical convergence, since the induced relative error stays below the robustness threshold.
- The practical ordering FGM beats Silver beats dynamic and constant-step (matching $O(1/N^2)$, $O(1/N^{1.27})$, and $O(1/N)$) persists under compression, so method choice in inexact settings can follow the exact-case ranking.
- Under adversarial corruption the paper recommends FGM as the default, with shortened Silver steps as the memoryless alternative.
- Worst-case analysis, although pessimistic for accelerated methods, retains value because it is what suggested the shortening factor that later proved useful empirically.
Reading between the lines
- If the shortening heuristic transfers beyond logistic regression, the same 1/(1+δ) rescaling could be applied to other fixed-schedule methods, such as restarted or adaptive variants, with similar robustness gains; this is a testable extension the paper does not perform.
- The large empirical gap between FGM's observed robustness and its worst-case rate suggests the relative-inexactness model is pessimistic for realistic compression, pointing toward stochastic or average-case analyses of compressed gradients.
- A natural next experiment is to measure the actual δ induced by mantissa truncation in neural-network training; the paper's bound δ=(1/2)^{nbit+1} likely overstates typical error, implying that even larger memory savings could be safe.
- The stability threshold h≤2/(1+δ) offers a rule of thumb for any gradient method: keep effective step sizes below the inexactness-adjusted limit, which may also explain why some adaptive methods tolerate low-precision gradients.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies the behavior of four first-order optimization methods—constant step-size gradient descent, dynamic step sizes, Silver step sizes, and Nesterov's fast gradient method (FGM)—when gradients are subject to relative inexactness of the form ||d_k - ∇f(x_k)|| ≤ δ||∇f(x_k)||. Using the Performance Estimation Problem (PEP) methodology, the authors numerically compute worst-case convergence rates in squared gradient norm for N = 50 iterations and find that long-step and accelerated methods degrade as δ grows. They then propose a 'shortening factor' 1/(1+δ), motivated by a divergence threshold for constant-step gradient descent on a quadratic, and show via PEP that shortened schedules have substantially better worst-case rates. The empirical part considers a logistic regression problem with two inexactness models: compressed mantissa gradients and an 'adversarial' corruption that adds the longest perturbation pointing away from the optimizer. The reported experiments indicate that compression is largely harmless, that FGM is surprisingly robust under the corruption model, and that shortening helps the long-step methods in that model. The paper recommends FGM, or shortened Silver steps when a memoryless method is needed.
Significance. If the claims are validated, the paper offers practically relevant guidance for using low-precision or compressed gradients on GPUs, and it demonstrates how PEP can be used to diagnose robustness issues beyond standard convergence-rate analysis. The authors release their code, which is a genuine strength, and the shortening-factor idea is natural and potentially useful. However, the central empirical claims are currently supported by a narrow experimental base—one dataset shown, six initial points, no error bars—and by a corruption model that is adversarial only in a limited sense. The PEP results are numerical SDP outcomes rather than certified proofs. The paper's contribution is therefore promising but not yet fully established; the gap between the strength of the recommendations and the evidence is the main issue.
major comments (5)
- [Section 3.2, Figures 2a and 4] The headline empirical finding that FGM is 'much more robust than expected' and the recommendation to prefer FGM rest entirely on the corruption model that adds the longest feasible perturbation pointing away from x*. That perturbation is one specific point on the boundary of the relative-inexactness ball and is not chosen adversarially with respect to FGM; for a method whose extrapolated point y_k can overshoot x*, this direction may be comparatively benign. The PEP worst-case rates in Figure 2a, which are adversarial over both the function and the error sequence, show FGM degrading substantially for δ ≥ 0.4. To make the robustness claim load-bearing, the authors should add corruption directions that are adversarial for each method (for example, aligned with the momentum or extrapolation direction) or report the worst performance over randomized boundary perturbations.
- [Section 2, shortening factor paragraph] The shortening factor 1/(1+δ) is introduced from a divergence threshold h > 2/(1+δ) for constant-step gradient descent on the quadratic f(x)=Lx^2/2, citing references [15,16]. Its extension to the Silver, dynamic, and FGM schedules is an assumption rather than a derived result. The PEP rates in Figure 2b provide numerical support for worst-case rates, but the practical recommendation that shortened Silver is a robust memoryless method is validated only on one logistic regression dataset, with only a mention of a phishing dataset and no shown results. The authors should either provide additional datasets, give a theoretical justification for why the factor transfers to these schedules, or explicitly restrict the claim to the tested setting.
- [Abstract and Sections 3.1–3.2] The abstract states that 'the shortening factor significantly helps the long-step methods,' but this is observed only in the adversarial corruption experiment of Section 3.2. In the compressed-gradient experiment of Section 3.1, the text explicitly says that shortened schedules 'do not seem to make much difference' (page 9). The conclusions should distinguish between the two inexactness models, because the current wording overgeneralizes the evidence.
- [Section 2, Figures 2a–2b] The paper refers to the PEP results as 'exact convergence rates' and states that long-step and accelerated methods are 'shown to be theoretically not robust.' These rates are outputs of a semidefinite program solved numerically with MOSEK, and no solver tolerance or certificate verification is reported. The authors should state the numerical nature of these results and either verify them with a rigorous post-processing step or soften the wording from 'shown' to something like 'numerically observed.'
- [Section 3, Figures 3 and 4] All empirical curves are means over six initial points on a single dataset, with no error bars, no per-seed results, and no statistical tests. The claims that compression 'does not seem to cause any significant harm' and that shortening 'greatly improves' performance are quantitative; without variability information, the reader cannot assess how robust these conclusions are to initialization or to the choice of dataset. The authors should report error bars or additional dataset results before drawing practical recommendations.
minor comments (4)
- [Figures 3 and 4] The axis labels in the submitted figures are visually corrupted (for example, 'Mean of min k ∈[N] --∇(f(x_k)--' and 'Mean of be() )rainin accuracie('). The figures should be regenerated with proper mathematical notation.
- [Section 3] The smoothness constant L used for normalized step sizes is estimated numerically, whereas the theoretical PEP analysis assumes L is known exactly. The paper should state clearly that the experiments use an estimate, and discuss how errors in this estimate interact with the relative inexactness level δ.
- [Table 1] The qualitative ratings (✗, ∼, ✓) in Table 1 are presented without explicit thresholds or a precise definition of when a method is considered 'bad,' 'intermediate,' or 'good.' A short criterion would make the table more informative.
- [Section 2] The derivation of the divergence threshold h > 2/(1+δ) referenced to [15,16] is not reproduced. Since this threshold is the motivation for the central shortening factor, a brief proof sketch in an appendix would improve the self-containedness of the paper.
Circularity Check
No significant circularity: PEP rates are exact reformulations, the shortening factor is an explicitly heuristic prior step, and experiments do not feed back into the analysis.
full rationale
The paper's theoretical contribution is the computation of exact worst-case convergence rates via Performance Estimation, by solving the stated PEP with the relative inexactness constraint (2) and fixed step-size schedules. These rates are outputs of semidefinite programs, not fitted to the empirical results. The shortening factor 1/(1+δ) is derived from a parameter-free divergence threshold for constant-step gradient descent on f(x)=L x^2/2 and from a proof-structure preservation observation, both cited to the authors' prior work [15,16]; this is a self-citation, but it is not load-bearing in a circular sense because the cited facts are independently checkable mathematical statements whose assumptions do not include the target result, and the factor is explicitly labelled 'semi-heuristic' before being evaluated on new schedules by PEP and on a separate logistic-regression experiment. The empirical claims about compression and adversarial corruption are observations under specified perturbation models, not predictions generated by fitting parameters to those same observations; no fitted input is renamed as a prediction. The concern that the adversarial corruption model is only one non-algorithm-specific direction is an external-validity caveat, not a circularity. Accordingly, the derivation chain is self-contained and no step reduces by construction to its own input.
Assumptions & free parameters
free parameters (1)
- shortening factor 1/(1+δ) =
1/(1+δ) for δ∈[0,1)
assumptions (4)
- domain assumption Relative inexactness model: ‖d_k−∇f(x_k)‖≤δ‖∇f(x_k)‖ with δ∈[0,1)
- standard math Smooth convex interpolation conditions from [13] characterize the function class for the PEP
- domain assumption Estimated Lipschitz constant L_N from the preliminary run is a valid upper bound for the iterates of the inexact methods
- ad hoc to paper The shortening factor preserves convergence behavior of silver/dynamic/FGM
Cite this review
Pith. "Pith review of Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization." pith.science (2026). https://pith.science/paper/FAJLNDFK
@misc{pith2026250609730,
author = {Pith},
title = {Pith review of: Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/FAJLNDFK}},
note = {Machine review of arXiv:2506.09730}
}
read the original abstract
This work assesses both empirically and theoretically, using the performance estimation methodology, how robust different first-order optimization methods are when subject to relative inexactness in their gradient computations. Relative inexactness occurs, for example, when compressing the gradient using fewer bits of information, which happens when dealing with large-scale problems on GPUs. Three major families of methods are analyzed: constant step gradient descent, long-step methods, and accelerated methods. The latter two are first shown to be theoretically not robust to inexactness. Then, a semi-heuristic shortening factor is introduced to improve their theoretical guarantees. All methods are subsequently tested on a concrete inexact problem, with two different types of relative inexactness, and it is observed that both accelerated methods are much more robust than expected, and that the shortening factor significantly helps the long-step methods. In the end, all shortened methods appear to be promising, even in this inexact setting.
Figures
Forward citations
Cited by 1 Pith paper
-
Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions
For smooth convex functions, relatively inexact gradient descent has a three-regime worst-case one-step rate, and the largest provably safe stepsize is 2/(1+delta).
Reference graph
Works this paper leans on
-
[1]
Neda Abdelhamid. Website Phishing. UCI Machine Learning Repository, 2014. DOI: https://doi.org/10.24432/C5B301
-
[2]
Acceleration by st epsize hedging: Silver stepsize schedule for smooth convex optimization
Jason M Altschuler and Pablo A Parrilo. Acceleration by st epsize hedging: Silver stepsize schedule for smooth convex optimization. Mathematical Programming, 2024
work page 2024
-
[3]
The MOSEK Python Fusion API manual
MOSEK ApS. The MOSEK Python Fusion API manual. Version 11.0. , 2025
2025
-
[4]
Smooth optimization with approx imate gradient
Alexandre d’Aspremont. Smooth optimization with approx imate gradient. SIAM Journal on Optimization , 19(3):1171–1183, 2008
work page 2008
-
[5]
Etienne De Klerk, Francois Glineur, and Adrien B Taylor. W orst-case convergence analysis of inexact gradient and newton methods through sem idefinite program- ming performance estimation. SIAM Journal on Optimization , 30(3):2053–2082, 2020
work page 2020
-
[6]
First-order methods of smooth convex optimization with inexact oracle
Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming , 146:37–75, 2014
work page 2014
-
[7]
Performance of first-order m ethods for smooth convex minimization: a novel approach
Yoel Drori and Marc Teboulle. Performance of first-order m ethods for smooth convex minimization: a novel approach. Mathematical Programming, 145(1-2):451– 482, 2014
work page 2014
-
[8]
Pepit: computer-assisted worst-case analyses of first-order optimization methods in python
Baptiste Goujaud, Céline Moucer, François Glineur, Juli en M Hendrickx, Adrien B Taylor, and Aymeric Dieuleveut. Pepit: computer-assisted worst-case analyses of first-order optimization methods in python. Mathematical Programming Compu- tation, 16(3):337–367, 2024
work page 2024
Show all 17 references
-
[9]
Sign language digits dataset, 2017
Arda Mavi. Sign language digits dataset, 2017
2017
-
[10]
A method for unconstrained convex minim ization problem with the rate of convergence O(1/k 2)
Yurii Nesterov. A method for unconstrained convex minim ization problem with the rate of convergence O(1/k 2). Doklady AN USSR , 269:543–547, 1983
1983
-
[11]
Introductory lectures on convex optimization: A basic cour se, vol- ume 87
Yurii Nesterov. Introductory lectures on convex optimization: A basic cour se, vol- ume 87. Springer Science & Business Media, 2013
2013
-
[12]
Exact worst-case per- formance of first-order methods for composite convex optimi zation
Adrien B Taylor, Julien M Hendrickx, and François Glineu r. Exact worst-case per- formance of first-order methods for composite convex optimi zation. SIAM Journal on Optimization , 27(3):1283–1313, 2017
2017
-
[13]
Smooth strongly con- vex interpolation and exact worst-case performance of first -order methods
Adrien B Taylor, Julien M Hendrickx, and François Glineu r. Smooth strongly con- vex interpolation and exact worst-case performance of first -order methods. Math- ematical Programming, 161:307–345, 2017
2017
-
[14]
An elementary approa ch to tight worst case complexity analysis of gradient based methods
Marc Teboulle and Yakov Vaisbourd. An elementary approa ch to tight worst case complexity analysis of gradient based methods. Mathematical Programming, 201(1):63–96, 2023
2023
-
[15]
Tight convergence analysis of exact a nd inexact gradient meth- ods with constant and silver schedules
Pierre Vernimmen. Tight convergence analysis of exact a nd inexact gradient meth- ods with constant and silver schedules. Master’s thesis, UC Louvain, 2024
2024
-
[16]
Worst-case conv ergence analysis of rel- atively inexact gradient descent on smooth convex function s
Pierre Vernimmen and François Glineur. Worst-case conv ergence analysis of rel- atively inexact gradient descent on smooth convex function s. arXiv:2406.17506, 2025
2025
-
[17]
Robust distributed gradi ent descent to corrup- tion over noisy channels
Shuche Wang and Vincent YF Tan. Robust distributed gradi ent descent to corrup- tion over noisy channels. In 2024 IEEE International Symposium on Information Theory (ISIT) , pages 2520–2525. IEEE, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.