REVIEW 2 major objections 4 minor 41 references
An Adaptive $s$-step Conjugate Gradient Algorithm with Dynamic Basis Updating
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An adaptive s-step conjugate gradient algorithm that rebuilds its Newton or Chebyshev bases from incremental Ritz-value estimates meets user-specified accuracy with fewer global synchronizations than the previous adaptive s-step method.
desk verdict A genuinely useful algorithmic improvement with an honest experimental section, but the new automatic safety parameter rests on a bound that goes in the wrong direction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is the s-step Krylov basis matrix $\widehat{Y}_{k,s}$ and its Gram matrix $G_{k,s}=\widehat{Y}_{k,s}^T\widehat{Y}_{k,s}$, whose condition number governs the finite-precision residual gap through the bound (2.10). The mechanism that keeps that condition number under control is the incremental estimation of the extreme Ritz values from the bidiagonal factor $L_i$ of the Lanczos matrix, combined with the scalar recurrence for $\psi_{m+j+1}$ that produces the adaptive constant $c_{m+j+1}$ from (4.6). These ingredients set the recurrence coefficients $\theta$, $\gamma$, $\mu$ for the Newton or Chebyshev bases and determine when the inner loop should stop, using the largest residual norm seen so far in the current outer loop.
What would settle it
On a small symmetric positive definite matrix with an isolated smallest eigenvalue, record $\widetilde{\lambda}_{\min}$ from Algorithm 4 at every inner iteration before convergence and compare it with the true $\lambda_{\min}(A)$ from a high-precision eigensolver. If $\widetilde{\lambda}_{\min} > \lambda_{\min}(A)$ at any such iteration, the premise behind the bound (4.6) is violated; one can then check whether the residual gap exceeds the bound (2.10) or the algorithm fails to reach the requested tolerance while standard CG succeeds.
Extended reading notes
Core claim
The paper's central claim is that incremental estimates $\widetilde{\lambda}_{\min}$ and $\widetilde{\lambda}_{\max}$, obtained from the bidiagonal Cholesky factor of the Lanczos tridiagonal matrix as the CG recurrences run, are sufficient to keep the s-step bases well conditioned and to set the inner-loop stopping rule automatically. Concretely, the algorithm builds Newton parameters by (2.7) or simplified Chebyshev parameters by (2.8), and sets $c_{m+j+1}=\max\{1, \widetilde{\lambda}_{\max}(\psi_{m+j+1}/\widetilde{\lambda}_{\min})^{1/2}\}$ with $\psi_{m+j+1}=\psi_{m+j}/(\psi_{m+j}+\beta_{m+j})$, which approximates the exact ratio $\xi_{m+j+1}=\|A\|\,\|x-\widehat{x}_{m+j+1}\|/\|\widehat{r}_{m+j+1}\|$. With this choice the paper argues that the basis condition-number bound can be enforced without user-supplied spectral information or a heuristic constant, and the numerical experiments support the claim that the resulting algorithm attains the prescribed accuracy with fewer global synchronizations than the earlier approach.
Load-bearing premise
The load-bearing premise is that the incremental Ritz-value estimates stay accurate enough in finite-precision s-step arithmetic that $\widetilde{\lambda}_{\min}$ never overestimates the true smallest eigenvalue, because equation (4.6) treats $\widetilde{\lambda}_{\min}$ as a lower bound when setting the stopping constant.
Editorial extensions
If this is right
- For the tested matrices, the improved algorithm converged to the prescribed accuracy in every case, while fixed s-step CG and the old adaptive algorithm frequently stagnated or diverged at $\sigma=10$ or $\sigma=15$.
- The number of outer-loop iterations, a proxy for global synchronizations, decreased as the maximum step size $\sigma$ increased, and in several tests dropped by more than a factor of 12 relative to HSCG.
- The total number of inner iterations needed by the improved algorithm stayed roughly constant as $\sigma$ grew, so choosing $\sigma$ too large no longer inflates the iteration count.
- The heuristic choice of $c_k$ in the previous algorithm is replaced by an automatic estimate, removing one user-supplied parameter and the need for a priori spectral information.
- Newton and Chebyshev bases both produced improved convergence behavior, but the paper does not claim either basis is uniformly better; the choice remains problem-dependent.
Reading between the lines
- If the finite-precision reliability of the incremental Ritz estimates were proven inside the s-step inner loop, the same dynamic basis restart idea could be transferred to s-step GMRES, s-step BiCGSTAB, or s-step Lanczos eigensolvers, where basis conditioning plays the same role.
- The adaptive formula for $c_{m+j+1}$ is essentially an estimate of the error-to-residual ratio; a direct test of its robustness would compare $\widetilde{\lambda}_{\min}$ from Algorithm 4 against the true smallest eigenvalue early in the iteration, because an overestimate would weaken the bound (4.6).
- The observed near-independence of total iterations from $\sigma$ suggests that an autotuner could safely choose a large maximum step size and let the adaptive loop trim $s$, simplifying parameter selection.
- A high-performance parallel implementation on a realistic machine is the natural next test: the paper's experiments are small and report synchronization counts as a proxy rather than wall-clock time.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an improved adaptive s-step conjugate gradient algorithm for SPD linear systems. It uses incremental estimates of the largest and smallest Ritz values, following Meurant and Tichý, to dynamically update Newton or Chebyshev basis parameters and to replace the previously user-supplied constant c_k by an automatically updated quantity c_{m+j+1} based on Eq. (4.6). The authors present numerical experiments on several small SuiteSparse matrices showing that the improved algorithm often converges to the requested accuracy with fewer outer iterations (global synchronizations) than fixed s-step CG and the previous adaptive s-step CG, while also honestly reporting a difficult case (nos1).
Significance. If the claims hold, the contribution is practically relevant: it removes one heuristic parameter from adaptive s-step CG and improves basis conditioning without user-supplied spectral information. The paper's strengths include a clear experimental setup, comparison against both classical HSCG and the prior adaptive algorithm, a range of tolerances and matrix sizes, and an honest discussion of cases where the improved method is not beneficial. The main weakness is that the derivation of the safeguard c_{m+j+1} in Section 4 contains a bound-direction error, so the central reliability claim is not theoretically substantiated; the experiments are encouraging but cannot by themselves close this gap.
major comments (2)
- [Section 4, Eq. (4.6) and Algorithm 5, lines 20–25] The substitution mu = tilde_lambda_min is invalid for the stated purpose. The bound quoted from [33] requires 0 < mu <= lambda_min(A), but in exact arithmetic the smallest Ritz value lambda_min(T_i) of the Lanczos tridiagonal matrix satisfies lambda_min(T_i) >= lambda_min(A) by the Rayleigh–Ritz (interlacing) property. Hence tilde_lambda_min in (4.2) is an upper bound, not a lower bound, for lambda_min(A). Consequently the quantity tilde_xi in (4.6) can be smaller than the true ratio xi in (4.4), and the break test (4.7) can allow an s_k that violates the sufficient condition (3.1). The claim that the improved algorithm attains the user-specified accuracy is therefore not substantiated. The experiments do not test this safeguard directly, because Figure 2 plots tilde_xi against the global bounds 1 and kappa(A), not against the true xi.
- [Section 4 (Algorithms 3–4) and Section 5] The incremental Ritz estimates are computed from the alpha and beta coefficients generated inside the finite-precision s-step inner loop (Algorithm 5, lines 13–17). No finite-precision analysis is provided for these estimates in the s-step setting, where the computed coefficients and coordinate recurrences can be amplified by kappa(Y_hat_{k,s}). Since both the basis parameter update (line 29) and the safeguard c_{m+j+1} (line 23) depend on these estimates, the paper does not establish that the dynamic bases are better conditioned than monomial bases in finite precision, nor that (4.7) enforces (3.1). I would ask for an explicit perturbation analysis, even a preliminary one, or for a clear statement that the dynamic basis update is heuristic and outside the scope of the theoretical guarantee.
minor comments (4)
- [Section 5, Tables 2–7] The value of c_k used for the old adaptive s-step CG baseline is not stated; since this heuristic strongly affects the number of inner iterations, please specify it for reproducibility.
- [Algorithm 5, line 1] The initial value c_1 = epsilon^{-1/2} is not derived or explained; please justify this choice or label it as a default heuristic.
- [Section 4, before Eq. (4.6)] The phrase 'approximately bound' is ambiguous: a bound is either valid or not. Given the issue raised above, please rephrase this to say explicitly that the quantity is heuristic, or provide a certified upper bound.
- [Section 5.3] The statement that the total number of iterations is 'independent of sigma' is not strictly true for all reported cases; for example, in Table 7 (nos1, epsilon* = 1e-6, Newton basis) the totals are 1217, 1290, and 1086 for sigma = 5, 10, 15. Please qualify this claim.
Circularity Check
No significant circularity: all claimed improvements are validated by external convergence benchmarks, and the Ritz-based parameter updates are algorithmic bootstrapping rather than fitted predictions.
full rationale
The paper's central claims—better numerical behavior and fewer synchronizations—are supported by experiments measuring true relative residuals against HSCG, fixed s-step CG, and the author's earlier adaptive algorithm. Nothing in the derivation is equivalent to its own inputs by construction. The new ingredients are (i) Newton/Chebyshev basis parameters updated from incremental extreme Ritz estimates via Meurant–Tichý, and (ii) automatic setting of c_{m+j+1} in (4.6) from λ̃max, λ̃min, and ψ. These quantities are produced by the CG/Lanczos recurrences, not by fitting a parameter to the target accuracy, so using them inside the algorithm is legitimate bootstrapping. Condition (3.1) is inherited from the author's published rounding-error analysis; citing one's prior work is normal cumulative research and is not load-bearing in the sense of being unverified or identical to the result. The skeptical issue that λmin(T_i) is an upper bound on λmin while (4.6) requires 0 < μ ≤ λmin is a genuine correctness/bound-direction concern, but it is not circularity: no equation reduces to a pre-supplied answer. The paper also states honest limitations (no demonstrated performance benefit over HSCG yet; unexplained behavior on nos1), consistent with a non-circular empirical study.
Assumptions & free parameters
free parameters (3)
- initial c_1 =
ε^{-1/2}
- maximum allowed block size σ =
5, 10, 15 in experiments
- maximum basis growth factor f =
f = σ in experiments
assumptions (5)
- domain assumption A is symmetric positive definite.
- domain assumption The residual gap bound (2.10) from [9] accurately models the amplification of local roundoff by κ(Y_hat_k,s).
- domain assumption The incremental norm estimates of Meurant and Tichý (Algorithms 3 and 4) give approximations of λ_max and λ_min that remain useful when computed inside finite-precision s-step CG iterations.
- domain assumption The quantity ~ξ in (4.6) is a valid replacement for the ratio ξ = ||A|| ||x̂ - x|| / ||r̂||, i.e., μ = ~λ_min is a lower bound for λ_min.
- domain assumption Number of outer loop iterations is a meaningful proxy for the number of global synchronizations.
Cite this review
Pith. "Pith review of An Adaptive $s$-step Conjugate Gradient Algorithm with Dynamic Basis Updating." pith.science (2026). https://pith.science/paper/J7F5RH2O
@misc{pith2026190804081,
author = {Pith},
title = {Pith review of: An Adaptive $s$-step Conjugate Gradient Algorithm with Dynamic Basis Updating},
year = {2026},
howpublished = {\url{https://pith.science/paper/J7F5RH2O}},
note = {Machine review of arXiv:1908.04081}
}
abstract
The adaptive $s$-step CG algorithm is a solver for sparse, symmetric positive definite linear systems designed to reduce the synchronization cost per iteration while still achieving a user-specified accuracy requirement. In this work, we improve the adaptive $s$-step conjugate gradient algorithm by use of iteratively updated estimates of the largest and smallest Ritz values, which give approximations of the largest and smallest eigenvalues of $A$, using a technique due to Meurant and Tich{\' y} [G. Meurant and P. Tich{\' y}, Numer. Algs. (2018), pp.~1--32]. The Ritz value estimates are used to dynamically update parameters for constructing Newton or Chebyshev polynomials so that the conditioning of the $s$-step bases can be continuously improved throughout the iterations. These estimates are also used to automatically set a variable related to the ratio of the sizes of the error and residual, which was previously treated as an input parameter. We show through numerical experiments that in many cases the new algorithm improves upon the previous adaptive $s$-step approach both in terms of numerical behavior and reduction in number of synchronizations.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[33]
G. Meurant and P. Tich´y, Approximating the extreme Ritz values and upper bounds for the A-norm of the error in CG , Numer. Algs., (2018), pp. 1–32
work page 2018
-
[1]
Z. Bai, D. Hu, and L. Reichel , A Newton basis GMRES implementation , IMA J. Numer. Anal., 14 (1994), pp. 563–581
work page 1994
-
[2]
A. Bouras and V. Frayss ´e, A relaxation strategy for inexact matrix-vector products for Krylov methods , Tech. Rep. CERFACS TR/PA/00/15, European Centre for Research and Advanced Training in Scientific Computation, 2000
work page 2000
-
[3]
, Inexact matrix-vector products in Krylov methods for solving linear sys- tems: a relaxation strategy, SIAM J. Matrix Anal. Appl., 26 (2005), pp. 660–678
work page 2005
-
[4]
D. Calvetti, G. H. Golub, and L. Reichel, An adaptive Chebyshev iterative method for nonsymmetric linear systems based on modified moments, Numerische Mathematik, 67 (1994), pp. 21–40
work page 1994
-
[5]
D. Calvetti and L. Reichel , On the evaluation of polynomial coefficients , Numer. Algs., 33 (2003), pp. 153–161
work page 2003
-
[6]
E. Carson and J. Demmel , A residual replacement strategy for improving the maximum attainable accuracy of s-step Krylov subspace methods, SIAM J. Matrix Anal. Appl., 35 (2014), pp. 22–43
work page 2014
-
[7]
E. Carson and J. W. Demmel , Accuracy of the s-step Lanczos method for the symmetric eigenproblem in finite precision , SIAM J. Matrix Anal. Appl., 36 (2015), pp. 793–819
work page 2015
Show all 41 references
-
[8]
E. C. Carson, Communication-avoiding Krylov subspace methods in theory and practice, PhD thesis, U.C. Berkeley, 2015
2015
-
[9]
E. C. Carson, The adaptive s-step conjugate gradient method, SIAM J. Matrix Anal. Appl., 39 (2018), pp. 1318–1338
2018
-
[10]
Chronopoulos and C
A. Chronopoulos and C. Gear,s-step iterative methods for symmetric linear systems, J. Comput. Appl. Math., 25 (1989), pp. 153–168
1989
-
[11]
T. A. Davis and Y. Hu , The University of Florida sparse matrix collection , ACM Transactions on Mathematical Software (TOMS), 38 (2011), pp. 1:1–1:25
2011
-
[12]
de Sturler, A parallel variant of GMRES(m) , in Proc
E. de Sturler, A parallel variant of GMRES(m) , in Proc. 13th IMACS World Congress Comput. Appl. Math., vol. 9, IMACS, Criterion Press, 1991
1991
-
[13]
De Sturler and H
E. De Sturler and H. A. van der Vorst, Reducing the effect of global com- munication in GMRES(m) and CG on parallel distributed memory computers , Appl. Numer. Math., 18 (1995), pp. 441–459
1995
-
[14]
Demmel, M
J. Demmel, M. Hoemmen, M. Mohiyuddin, and K. Yelick, Avoiding com- munication in sparse matrix computations , in IEEE Int. Symp. Parallel Dist. Proc., IEEE, 2008, pp. 1–12. 24
2008
-
[15]
Dongarra et al., The international exascale software project roadmap, Int
J. Dongarra et al., The international exascale software project roadmap, Int. J. High Perf. Comput. Appl., 25 (2011), pp. 3–60
2011
-
[16]
Dongarra, M
J. Dongarra, M. A. Heroux, and P. Luszczek , High-performance conjugate-gradient benchmark: A new metric for ranking high-performance com- puting systems, Int. J. High Perf. Comput. Appl., 30 (2016), pp. 3–10
2016
-
[17]
Erhel , A parallel GMRES version for general sparse matrices , Electronic Trans
J. Erhel , A parallel GMRES version for general sparse matrices , Electronic Trans. Numer. Anal., 3 (1995), pp. 160–176
1995
-
[18]
Gautschi, The condition of polynomials in power form , Math
W. Gautschi, The condition of polynomials in power form , Math. Comput., 33 (1979), pp. 343–352
1979
-
[19]
Ghysels, T
P. Ghysels, T. J. Ashby, K. Meerbergen, and W. Vanroose , Hiding global communication latency in the GMRES algorithm on massively parallel ma- chines, SIAM J. Sci. Comput., 35 (2013), pp. C48–C71
2013
-
[20]
Ghysels and W
P. Ghysels and W. Vanroose , Hiding global synchronization latency in the preconditioned conjugate gradient algorithm , Parallel Comput., 40 (2014), pp. 224–238
2014
-
[21]
Greenbaum, Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences, Lin
A. Greenbaum, Behavior of slightly perturbed Lanczos and conjugate-gradient recurrences, Lin. Alg. Appl., 113 (1989), pp. 7–63
1989
-
[22]
Matrix Anal
, Estimating the attainable accuracy of recursively computed residual meth- ods, SIAM J. Matrix Anal. Appl., 18 (1997), pp. 535–551
1997
-
[23]
Gutknecht and Z
M. Gutknecht and Z. Strako ˇs, Accuracy of two three-term and three two- term recurrences for Krylov space solvers, SIAM J. Matrix Anal. Appl., 22 (2000), pp. 213–229
2000
-
[24]
Heroux, R
M. Heroux, R. Bartlett, V. H. R. Hoekstra, J. Hu, T. Kolda, R. Lehoucq, K. Long, R. Pawlowski, E. Phipps, A. Salinger, H. Thorn- quist, R. Tuminaro, J. Willenbring, and A. Williams , An Overview of Trilinos, Tech. Rep. SAND2003-2927, Sandia National Laboratories, 2003
2003
-
[25]
M. R. Hestenes and E. Stiefel , Methods of conjugate gradients for solving linear systems, J. Research Nat. Bur. Standards, 49 (1952), pp. 409–436
1952
-
[26]
Hindmarsh and H
A. Hindmarsh and H. Walker, Note on a Householder implementation of the GMRES method, Tech. Rep. UCID-20899, Lawrence Livermore National Lab., CA., 1986
1986
-
[27]
Hoemmen, Communication-avoiding Krylov subspace methods , PhD thesis, U.C
M. Hoemmen, Communication-avoiding Krylov subspace methods , PhD thesis, U.C. Berkeley, 2010
2010
-
[28]
Imberti and J
D. Imberti and J. Erhel , Varying the s in your s-step GMRES , Electronic Trans. Numer. Anal., 47 (2017), pp. 206–230
2017
-
[29]
W. D. Joubert and G. F. Carey , Parallelizable restarted iterative methods for nonsymmetric linear systems. Part I: Theory , Int. J. Comput. Math., 44 (1992), pp. 243–267
1992
-
[30]
Liesen and Z
J. Liesen and Z. Strakoˇs, Krylov Subspace Methods: Principles and Analysis , Oxford University Press, 2013. 25
2013
-
[31]
T. A. Manteuffel, Adaptive procedure for estimating parameters for the non- symmetric Tchebychev iteration , Numerische Mathematik, 31 (1978), pp. 183– 208
1978
-
[32]
Meurant and Z
G. Meurant and Z. Strakoˇs, The Lanczos and conjugate gradient algorithms in finite precision arithmetic , Acta Numerica, 15 (2006), pp. 471–542
2006
-
[34]
Philippe and L
B. Philippe and L. Reichel , On the generation of Krylov subspace bases , Appl. Numer. Math., 62 (2012), pp. 1171–1186
2012
-
[35]
Reichel, Newton interpolation at Leja points , BIT, 30 (1990), pp
L. Reichel, Newton interpolation at Leja points , BIT, 30 (1990), pp. 332–346
1990
-
[36]
Saad, Iterative methods for sparse linear systems, SIAM, Philadelphia, Penn- sylvania, 2 ed., 2003
Y. Saad, Iterative methods for sparse linear systems, SIAM, Philadelphia, Penn- sylvania, 2 ed., 2003
2003
-
[37]
Simoncini and D
V. Simoncini and D. Szyld , Theory of inexact Krylov subspace methods and applications to scientific computing , SIAM J. Sci. Comput., 25 (2003), pp. 454– 477
2003
-
[38]
G. L. G. Sleijpen and H. A. van der Vorst , Reliable updated residuals in hybrid Bi-CG methods , Computing, 56 (1996), pp. 141–163
1996
-
[39]
H. A. van der Vorst and Q. Ye , Residual replacement strategies for Krylov subspace iterative methods for the convergence of true residuals , SIAM J. Sci. Comput., 22 (1999), pp. 835–852
1999
-
[40]
Van Rosendale, Minimizing inner product data dependencies in conjugate gradient iteration, in Proc
J. Van Rosendale, Minimizing inner product data dependencies in conjugate gradient iteration, in Proc. IEEE Int. Conf. Parallel Proc., IEEE, 1983, pp. 44–46
1983
-
[41]
Williams, M
S. Williams, M. Lijewski, A. Almgren, B. Van Straalen, E. Carson, N. Knight, and J. Demmel, s-step Krylov subspace methods as bottom solvers for geometric multigrid , in Proc. IEEE Int. Symp. Parallel Dist. Proc. (IPDPS), IEEE, 2014, pp. 1149–1158
2014
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.