REVIEW 3 major objections 5 minor 28 references
Randomized and Inner-product Free Krylov Methods for Large-scale Inverse Problems
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Two new inner-product-free Krylov methods track the residual norms of GMRES and LSQR without computing inner products.
desk verdict The core idea is sound and worth pursuing, but Algorithm 1 as printed cannot implement the claimed method, so the experimental results are not independently verifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sketched projected least-squares problem $\min_y \|S(AL_k y - b)\|$, in which $L_k$ is the nonorthogonal Krylov basis produced by the (generalized) Hessenberg method with partial pivoting and $S$ is a Gaussian sketching matrix. The Hessenberg construction removes inner products, and the sketch turns the tall-skinny projected problem into a small least-squares problem. The bound in Eq. (14) is the engine: it converts the subspace embedding property into multiplicative control of the residual norm, showing that the sketched solution stays within a factor $(1+\epsilon)/(1-\epsilon)$ of the minimal-residual solution in that subspace.
What would settle it
Recompute the deblurring example with the paper's sketch size and plot per-iteration residual norms of sCMRH against GMRES and CMRH; the central claim fails if the sCMRH curve rises above CMRH, or if its gap above GMRES ever exceeds the sketch condition number $\kappa_2(SQ_{[A b]})$ that the paper identifies with $(1+\epsilon)/(1-\epsilon)$.
Extended reading notes
Core claim
At iteration $k$, the proposed methods generate a nonorthogonal basis $L_k$ for a Krylov subspace using the Hessenberg method (for sCMRH) or the generalized Hessenberg method (for sLSLU), with no inner products. Instead of the quasi-minimal-residual reduction used by CMRH and LSLU, they solve the sketched projected least-squares problem $\min_{y} \|S(AL_k y - b)\|$, where $S$ is a Gaussian sketch. By the subspace embedding property, the resulting residual norm satisfies $\|Ax_{\mathrm{LS}}-b\| \le \|Ax_k^{(S)}-b\| \le \frac{1+\epsilon}{1-\epsilon}\|Ax_{\mathrm{LS}}-b\|$, where $x_{\mathrm{LS}}$ is the true minimal-residual solution in the same Krylov subspace; and the sketch-and-solve estimate is unbiased, with expected squared residual norm inflated only by a factor $1 + k/(\ell-k-1)$. The paper's numerical experiments show sCMRH residual norms tracking GMRES and sLSLU tracking LSQR, while CMRH and LSLU residual norms drift larger.
Load-bearing premise
The method's practical success rests on the assumption that a small random projection of dimension ten times the iteration count preserves residual norms well enough, but the paper's own theoretical guarantee needs a much larger projection that the experiments do not use.
Editorial extensions
If this is right
- If the central claim holds, sCMRH and sLSLU can stand in for GMRES and LSQR in inverse-problem solves where inner products are the bottleneck, because their residual norms follow the same curves.
- Inner-product-free operation removes a common source of breakdown in low-precision arithmetic, since no orthogonal projection is computed.
- The same sketch-and-solve reduction applies to the Tikhonov-regularized problem with two sketches, giving residual-norm behavior close to LSQR for a fixed regularization parameter.
- On the tested problems, the sketched methods achieve smaller residual norms than their non-sketched inner-product-free predecessors CMRH and LSLU.
Reading between the lines
- Extending the paper's idea, a natural test is to vary the sketch dimension $\ell$ with the iteration count or problem size: the fixed choice $\ell = 10(\text{maxiter}+1)$ used everywhere is far smaller than the theory requires, so residual tracking might eventually break as the problem grows.
- The same sketched-projected-problem idea could be applied to other nonorthogonal Krylov bases, such as truncated Arnoldi or incomplete Golub-Kahan, replacing a reorthogonalization-depth hyperparameter with a sketch.
- Because the sketch-and-solve solution is unbiased, averaging several independent sLSLU runs could reduce the variance of the residual-norm estimates; the paper does not explore this averaging.
- One further consequence the authors do not draw is that, for hybrid approaches that estimate a regularization parameter from the projected problem, the closer residual norm should make parameter-selection heuristics more reliable than with CMRH and LSLU.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes two inner-product-free Krylov subspace methods, sCMRH and sLSLU, that combine Hessenberg/generalized-Hessenberg basis construction with randomized sketch-and-solve for the projected least-squares and Tikhonov problems. Section 3.2 derives a residual-norm approximation bound (Eq. 14), notes the unbiasedness of the sketched estimator, and gives an expected squared residual formula. Algorithms 1 and 2 provide pseudocode, and Section 4 reports experiments on deblurring, simulated neutron tomography, and two real X-ray datasets, with the conclusion that the sketched methods track GMRES/LSQR residual norms more closely than CMRH/LSLU while remaining inner-product free.
Significance. If the methods perform as claimed, the contribution is useful: it offers inner-product-free alternatives to GMRES and LSQR whose residual norm is closer to the true minimal residual than that of quasi-minimal residual methods, with potential benefits for parallel and low-precision computation. The theoretical derivation in Section 3.2 is standard and largely correct, and the paper honestly acknowledges that the subspace-embedding guarantee cannot always be guaranteed for the sketch sizes used in practice. However, the pseudocode as printed does not define the sketched projected problem it claims to solve, and the experiments use sketch dimensions far below the theory with no variance reporting. These issues currently prevent the numerical evidence from being verified.
major comments (3)
- [Algorithm 1, line 31; Algorithm 2, line 32] The solve step is not executable from the pseudocode. Line 31 states that y_k is the minimizer of ||S2 r0 - S2 A L_k y||^2 = ||S2 r0 - Z y||^2, but no matrix Z is constructed anywhere. The only sketch computed in the loop, z_k = S2 * A^T d_k (line 11), is never used and is not a column of S2 D_{k+1} H_{k+1,k} or of S2 A L_k; similarly, Algorithm 2 computes f1 and f_{k+1} but never assembles S1 L_k. Consequently, the numerical results in Section 4 may come from an implementation different from the method defined by Eq. (13). Please specify explicitly how S2 A L_k (or an equivalent factored form) is formed, for example by maintaining sketches of the D columns and multiplying by H, and correct both algorithms accordingly.
- [Algorithm 1, lines 17, 21, and 27] The basis update contains inconsistent variable reuse that prevents verification of the generalized Hessenberg recurrence (7). Line 17 overwrites d_k with the newly computed basis vector, so d_k no longer denotes the kth column of D when it is used in line 21 and in the subsequent W update; the new vector should be stored as d_{k+1}. In addition, line 27 sets W(k+1,k+1) using u(g(i2)), although the quantity available for the L-basis update at that point is q(g(i2)). These are not mere notation slips: they make the algorithm ambiguous about which vectors enter the H and W relations.
- [Section 4 and Eq. (14)] The experiments set ell = 10*(maxiter+1), which equals at most a few hundred in the reported runs, while the theory in Section 3.2 requires ell ~ m log(m)/eps^2 for Eq. (14) to hold with high probability; for the tested image and tomography problems, m is orders of magnitude larger. The paper acknowledges that the guarantee cannot always be ensured in practice, but every figure shows a single run with no error bars or repeated trials, and the random pivoting sample size is not systematically studied. Please report the distribution of residual norms over repeated sketches and pivot samples, or otherwise provide empirical justification for the small sketch dimension.
minor comments (5)
- [Throughout] There are several typos and grammatical slips, including "mulitplications" (Section 1), "are are" (Section 4.1), and "perforamcne" (Section 5). A careful proofread is needed.
- [References] References [15] and [23] are the same paper (Sarlos, FOCS 2006) and should be consolidated.
- [Section 3.3, Eq. (17)] The two terms in Eq. (17) are missing the superscript 2 and the subscript 2: the expression should read ||S2(ALky - b)||_2^2 + lambda^2 ||S1(Lky)||_2^2. The same omission appears in Algorithm 2, line 32.
- [Figure 2 caption and Section 4.1] The caption states that sCMRH uses a sample size of 5 for the random pivot search, but Section 4.2 describes a sample size of 25 and the text never defines how the sample size for sCMRH was chosen or whether the results are sensitive to it.
- [Figures 4, 6, 8, and 11] Several residual-norm panels appear to lack curve labels or legends, making it difficult to verify which curve corresponds to sLSLU, LSLU, and LSQR; please add legends or otherwise identify each curve.
Circularity Check
No significant circularity: the sketch-and-solve residual-norm claim is a standard embedding argument, not a restatement of the method's inputs.
full rationale
The central derivation—that solving the sketched projected problem (Eq. 13) yields residual norms bounded relative to the minimal-residual solution by (1+ε)/(1−ε) (Eq. 14)—is a direct application of the subspace-embedding property (Eq. 4), not an equivalence-by-construction. The Gaussian expectation identity for sketch-and-solve is cited to [24,25] and is independent of the paper's fitted values. The generalized Hessenberg relations (Eqs. 6–7) are inherited from the authors' prior work [9], but they are used as a basis-construction tool rather than as the source of the residual-norm claim; the residual-norm claim stands on the embedding property. The comparisons with CMRH, LSLU, GMRES, and LSQR are benchmarks, not circular premises. The paper explicitly concedes that the small sketch dimension used in experiments does not always satisfy the theory ('this cannot, however, always be guaranteed in practice'), which is a validity gap rather than a circular reduction. One non-circular but serious issue: Algorithm 1 line 31 refers to an undefined Z (and line 11's z_k = S2 A^T d_k is unused), so the printed pseudocode may not implement Eq. (13); this is an algorithmic correctness concern, not a circularity.
Assumptions & free parameters
free parameters (3)
- sketch dimension ell =
10 * (maxiter + 1)
- sampling size for partial pivoting =
25 entries
- regularization parameter lambda =
26 (neutron tomography), 1 (cheese and walnut)
assumptions (3)
- standard math The sketching matrix S satisfies the subspace embedding property (4) for the vectors in the projected Krylov subspace, with a given probability and distortion 0 < eps < 1.
- domain assumption The generalized Hessenberg method with partial pivoting generates linearly independent basis vectors for K_k(A^T A, A^T b) and K_k(A A^T, b), with manageable condition numbers.
- domain assumption The matrix A L_k has full column rank, so the least-squares and sketched least-squares problems have unique solutions and the unbiasedness result E[x(S)_k] = x(LS)_k holds.
Cite this review
Pith. "Pith review of Randomized and Inner-product Free Krylov Methods for Large-scale Inverse Problems." pith.science (2026). https://pith.science/paper/QLOPH2RJ
@misc{pith2026250202721,
author = {Pith},
title = {Pith review of: Randomized and Inner-product Free Krylov Methods for Large-scale Inverse Problems},
year = {2026},
howpublished = {\url{https://pith.science/paper/QLOPH2RJ}},
note = {Machine review of arXiv:2502.02721}
}
read the original abstract
Iterative Krylov projection methods have become widely used for solving large-scale linear inverse problems. However, methods based on orthogonality include the computation of inner-products, which become costly when the number of iterations is high; are a bottleneck for parallelization; and can cause the algorithms to break down in low precision due to information loss in the projections. Recent works on inner-product free Krylov iterative algorithms alleviate these concerns, but they are quasi-minimal residual rather than minimal residual methods. This is a potential concern for inverse problems where the residual norm provides critical information from the observations via the likelihood function, and we do not have any way of controlling how close the quasi-norm is from the norm we want to minimize. In this work, we introduce a new Krylov method that is both inner-product-free and minimizes a functional that is theoretically closer to the residual norm. The proposed scheme combines an inner-product free Hessenberg projection approach for generating a solution subspace with a randomized sketch-and-solve approach for solving the resulting strongly overdetermined projected least-squares problem. Numerical results show that the proposed algorithm can solve large-scale inverse problems efficiently and without requiring inner-products.
Reference graph
Works this paper leans on
-
[1]
Chung, J., Knepper, S., Nagy, J.G.: Large-scale inverse problems in imaging. In: Scherzer, O. (ed.) Handbook of Mathematical Methods in Imaging, pp. 47–90. Springer, New York, NY (2015). https://doi.org/10.1007/978-1-4939-0790-8 2
-
[2]
Hansen, P.C.: Discrete Inverse Problems: Insight and Algorithms . SIAM, Philadel- phia (2010)
work page 2010
-
[3]
Vogel, C.R.: Computational Methods for Inverse Problems. SIAM , Philadelphia (2002)
work page 2002
-
[4]
Zhdanov, M.: Geophysical Inverse Theory and Regularization Pr oblems vol. 36. Elsevier, New York (2002)
work page 2002
-
[5]
Freeman, San Francisco and London (1963)
Faddeev, D.K., Faddeeva, V.N.: Computational Methods of Linear Algebra. Freeman, San Francisco and London (1963). Translated by R. C. W illiams
work page 1963
-
[6]
Sadok, H.: CMRH: A new method for solving nonsymmetric linear sys tems based on the Hessenberg reduction algorithm. Numer. Algor. 20, 303–321 (1999)
work page 1999
-
[7]
B IT Numerical Mathematics 52, 485–501 (2012) https://doi.org/10.1007/s10543-011- 0365-x
Sadok, H., Szyld, D.B.: A new look at CMRH and its relation to GMRES. B IT Numerical Mathematics 52, 485–501 (2012) https://doi.org/10.1007/s10543-011- 0365-x
-
[8]
Brown, A.N., Sabat´ e Landman, M., Nagy, J.G.: H-CMRH: An inner pr oduct free hybrid Krylov method for large-scale inverse problems. SIAM J ournal on Matrix Analysis and Applications 46(1), 232–255 (2025) https://doi.org/10.1137/ 24M1634874
work page 2025
Show all 28 references
-
[9]
https://arxiv.org/abs/ 2409.05239
Brown, A.N., Chung, J., Nagy, J.G., Sabat´ e Landman, M.: Inner pr oduct free Krylov methods for large-scale inverse problems (2024). https://arxiv.org/abs/ 2409.05239
2024 arXiv
-
[10]
ACM Transactions on Mathematical Sof tware (TOMS) 8(1), 43–71 (1982) 20
Paige, C.C., Saunders, M.A.: LSQR: An algorithm for sparse linear e quations and sparse least squares. ACM Transactions on Mathematical Sof tware (TOMS) 8(1), 43–71 (1982) 20
1982
-
[11]
ACM Transactions on Mathematical Softw are (TOMS) 8(2), 195–209 (1982)
Paige, C.C., Saunders, M.A.: Algorithm 583: LSQR: Sparse linear eq uations and least squares problems. ACM Transactions on Mathematical Softw are (TOMS) 8(2), 195–209 (1982)
1982
-
[12]
SIAM, Philadel- phia (2003)
Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd e dn. SIAM, Philadel- phia (2003)
2003
-
[13]
Calvetti, D., Somersalo, E.: An Introduction to Bayesian Scientifi c Computing: Ten Lectures on Subjective Computing vol. 2. Springer, New York ( 2007)
2007
-
[14]
Acta Numerica 29, 403–572 (2020) https://doi.org/10
Martinsson, P.-G., Tropp, J.A.: Randomized numerical linear algeb ra: Founda- tions and algorithms. Acta Numerica 29, 403–572 (2020) https://doi.org/10. 1017/S0962492920000021
2020
-
[16]
SIAM Journal on Matrix Analysis and A pplications 44(3), 1073–1095 (2023) https://doi.org/10.1137/22M1518062
G¨ uttel, S., Schweitzer, M.: Randomized sketching for Krylov ap proximations of large-scale matrix functions. SIAM Journal on Matrix Analysis and A pplications 44(3), 1073–1095 (2023) https://doi.org/10.1137/22M1518062
2023 doi
-
[17]
SIAM Journal on Scientific Computing 44(3), 1450–1474 (2022) https://doi.org/10.1137/20M138870X https://doi.org/10.1137/2 0M138870X
Balabanov, O., Grigori, L.: Randomized Gram–Schmidt process wit h applica- tion to GMRES. SIAM Journal on Scientific Computing 44(3), 1450–1474 (2022) https://doi.org/10.1137/20M138870X https://doi.org/10.1137/2 0M138870X
2022 doi
-
[18]
SIAM Journal o n Scientific Computing 36(2), 95–118 (2014) https://doi.org/10.1137/120866580
Meng, X., Saunders, M.A., Mahoney, M.W.: LSRN: A parallel iterativ e solver for strongly over- or underdetermined systems. SIAM Journal o n Scientific Computing 36(2), 95–118 (2014) https://doi.org/10.1137/120866580
2014 doi
-
[19]
SIAM Journal on Scientific Computing 32(3), 1217–1236 (2010) https://doi.org/10.1137/090767911
Avron, H., Maymounkov, P., Toledo, S.: Blendenpik: Supercharg ing LAPACK’s least-squares solver. SIAM Journal on Scientific Computing 32(3), 1217–1236 (2010) https://doi.org/10.1137/090767911
2010 doi
-
[20]
SIAM Journal on Matrix Ana lysis and Applications 45(2), 1183–1214 (2024) https://doi.org/10.1137/23M1565413 https://doi.org/10.1137/23M1565413
Nakatsukasa, Y., Tropp, J.A.: Fast and accurate randomized a lgorithms for linear systems and eigenvalue problems. SIAM Journal on Matrix Ana lysis and Applications 45(2), 1183–1214 (2024) https://doi.org/10.1137/23M1565413 https://doi.org/10.1137/23M1565413
2024 doi
-
[21]
John H opkins University Press, Baltimore (2013)
Golub, G.H., Van Loan, C.V.: Matrix Computations, 4th edn. John H opkins University Press, Baltimore (2013)
2013
-
[22]
Meier, M., Nakatsukasa, Y., Townsend, A., Webb, M.: Are sketch -and- precondition least squares solvers numerically stable? SIAM Journa l on Matrix Analysis and Applications 45(2), 905–929 (2024) https://doi.org/10.1137/ 23M1551973 21
2024
-
[23]
In: 2006 47th Annual IEEE Symposium on Foundation s of Computer Science (FOCS’06), pp
Sarlos, T.: Improved approximation algorithms for large matrice s via random projections. In: 2006 47th Annual IEEE Symposium on Foundation s of Computer Science (FOCS’06), pp. 143–152 (2006). https://doi.org/10.1109/FOCS.2006.37
2006 doi
-
[24]
In: Proceedin gs of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Minin g, pp
Derezi´ nski, M., Mahoney, M.W.: Recent and upcoming developme nts in ran- domized numerical linear algebra for machine learning. In: Proceedin gs of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Minin g, pp. 6470–6479 (2024)
2024
-
[25]
Numerische mathematik 117(2), 219–249 (2011)
Drineas, P., Mahoney, M.W., Muthukrishnan, S., Sarl´ os, T.: Fast er least squares approximation. Numerische mathematik 117(2), 219–249 (2011)
2011
-
[26]
Gazzola, S., Hansen, P.C., Nagy, J.G.: IR Tools: a MATLAB package of iterative regularization methods and large-scale test problems. Numer. Algo rithms 81(3), 773–811 (2019) https://doi.org/10.1007/s11075-018-0570-7
2019 doi
-
[27]
arXiv pr eprint arXiv:1705.05732 (2017)
Bubba, T.A., Juvonen, M., Lehtonen, J., M¨ arz, M., Meaney, A., P urisha, Z., Siltanen, S.: Tomographic x-ray data of carved cheese. arXiv pr eprint arXiv:1705.05732 (2017)
2017 arXiv
-
[28]
arXiv preprint arXiv:1502.04 064 (2015)
H¨ am¨ al¨ ainen, K., Harhanen, L., Kallonen, A., Kujanp¨ a¨ a, A., Niemi, E., Siltanen, S.: Tomographic x-ray data of a walnut. arXiv preprint arXiv:1502.04 064 (2015)
2015
-
[29]
ArXiv pr eprint (2024) arXiv:2412.10129 22
Biguri, A., Sadakane, T., Lindroos, R., Liu, Y., Landman, M.S., Du, Y ., Lohvithee, M., Kaser, S., Hatamikia, S., Bryll, R., Valat, E., Wonglee, S., Blumen- sath, T., Sch¨ onlieb, C.-B.: Tigre v3: Efficient and easy to use iterativ e computed tomographic reconstruction toolbox f...
2024 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.