REVIEW 4 major objections 5 minor 38 references
A randomized pivoted QR on the target tensor selects the rows that matter for every CPD-ALS least-squares step, with one fixed sampling matrix per mode.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 05:29 UTC pith:57LHBIGU
load-bearing objection New sampling strategy worth a serious look; main theorem does not cover the implemented method, but experiments are strong enough to justify referee time. the 4 major comments →
Accelerating the Canonical Polyadic Alternating Least Squares Optimization via a Randomized Interpolative Decomposition
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that row-wise sampling for each CPD-ALS least-squares subproblem can be based on the leverage scores of the matricized target tensor rather than on the ever-changing Khatri–Rao product of the current factor matrices. When the CP rank R is at most the mode dimension n_k and the factor matrix has full column rank, the row leverage scores of the mode-k unfolding T_k^T are exactly those of the exact KRP W*_k; when R > n_k they are the leverage scores of an n_k-dimensional projection of W*_k. Theorem IV.6 then proves that sampling s rows proportionally to these leverage scores gives a (1+ε)-accurate sketched least-squares problem with probability 1−δ for s = (Rγ/β) max(4/(δϵ)
What carries the argument
The central object is the row leverage score of the mode-k unfolding of the target tensor, defined through an orthogonal basis of its column space. The paper's key identity is that for R ≤ n_k with full column rank factors, the leverage scores of T_k^T equal those of the exact Khatri–Rao product W*_k. The algorithm approximates this distribution not by computing scores but by a randomized strong rank-revealing column-pivoted QR (SE-QRCS), whose pivot order is treated as a deterministic proxy for the largest leverage scores; a hybrid scheme then tops up with uniform samples from the remaining rows. The proof machinery is a Bernstein/Markov analysis showing that this distribution satisfies the
Load-bearing premise
The argument hinges on the assumption that the pivot columns selected by the randomized QR on the target tensor match the rows that have the largest leverage scores of the exact Khatri–Rao product; the paper proves equality of the leverage scores themselves but not that the greedy pivoting algorithm discovers them in that order.
What would settle it
Construct a tensor of rank R ≤ n_k with known factor matrices whose mode-k unfolding has a row-leverage distribution with a long tail, run SE-QRCS on T_k, and compare the pivot set to the rows with the top leverage scores of W*_k. If the pivot set does not contain most of the high-leverage rows (beyond α ≈ n_k), then Theorem IV.6's guarantee does not apply to Algorithm 1 and the method's reconstructions should degrade on synthetic tensors with known ground truth.
If this is right
- Every mode of the tensor is sampled exactly once: the target tensor need not be touched again during the ALS sweep, so the tensor can be streamed or stored in compressed sampled form.
- The storage cost drops from O(∏ n_k) for the full tensor to O(∑ n_k s) for the sampled unfoldings, and per-iteration cost becomes O(sR² + n_k sR + R³) for the normal-equation solve.
- The sample count adapts to the ALS trajectory: γ shrinks as factor matrices converge, so fewer samples suffice in later iterations (the paper uses a fixed conservative budget).
- The method remains effective when the CP rank exceeds a mode dimension, a regime where factor-leverage-score based samplers (which reduce to uniform sampling) break down.
- For chemistry's three-center two-electron integral tensors, the method can decompose ranks up to 2I_aux with the same fit as leverage-score sampling but without per-iteration resampling in the small-rank regime.
Where Pith is reading between the lines
- An implication left implicit is that the pivot set could be reused across multiple decompositions of the same tensor (different ranks, initializations, or updated factor matrices), amortizing the one-time SE-QRCS cost over several ALS runs.
- The theorem's ε-errors accumulate across ALS iterations; the paper leaves the global non-asymptotic convergence rate open. A natural extension is to prove that a single fixed sampling matrix preserves the linear convergence of exact CPD-ALS when γ is small.
- The paper's connection among sRRQR pivots, leverage scores, and determinantal point processes suggests a testable prediction: the pivot set should approximate the maximum-volume subset of the mode-k unfolding. Measuring the volume overlap between pivot sets and top-leverage rows on structured tensors would directly probe the load-bearing assumption.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a randomized sampling strategy for the canonical polyadic alternating least squares (CPD-ALS) optimization. Instead of sampling rows of each least-squares subproblem from the Khatri–Rao product (KRP) using factor-level leverage scores, the authors propose to sample rows from the matricized target tensor, using a randomized strong rank-revealing QR factorization (SE-QRCS) to select pivots. The pivots are then used to define one fixed sampling matrix per mode for the whole ALS iteration, so the target tensor is sampled only once per mode. The main theoretical result, Theorem IV.6, gives a sample size that guarantees (1+epsilon)-relative accuracy for a least-squares subproblem when the rows are sampled i.i.d. from the leverage scores of the exact (but unknown) KRP W*. Section III.A proves that, when R <= n_k and F*_k has full column rank, the row leverage scores of the matricized target tensor equal those of W*_k. Numerical experiments on synthetic tensors, the COIL dataset, and a three-center two-electron Coulomb integral tensor compare the method with the leverage-score-based CPD-ALS of Larsen and Kolda.
Significance. If the theoretical guarantee could be connected to the implemented Algorithm 1, the contribution would be significant: replacing per-iteration KRP leverage-score sampling with a one-time, target-tensor-derived sampling scheme is a genuine practical advance, with potential storage and runtime benefits. The paper is also strong on empirical evaluation: it uses reproducible code, a realistic chemistry tensor, and an honest discussion of limitations. The central caveat is that Theorem IV.6 applies to a sampling distribution that Algorithm 1 does not actually use. Section IV.B bridges this gap only by an unproved assumption, and Section VI concedes that the QR-based ordering can fail for large ranks. Thus the main formal claim is not currently tied to the method being proposed. The stress-test concern lands: the load-bearing step is the assertion that SE-QRCS pivots of T_k behave like samples from the top leverage scores of W*_k, and this is not proved.
major comments (4)
- [IV.A, IV.B, Algorithm 1] Theorem IV.6 applies to a row sampling matrix S whose rows are drawn i.i.d. from a distribution p satisfying p_i >= beta * l_i(W*)/R, where l_i(W*) are the leverage scores of the exact KRP. Algorithm 1 instead selects rows via SE-QRCS pivots on T_k and then uniformly samples the remaining rows. Section III.A only proves equality of the leverage scores of T^T_k and W*_k when R <= n_k and F*_k has full column rank; it does not prove that the pivots returned by a greedy sRRQR follow the leverage-score distribution. Section IV.B states 'we first note that... the pivots obtained by SE-QRCS on T^T_k are close to those obtained deterministically via the highest leverage scores of W*_k' and 'we assume that... the remaining rows are likely incoherent and have small leverage scores.' These are assumptions, not consequences of Section III.A. Section VI explicitly concedes that for large tensors and
- [III.A, abstract] The abstract claims the method 'is still effective when the CP rank is greater than the dimension of any one mode.' Section III.A shows that for R > n_k the factorization yields U_T = U_W \bar U in R^{m x n_k}, so the leverage scores of T^T_k are those of a rank-n_k projection of W*_k, not the leverage scores of W*_k itself. The theorem's hypotheses -- namely p_i >= beta * l_i(W*)/R and the definition of gamma involving l_i(W*) -- are not established in this regime. The claim should either be restricted to R <= n_k or supported by a separate analysis; Figure 3 is empirical evidence, not a proof.
- [IV.A, IV.B] Theorem IV.6 is explicitly an one-iteration statement: 'if a sampling matrix was drawn independently according to leverage scores of W*.' Algorithm 1, however, fixes one sampling matrix per mode for the entire ALS procedure (last sentence of Section IV.B). During ALS, the KRP W_t changes at every iteration, so the two structural conditions (9)-(10) would need to hold for the same fixed S against changing subspaces. No argument shows this. The discussion of gamma decreasing as W_t approaches W* concerns a freshly drawn sampling matrix; it does not control a fixed S. The paper should either prove a fixed-sampling variant of the theorem or clearly state that the theory applies only when S is redrawn per iteration.
- [IV.B] The hybrid sampling argument invokes the result of Hayashi et al. [27] for a matrix S_H composed of alpha deterministic rows plus additional rows sampled according to the remaining leverage scores. In Algorithm 1, the remaining rows are sampled uniformly at random. The transition from 'the remaining rows are likely incoherent and have small leverage scores' to the required lower bound p_i >= beta * l_i(W*)/R for uniform sampling is not quantified. Without a bound on the largest remaining leverage score, uniform sampling of the remainder does not satisfy the hypotheses of the cited result.
minor comments (5)
- [IV.A, Theorem IV.6] The proof of Theorem IV.6 uses a union bound over the two structural conditions, but the displayed sample complexity does not reflect the need to set the failure probability to delta/2 in each lemma. Lemma IV.3 with failure probability delta/2 would require a factor ln(4R/delta) in the log term rather than ln(2R/delta). This is a small but real correction.
- [IV.A, Eq. (7)] A regularized sampling probability is introduced in (7) for rows with zero leverage score, but the subsequent theorems assume p_i > 0 for all i and do not carry the regularization parameter through the sample complexity. The statement should specify whether the guarantee covers the regularized distribution.
- [IV.A, Proposition IV.7] The proof of Proposition IV.7 is hard to follow: several displayed inequalities appear to omit denominators or intermediate steps, and the normalization convention for the factor matrices is not stated. Please rewrite the proof with explicit definitions of P, D_k, and the Frobenius norms used.
- [Algorithm 1, line 6] The symbol epsilon is used both for the rank-truncation threshold in line 6 (|R_k(i,i)| >= epsilon) and for the accuracy tolerance in Theorem IV.6 and the abstract. Use a distinct symbol, for example tau_k, for the truncation threshold.
- [Section II.D] The expected number of pivots after SE-QRCS is given as E(p) == m(1 - (1 - k/l)^tau), but the notation and derivation are compressed. A short explanation or reference would improve clarity, since the quantity is used to justify the method's cost.
Circularity Check
No significant circularity; the theoretical gap is an unproven algorithmic assumption, not a circular reduction.
full rationale
The central guarantee (Theorem IV.6) is a standard leverage-score sampling bound for the exact, unknown KRP W*; the paper proves in Section III.A that when R <= n_k and F*_k has full column rank, the row leverage scores of T_k^T equal those of W*_k via a deterministic SVD argument. The sample complexity s = R*gamma/beta * max(...) depends on gamma and beta, but these are mathematical parameters, not fitted to data. The algorithm's SE-QRCS pivots are presented as a proxy for leverage-score rows; Section IV.B explicitly assumes, rather than defines, that the pivots are close to the highest leverage scores and that remaining rows are incoherent. That assumption is an unproven correctness gap, not a circular reduction: Theorem IV.6 is not being used to define the pivots. Self-citations (e.g., SE-QRCS from [18], randomized sRRQR from [17]) supply algorithmic building blocks and are not load-bearing to the main result. Section VI's stated limitation that pivoted QR can miss large leverage scores for large ranks further confirms the gap is empirical, not definitional. No equation is equal-by-construction to its input.
Axiom & Free-Parameter Ledger
free parameters (4)
- gamma (KRP-to-true-KRP coherence)
- beta (leverage-score approximation constant)
- alpha_k (number of deterministic pivots in hybrid sampling)
- SE-QRCS embedding dimension l and sparsity tau =
l=ceil(3 n_k log n_k), tau=ceil(log n_k)
axioms (6)
- domain assumption Exact rank-R CP decomposition exists: T = [[F*_1, ..., F*_N]]
- domain assumption Good initialization ensuring convergence to global solution
- ad hoc to paper SE-QRCS pivots of T_k are close to the top leverage scores of W*_k
- ad hoc to paper Remaining rows after alpha pivots are incoherent with small leverage scores
- ad hoc to paper Sampling distribution satisfies p_i >= beta * l_i(W*)/R
- domain assumption sRRQR approximately maximizes the same volume objective as DPP/leverage-score sampling
read the original abstract
We present a novel leverage score-based sampling strategy for the randomized alternating least squares optimization (ALS) of the canonical polyadic decomposition (CPD-ALS). Unlike previous strategies, we determine row-wise samples for the CPD-ALS problem from the leverage scores of the target tensor which is being decomposed. We demonstrate that, when rows are sampled according to the leverage score distribution of the matricized target tensor, each least squares subproblem of the CPD-ALS problem achieves $(1+\epsilon)-$relative accuracy in the residual norm with probability at least $1-\delta$ using a sampling $s=\frac{R\gamma}{\beta} \max\left(\frac{4}{\delta \epsilon}, \frac{144\ln(2R/\delta)}{\epsilon_{0}^{2}}\right)$, where $\epsilon_{0}$ is a constant, $\beta$ is leverage score's approximation constant, $R$ is the target rank and $\gamma$ captures the coherence between the Khatri Rao product (KRP) of the CPD factor matrices and the exact KRP; $\gamma$ decreases as the ALS iterates converge. To efficiently approximate the leverage score distribution for each matricization of the target tensor without explicitly computing leverage scores we use a randomized strong rank-revealing QR (sRRQR) factorizations, SE-QRCS. By construction, this QR-based leverage score sampling method outperforms previously published schemes as it does not, in principle, require the resampling of the target tensor or recomputing the leverage scores of the KRP, minimizing the computational and storage overhead of the CPD-ALS procedure.
Figures
Reference graph
Works this paper leans on
-
[1]
F. L. Hitchcock, The expression of a tensor or a polyadic as a sum of products, Journal of Mathematics and Physics6, 164 (1927)
1927
-
[2]
eckart-young
J. D. Carroll and J.-J. Chang, Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition, Psychometrika35, 283 (1970)
1970
-
[3]
explanatory
R. A. Harshmanet al., Foundations of the parafac procedure: Models and conditions for an “explanatory” multi-modal factor analysis, UCLA working papers in phonetics16, 84 (1970)
1970
-
[4]
P. M. Kroonenberg and J. de Leeuw, Principal component analysis of three-mode data by means of alternating least squares algorithms, Psychometrika45, 69 (1980)
1980
-
[5]
Beylkin and M
G. Beylkin and M. J. Mohlenkamp, Numerical operator calculus in higher dimensions, Proc. Natl. Acad. Sci.99, 10246 (2002)
2002
-
[6]
B. W. Larsen and T. G. Kolda, Practical leverage-based sampling for low-rank tensor decomposition, SIAM Journal on Matrix Analysis and Applications43, 1488 (2022)
2022
-
[7]
Battaglino, G
C. Battaglino, G. Ballard, and T. G. Kolda, A practical randomized cp tensor decomposition, SIAM Journal on Matrix Analysis and Applications39, 876 (2018)
2018
-
[8]
Bharadwaj, O
V. Bharadwaj, O. A. Malik, R. Murray, L. Grigori, A. Buluc, and J. Demmel, Fast exact leverage score sampling from khatri-rao products with applications to tensor decomposition, inAdvances in Neural Information Processing Systems, Vol. 36, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Inc., 2023) pp. 47874–47901
2023
-
[9]
Drineas, M
P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff, Fast approximation of matrix coherence and statistical leverage, The Journal of Machine Learning Research13, 3475 (2012)
2012
-
[10]
T. G. Kolda and B. W. Bader, Tensor decompositions and applications, SIAM review51, 455 (2009)
2009
-
[11]
Gu and S
M. Gu and S. C. Eisenstat, Efficient algorithms for computing a strong rank-revealing qr factorization, SIAM Journal on Scientific Computing17, 848 (1996)
1996
-
[12]
D. P. Woodruffet al., Sketching as a tool for numerical linear algebra, Foundations and Trends®in Theoretical Computer Science10, 1 (2014)
2014
-
[13]
Nelson and H
J. Nelson and H. L. Nguyˆ en, Osnap: Faster numerical linear algebra algorithms via sparser subspace embeddings, in2013 ieee 54th annual symposium on foundations of computer science(IEEE, 2013) pp. 117–126. 19
2013
-
[14]
Martinsson, G
P.-G. Martinsson, G. Quintana Ort ´I, N. Heavner, and R. Van De Geijn, Householder qr factorization with randomization for column pivoting (hqrrp), SIAM Journal on Scientific Computing39, C96 (2017)
2017
-
[15]
J. A. Duersch and M. Gu, Randomized qr with column pivoting, SIAM Journal on Scientific Computing39, C263 (2017)
2017
-
[16]
J. Xiao, M. Gu, and J. Langou, Fast parallel randomized qr with column pivoting algorithms for reliable low-rank matrix approximations, in2017 IEEE 24th international conference on high performance computing (HiPC)(IEEE, 2017) pp. 233–242
2017
-
[17]
L. Grigori and Z. Xue, Randomized strong rank-revealing qr for column subset selection and low-rank matrix approximation, arXiv preprint arXiv:2503.18496 (2025)
Pith/arXiv arXiv 2025
-
[18]
I. Fakih and L. Grigori, Efficient qr-based column subset selection through randomized sparse embeddings, arXiv preprint arXiv:2509.03198 (2025)
Pith/arXiv arXiv 2025
-
[19]
Cheng, R
D. Cheng, R. Peng, Y. Liu, and I. Perros, Spals: Fast alternating least squares via implicit leverage scores sampling, Advances in neural information processing systems29(2016)
2016
-
[20]
Grigori and Z
L. Grigori and Z. Xue,Incremental Column Subset Selection via Conditional Determinantal Point Processes, Tech. Rep. In preparation (EPFL and PSI, 2026)
2026
-
[21]
Derezinski and M
M. Derezinski and M. W. Mahoney, Determinantal point processes in randomized numerical linear algebra, Notices of the American Mathematical Society68, 34 (2021)
2021
-
[22]
S. A. Nene, S. K. Nayar, and H. Murase, Columbia object image library (coil100) (1996)
1996
-
[23]
G. Zhou, A. Cichocki, and S. Xie, Decomposition of big tensors with low multilinear rank, CoRRabs/1412.1885(2014), 1412.1885
Pith/arXiv arXiv 2014
-
[24]
Uschmajew, Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation, SIAM J
A. Uschmajew, Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation, SIAM J. Matrix Anal. Appl.33, 639 (2012)
2012
-
[25]
Drineas, M
P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarl´ os, Faster least squares approximation, Numerische mathematik 117, 219 (2011)
2011
-
[26]
Drineas, R
P. Drineas, R. Kannan, and M. W. Mahoney, Fast monte carlo algorithms for matrices i: Approximating matrix multipli- cation, SIAM Journal on Computing36, 132 (2006)
2006
-
[27]
Hayashi, S
K. Hayashi, S. G. Aksoy, G. Ballard, and H. Park, Randomized algorithms for symmetric nonnegative matrix factorization, SIAM Journal on Matrix Analysis and Applications46, 584 (2025)
2025
-
[28]
Itensorcpd library (2026), accessed: 2026-01-01
2026
-
[29]
Fishman, S
M. Fishman, S. R. White, and E. M. Stoudenmire, The ITensor Software Library for Tensor Network Calculations, SciPost Phys. Codebases , 4 (2022)
2022
-
[30]
Benedikt, A
U. Benedikt, A. A. Auer, M. Espig, and W. Hackbusch, Tensor decomposition in post-Hartree–Fock methods. I. Two- electron integrals and MP2, J. Chem. Phys.134, 054118 (2011)
2011
-
[31]
E. G. Hohenstein, R. M. Parrish, and T. J. Mart ´ ınez, Tensor hypercontraction density fitting. I. Quartic scaling second- and third-order Møller-Plesset perturbation theory, J. Chem. Phys.137, 044103 (2012)
2012
-
[32]
Pierce and M
K. Pierce and M. Morales, Using matrix-free tensor-network optimizations to construct a reduced-scaling and robust second-order møller-plesset theory, Journal of Chemical Theory and Computation21, 5952 (2025)
2025
-
[33]
Pierce, Toward using matrix-free tensor decompositions to systematically improve approximate tensor-networks, Journal of Chemical Theory and Computation21, 6464 (2025)
K. Pierce, Toward using matrix-free tensor decompositions to systematically improve approximate tensor-networks, Journal of Chemical Theory and Computation21, 6464 (2025)
2025
-
[34]
Jørgensen and J
P. Jørgensen and J. Simons, Ab initio analytical molecular gradients and Hessians, J. Chem. Phys.79, 334 (1983)
1983
-
[35]
D. J. Wales and M. P. Hodges, Global minima of water clusters (H2O)n , n≤21 , described by an empirical potential, Chem. Phys. Lett.286, 65 (1998)
1998
-
[36]
T. H. Dunning, Gaussian basis sets for use in correlated molecular calculations. I. The atoms boron through neon and hydrogen, J. Chem. Phys.90, 1007 (1989)
1989
-
[37]
R. A. Kendall, T. H. Dunning, and R. J. Harrison, Electron affinities of the first-row atoms revisited. Systematic basis sets and wave functions, J. Chem. Phys.96, 6796 (1992)
1992
-
[700]
The angle between two CPD approximated tensors is computed efficiently by leveraging the Khatri-Rao product structure of the CPD approximation
In the SE-QRCS-based ALS procedure 80 columns are sampled and the algorithm terminates when the change in the angle between two CPD approximated tensors is less than 10 −5 or the number of iterations exceeds 200. The angle between two CPD approximated tensors is computed efficiently by leveraging the Khatri-Rao product structure of the CPD approximation. ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.