Pith. sign in

REVIEW 4 minor 34 references

This paper shows that two computational steps in Improved Kernel Partial Least Squares can be replaced by faster ones — a direct matrix-vector form for X rotations and a reuse of earlier quantities for Y loadings — without changing any outp

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:13 UTC pith:M7UOB4O4

load-bearing objection A narrow, honest, and correct algorithms paper: the Q-loading identity is a genuine new result with a clean proof, and the benchmarks are more candid than most.

arxiv 2607.16138 v1 pith:M7UOB4O4 submitted 2026-07-17 cs.LG cs.DS

Improving Improved Kernel PLS

Ole-Christian Galbo Engstr{\o}m This is my paper
classification cs.LG cs.DS
keywords partial least squaresIKPLSX rotationsY loadingscomputational complexitychemometricseigenvalueparallelization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper targets two bottlenecks shared by the Improved Kernel Partial Least Squares (IKPLS) algorithms, the X rotation matrix R and the Y loading matrix Q. It proves that R can be evaluated as a single matrix-vector product using already-accumulated matrices rather than by term-by-term accumulation, reducing sequential work while keeping the same multiplication count. For Q, it establishes — apparently for the first time — that each Y loading equals a constant multiple of a vector already computed in the previous step, cutting the operation count from Θ(KM) to Θ(M) whenever there is one response variable or fewer response variables than predictor variables. Both substitutions are shown to produce bit-identical W, P, Q, R, and T, and benchmarks report speedups of up to about 2× on CPU and 6× on GPU for full fits.

Core claim

The central discovery is a pair of algebraic identities inside the IKPLS iterations. First, the loading r_a at component a satisfies r_a = w_a − R_{a−1}(P^T_{a−1}w_a), which is the same sum as the original recurrence but organized as one matrix-vector product. Second, the Y loading q_a can be recovered from quantities computed in step 2: when M=1, q_a = ||w̃_a||_2 / ||t_a||_2^2, and when 2≤M<K, q_a = √λ_a q̃_a / (||q̃_a||_2 ||t_a||_2^2), where λ_a and q̃_a are the leading eigenvalue and eigenvector of (X^T Y)_a^T (X^T Y)_a. The proof rests on Lemma 1, which states that P_a^T R_a = I, a consequence of the mutual orthogonality of the score columns. Theorem 1 shows these formulas equal the stan

What carries the argument

The workhorses are two identities. Lemma 1, P_a^T R_a = I, is an orthonormality relation between the PLS loading and rotation matrices that makes the deflation of X^T Y collapse in a controlled way, so that residual covariance with earlier components vanishes. Proposition 1 rewrites the recurrence for r_a as a single matrix-vector product R_{a−1}(P^T_{a−1}w_a). Theorem 1 uses these to show that (X^T Y)_a^T r_a = (X^T Y)_a^T w_a, and then that the step-2 quantities λ_a and q̃_a carry enough information to compute q_a in O(M) rather than O(KM) operations.

Load-bearing premise

The proofs assume the columns of the score matrix T are mutually orthogonal in exact arithmetic; if numerical rounding breaks this orthogonality, the simplified loading formula is no longer guaranteed to match the original, and the claimed 'identical outputs' are exact in the mathematical sense, not necessarily bit-for-bit in floating point.

What would settle it

Run IKPLS with the improved Q computation on a dataset with nearly collinear predictor variables (or with components that make T near-singular) and compare each q_a against the original Algorithm 2 to the last significant digit; any discrepancy beyond rounding would refute the drop-in identity claim. Alternatively, count operations on a dataset with M=1 and K≫1 and check that the Q step takes time independent of K, as predicted by the Θ(M) bound.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any implementation of IKPLS that swaps in these two steps will produce exactly the same model matrices as the original, so the speedups come with no accuracy trade-off.
  • In single-response (PLS1) and wide-data PLS2 (fewer responses than predictors), the per-component cost of computing Q drops by a factor of K, which matters for iterative procedures like cross-validation that fit many models.
  • The improved R evaluation is always applicable (A≥2) and makes the rotation computation more parallel, benefiting GPU implementations disproportionately.
  • The new Q formulas reuse the eigenvalue/eigenvector computed in step 2, eliminating the need to form the large intermediate (X^T Y)_a times r_a product in the common cases.
  • The paper proves a property that an existing implementation had only guessed at in a source comment, meaning the same correction can be adopted there immediately.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • By extension, the same orthogonality-based shortcut might carry over to nonlinear or kernel PLS variants, since the deflation structure that makes the O(M) loading computation possible is generic rather than specific to linear X.
  • The 'identical outputs' claim is mathematical; on real hardware with rounding and ill-conditioned data, small deviations could appear, so practitioners should verify that the simplified loading formula remains within tolerance for their datasets.
  • The reported speedups are for full fits; repeated cross-validation loops amortize the per-fit gains, potentially amplifying the 2–6× improvement substantially.
  • The direct R evaluation, though asymptotically identical in multiplication count, changes the pattern of memory access and parallelism, so the observed speedup may shift as hardware evolves.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper proposes two improvements to the two IKPLS PLS algorithms. The first replaces the term-by-term accumulation in the Hoskuldsson recurrence for the X-rotation vectors r_a with a direct evaluation R_{a-1}(P_{a-1}^T w_a). The second, Algorithm 3, computes each Y-loading q_a directly from quantities already available from the step-2 eigen-decomposition in IKPLS (the norm of \tilde w_a in the PLS1 case, and the leading eigenvalue/eigenvector pair in the PLS2 case with M<K), reducing the per-loading cost from Θ(KM) to Θ(M) in those regimes. The paper proves exact-arithmetic correctness of both changes (Proposition 1; Theorem 1 and Corollary 1), gives operation counts (Propositions 2–9), and presents CPU/GPU benchmarks in the ikpls package showing large speedups on the isolated steps and 2–6× on full fits.

Significance. If the results hold, the Q improvement is a clean asymptotic speedup for the common PLS1 and PLS2-with-M<K cases, and the R improvement is a practical parallelization gain with no change to the computed mathematical quantities. The proofs are elementary, well structured, and I did not find a load-bearing flaw. A particular strength is that the paper connects the new Q formula to an unproved source-code question in the R pls package, answering it affirmatively. The benchmarks are careful (float64, median timings, CPU and GPU, both IKPLS variants), and the claims are mathematical equivalences rather than fitted predictions, so overfitting or circularity is not a concern.

minor comments (4)
  1. [Section 4.1, Lemma 1] The last equality, t_i^T t_j / ||t_i||^2 = δ_{i,j}, is justified by citing Hoskuldsson for mutual orthogonality of T. This property is standard, but it can be derived in one line from the algorithm's own recurrence: p_i^T r_j = t_i^T t_j / ||t_i||^2, and P_{a-1}^T R_{a-1} = I forces p_i^T r_j = 0 for i<j and 1 for i=j. Adding this derivation would make Lemma 1 self-contained and would remove any appearance of an unexamined external assumption.
  2. [Abstract / Theorem 1 / Corollary 1] The statement that the improved algorithms produce 'exactly the same' W, P, Q, R, and T should be qualified as 'in exact arithmetic.' Algorithm 3 evaluates different expressions (e.g., a scalar division by ||\tilde w_a||_2) than Algorithm 2, so floating-point round-off may differ at the last bits. This is not a correctness issue, but a one-sentence caveat would prevent misreading by implementers.
  3. [Section 4.2, Proposition 6] Writing Θ(M) = Θ(1) when M=1 is slightly nonstandard; the statement is clearer as simply 'requires Θ(1) operations.' This is purely stylistic.
  4. [Section 6.2, Figure 3] The panel headings in the figure appear garbled in the manuscript (e.g., 'T,e res) of ),e .K/L1 fi ) R Q'). This looks like an encoding/rendering artifact, and the labels should be fixed so the figure is readable on its own.

Circularity Check

0 steps flagged

No circularity: the R and Q improvements are proven algebraic equivalences; self-citations are contextual and not load-bearing.

full rationale

Every load-bearing step is an algebraic identity proved from the algorithm's own recurrences. Proposition 1 expands Eq. (2) into exactly the sum in Eq. (1), so the improved R computation is a re-association of the same multiplications. Theorem 1 proves that the qa computed by Algorithm 3 equals line 8 of Algorithm 2 by strong induction: the induction hypothesis supplies the supposition of Lemma 2 for previous components, and the deflation recurrence (4), Lemma 1, and the eigenvalue identity (9) yield the reduced formulas in the M=1 and 2<=M<K branches; the remaining branch is literally the original line 8. Lemma 1 uses T-column orthogonality, cited to Hoskuldsson as an external classical PLS property; this is not a self-citation, it is an explicit assumption shared by both algorithms, and it is not equivalent to the target result being proved. No parameter is fitted and then reported as a prediction; the claim of identical W, P, Q, R, and T is a mathematical equivalence in exact arithmetic, with floating-point differences explicitly outside the claim. Self-citations to the ikpls package and to prior cross-validation work appear only for software context and benchmarking, not as evidence for Theorem 1 or Corollary 1. The derivation is self-contained against the cited IKPLS definitions and external classical results, and I find no circular step.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper adds no free parameters and postulates no new entities. It relies on standard linear algebra and standard PLS properties, mainly the orthogonality of score columns.

axioms (3)
  • domain assumption The columns of the score matrix T are mutually orthogonal in exact arithmetic.
    Used in Lemma 1 to establish P_a^T R_a = I; cited to Hoskuldsson, not proved in this paper. If this fails, the key simplification for q_a breaks.
  • domain assumption Extraction halts if (X^TY)_a becomes a zero matrix or t_a = 0_N, so every extracted component has nonzero covariance and nonzero score norm.
    Stated below Equation (4); needed to ensure ||wtilde_a|| and ||t_a||^2 are positive and the divisions in Algorithm 3 are well-defined.
  • standard math Symmetric positive-semidefinite matrices have real nonnegative eigenvalues and eigenvectors as used in the M<K branch.
    Assumed for the eigendecomposition of (Y^T X X^T Y)_a in Algorithm 1 and used in the proof of Theorem 1, case (b).

pith-pipeline@v1.3.0-alltime-deepseek · 16683 in / 12067 out tokens · 96708 ms · 2026-08-01T21:13:11.005750+00:00 · methodology

0 comments
read the original abstract

Improved Kernel Partial Least Squares (IKPLS) algorithms 1 and 2 are among the fastest PLS calibration algorithms. This article focuses on two shared steps, the computation of the $\mathbf{X}$ rotations, $\mathbf{R}$, and the $\mathbf{Y}$ loadings, $\mathbf{Q}$, and accelerates both. For $\mathbf{R}$, term-by-term accumulation is replaced by a direct evaluation strategy that requires the same number of multiplications but parallelizes better on modern hardware. For $\mathbf{Q}$, I identify - to the best of my knowledge, for the first time - equivalences showing that each $\mathbf{Y}$ loading is obtainable, up to explicitly derived constants, from quantities already computed earlier in the same iteration, and I exploit them in IKPLS to reduce the cost of each loading from $\Theta\left(KM\right)$ to $\Theta\left(M\right)$ operations whenever $M = 1$ or $2 \leq M < K$, with $K$ predictor variables (number of columns in $\mathbf{X}$) and $M$ response variables (number of columns in $\mathbf{Y}$). Both improvements provably yield exactly the same $\mathbf{W}$, $\mathbf{P}$, $\mathbf{Q}$, $\mathbf{R}$, and $\mathbf{T}$ as the original algorithms. Benchmarks with NumPy (CPU) and JAX (GPU) show speedups of up to two orders of magnitude for the isolated steps and of approximately $2\times$ (CPU) and $6\times$ (GPU) for entire fits. Both improvements are implemented in the free, open-source Python package \texttt{ikpls}.

Figures

Figures reproduced from arXiv: 2607.16138 by Ole-Christian Galbo Engstr{\o}m.

Figure 1
Figure 1. Figure 1: Heatmap showing the speedup of the computation of [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Heatmap showing the speedup of the computation of [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Time spent on full fits for the original and improved [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages

  1. [1]

    Comparison of PLS algorithms when number of objects is much larger than number of variables

    Alin, A., 2009. Comparison of PLS algorithms when number of objects is much larger than number of variables. Statistical Papers 50, 711–

    2009

  2. [2]

    A comparison of nine PLS1 algorithms

    Andersson, M., 2009. A comparison of nine PLS1 algorithms. Journal of Chemometrics 23, 518–529. doi: 10.1002/cem.1248

    work page doi:10.1002/cem.1248 2009

  3. [3]

    Partial least squares for discrimination

    Barker, M., Rayens, W., 2003. Partial least squares for discrimination. Journal of Chemo- metrics 17, 166–173. doi: 10.1002/cem.785

    doi:10.1002/cem.785 2003

  4. [4]

    Account- ing for sampling weights in pls path mod- eling: Simulations and empirical examples

    Becker, J.M., Ismail, I.R., 2016. Account- ing for sampling weights in pls path mod- eling: Simulations and empirical examples. European Management Journal 34, 606–617. doi:10.1016/j.emj.2016.06.009

    work page doi:10.1016/j.emj.2016.06.009 2016

  5. [5]

    Fast and stable partial least squares modelling: A benchmark study with theoretical com- ments

    Bj¨ orck, ˚ A., Indahl, U.G., 2017. Fast and stable partial least squares modelling: A benchmark study with theoretical com- ments. Journal of Chemometrics 31, e2898. doi:10.1002/cem.2898. e2898 cem.2898

    work page doi:10.1002/cem.2898 2017

  6. [6]

    JAX: composable trans- formations of Python+NumPy programs

    Bradbury, J., Frostig, R., Hawkins, P., John- son, M.J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., Zhang, Q., 2018. JAX: composable trans- formations of Python+NumPy programs. URL: http://github.com/google/jax

    2018

  7. [7]

    Chemo- metrics in analytical chemistry—part II: mod- eling, validation, and applications

    Brereton, R.G., Jansen, J., Lopes, J., Marini, F., Pomerantsev, A., Rodionova, O., Roger, J.M., Walczak, B., Tauler, R., 2018. Chemo- metrics in analytical chemistry—part II: mod- eling, validation, and applications. Analytical and Bioanalytical Chemistry 410, 6691–6704. doi:10.1007/s00216-018-1283-4

    work page doi:10.1007/s00216-018-1283-4 2018

  8. [8]

    Partial least squares discriminant analysis: taking the magic away

    Brereton, R.G., Lloyd, G.R., 2014. Partial least squares discriminant analysis: taking the magic away. Journal of Chemometrics 28, 213–225. doi:10.1002/cem.2609

    work page doi:10.1002/cem.2609 2014

  9. [9]

    Improved PLS algorithms

    Dayal, B.S., MacGregor, J.F., 1997. Improved PLS algorithms. Jour- nal of Chemometrics 11, 73–85. doi:10.1002/(SICI)1099-128X(199701)11:1<73::AID-CEM435>

    work page doi:10.1002/(sici)1099-128x(199701)11:1 1997

  10. [10]

    Near-infrared hy- perspectral imaging applications in food anal- ysis – improving algorithms and methodolo- gies

    Engstrøm, O.C.G., 2025. Near-infrared hy- perspectral imaging applications in food anal- ysis – improving algorithms and methodolo- gies. Ph.D. thesis. University of Copenhagen. doi:10.48550/arXiv.2510.13452

    work page doi:10.48550/arxiv.2510.13452 2025

  11. [11]

    IKPLS: improved kernel partial least squares and fast cross- validation algorithms for Python with CPU and GPU implementations using NumPy and JAX

    Engstrøm, O.C.G., Dreier, E.S., Jespersen, B.M., Pedersen, K.S., 2024. IKPLS: improved kernel partial least squares and fast cross- validation algorithms for Python with CPU and GPU implementations using NumPy and JAX. Journal of Open Source Software 9, 6533. doi:10.21105/joss.06533

    work page doi:10.21105/joss.06533 2024

  12. [12]

    Fast partition-based cross-validation with centering and scaling for XTX and XTY

    Engstrøm, O.C.G., Jensen, M.H., 2025. Fast partition-based cross-validation with centering and scaling for XTX and XTY. Journal of Chemometrics 39, e70008. doi:10.1002/cem.70008

    work page doi:10.1002/cem.70008 2025

  13. [13]

    Array programming with NumPy

    Harris, C.R., Millman, K.J., van der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M.H., Brett, M., Haldane, A., del R ´ ıo, J.F., Wiebe, M., Peterson, P., G´ erard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Ab- basi, H., Gohlke, C., Oliphant, ...

    2020

  14. [14]

    PLS regression meth- ods

    H¨ oskuldsson, A., 1988. PLS regression meth- ods. Journal of Chemometrics 2, 211–228. doi:10.1002/cem.1180020306

    work page doi:10.1002/cem.1180020306 1988

  15. [15]

    The h-principle in modelling with applications to chemo- metrics

    H¨ oskuldsson, A., 1992. The h-principle in modelling with applications to chemo- metrics. Chemometrics and Intelli- gent Laboratory Systems 14, 139–153. doi:10.1016/0169-7439(92)80099-P . pro- ceedings of the 2nd Scandinavian Symposium on Chemometrics. 13

    work page doi:10.1016/0169-7439(92)80099-p 1992

  16. [16]

    SIMPLS: An alter- native approach to partial least squares regression

    de Jong, S., 1993. SIMPLS: An alter- native approach to partial least squares regression. Chemometrics and Intelli- gent Laboratory Systems 18, 251–263. doi:10.1016/0169-7439(93)85002-X

    work page doi:10.1016/0169-7439(93)85002-x 1993

  17. [17]

    Liland, K.H., Mevik, B.H., Wehrens, R.,

  18. [18]

    Liland, K.H., Stefansson, P., Indahl, U.G.,

  19. [19]

    The PLS package: Principal component and partial least squares regression in r

    Mevik, B.H., Wehrens, R., 2007. The PLS package: Principal component and partial least squares regression in r. Journal of Statistical Software 18, 1–23. doi:10.18637/jss.v018.i02

    work page doi:10.18637/jss.v018.i02 2007

  20. [20]

    Rinnan, ˚ A., van den Berg, F., Engelsen, S.B.,

  21. [21]

    Shenk, J.S., Westerhaus, M.O., Berzaghi, P.,

  22. [22]

    PLS discriminant plots

    Sj¨ ostr¨ om, M., Wold, S., S¨ oderstr¨ om, B., 1986. PLS discriminant plots. Elsevier

    1986

  23. [23]

    NIR data exploration and re- gression by chemometrics—a primer

    Sørensen, K.M., van den Berg, F., Engelsen, S.B., 2021. NIR data exploration and re- gression by chemometrics—a primer. Near- Infrared Spectroscopy: Theory, Spectral Anal- ysis, Instrumentation, and Applications , 127– 189doi:10.1007/978-981-15-8648-4_7

    work page doi:10.1007/978-981-15-8648-4_7 2021

  24. [24]

    Partial least squares analysis with cross-validation for the two-class problem: a monte carlo study

    St ˚ ahle, L., Wold, S., 1987. Partial least squares analysis with cross-validation for the two-class problem: a monte carlo study. Journal of Chemometrics 1, 185–196. doi:10.1002/cem.1180010306

    work page doi:10.1002/cem.1180010306 1987

  25. [25]

    Estimation of principal com- ponents and related models by iterative least squares

    Wold, H., 1966. Estimation of principal com- ponents and related models by iterative least squares. Multivariate analysis , 391–420

    1966

  26. [26]

    Wold, S., Sj¨ ostr¨ om, M., Eriksson, L.,

  27. [362]

    doi: 10.1038/s41586-020-2649-2

    doi:10.1038/s41586-020-2649-2

  28. [720]

    doi: 10.1007/s00362-009-0251-7

    work page doi:10.1007/s00362-009-0251-7

  29. [1222]

    doi: 10.1016/j.trac.2009.07.007

    doi:10.1016/j.trac.2009.07.007 2009

  30. [1997]

    Jour- nal of Near Infrared Spectroscopy 5, 223–232

    Investigation of a local calibration pro- cedure for near infrared instruments. Jour- nal of Near Infrared Spectroscopy 5, 223–232. doi:10.1255/jnirs.115

    work page doi:10.1255/jnirs.115

  31. [2001]

    Chemometrics and Intel- ligent Laboratory Systems 58, 109–130

    PLS-regression: a basic tool of chemometrics. Chemometrics and Intel- ligent Laboratory Systems 58, 109–130. doi:10.1016/S0169-7439(01)00155-1. 14

    doi:10.1016/s0169-7439(01)00155-1

  32. [2009]

    TrAC Trends in Analytical Chemistry 28, 1201–

    Review of the most common pre- processing techniques for near-infrared spectra. TrAC Trends in Analytical Chemistry 28, 1201–

  33. [2020]

    Journal of Chemometrics 34, e3201

    Much faster cross-validation in PLSR- modelling by avoiding redundant calcula- tions. Journal of Chemometrics 34, e3201. doi:10.1002/cem.3201

    work page doi:10.1002/cem.3201

  34. [2026]

    URL: https://CRAN.R-project.org/package=pls

    pls: Partial least squares and principal component regression. URL: https://CRAN.R-project.org/package=pls. r package version 2.9-0