REVIEW 4 major objections 5 minor 35 references
The Price of Linear Time: Error Analysis of Structured Kernel Interpolation
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper proves the first rigorous error bounds for Structured Kernel Interpolation: inducing points should grow as $n^{d/3}$, and in up to three dimensions any fixed error tolerance is reachable in linear time.
desk verdict The SKI error analysis is mostly sound in the interior, but the advertised d≤3 linear-time guarantee is wrong at d=3 — the RHS of its own Eq. (4) is Θ(log n) there. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the multivariate tensor-product cubic convolutional interpolation error of Lemma 4.1: interpolating a smooth function on a regular grid with spacing $h$ in $d$ dimensions costs $O(c^d h^3)$, equivalently $O(c^d / m^{3/d})$ with $m$ inducing points. The proof inducts over coordinates, starting from the one-dimensional cubic convolution bound and multiplying the error by a weight-sum constant $c$ at each step, where $c$ bounds the sum of absolute interpolation weights in a single dimension. All later results — the SKI kernel bound $O(c^{2d}/m^{3/d})$, the Gram spectral-norm bound $O(n c^{2d}/m^{3/d})$, the score, posterior mean, and posterior covariance bounds — inherit their $m^{-3/d}$ decay and their $c^{2d}$ exponential factor from this lemma, and the $m \propto n^{d/3}$ recipe is just the inversion of the Gram bound, while the crossover dimension $d = 3$ comes from comparing that $m$ to $n/\log n$.
What would settle it
Take a smooth stationary kernel such as the RBF, place $n$ training points in $d = 2$ and $d = 4$, and for a grid of $m$ values measure the spectral-norm error $\|K - \tilde K\|_2$ of SKI with cubic interpolation on regular grids. If the error does not scale as $m^{-3/d}$ with a constant independent of $m$, or if in $d = 2$ the recipe $m = \mathrm{const}\cdot n^{2/3}$ fails to hold the error flat as $n$ grows (with the analogous prediction failing in $d = 4$), the central scaling claim is refuted; a second check is whether the posterior covariance error stops decreasing with $m$ once $d$ reaches 3, as Lemma 5.10 predicts.
Extended reading notes
Core claim
The paper's claim, stated as a corollary of its main scaling analysis, is that SKI with convolutional cubic interpolation is a guaranteed linear-time method for Gaussian process inference whenever the input dimension $d$ is at most three: for any tolerance $\epsilon > 0$, once the sample size $n$ is sufficiently large, choosing the number of inducing points from Theorem 4.5 yields both spectral-norm error $\|K - \tilde K\|_2 \le \epsilon$ and $O(n)$ complexity. The engine behind this is the elementwise kernel bound $|k(x,x') - \tilde k(x,x')| = O(c^{2d}/m^{3/d})$, obtained by extending the classical one-dimensional cubic convolution error analysis to tensor products in $d$ dimensions; the Gram-matrix bound then follows because the symmetric error matrix's spectral norm is controlled by its largest row sum, giving a factor of $n$. For $d > 3$ the same arithmetic flips: meeting a fixed tolerance forces $m$ to grow faster than $n/\log n$, so linear time can only be maintained if the error is allowed to grow with the sample. The same kernel-level bound is differentiated and pushed through the GP log-likelihood and posterior, producing explicit rates for score-function error, posterior mean error, and posterior covariance error.
Load-bearing premise
The entire scaling picture rests on Lemma 4.1's assumption that interpolating dimension by dimension with cubic convolution multiplies the error by at most a fixed constant $c$ per coordinate, so the $d$-dimensional error stays $O(c^d h^3)$; if the weight-sum constant is not bounded, or the error compounds faster than geometrically with the dimension, every bound in the paper—and the $m \propto n^{d/3}$ recipe—shifts.
Editorial extensions
If this is right
- Any user of SKI with cubic interpolation who wants Gram-matrix error below a fixed threshold can set $m \propto n^{d/3}$; the paper proves this is sufficient, and in $d \le 3$ the resulting complexity is $O(n)$ for sufficiently large $n$.
- In three dimensions and below, SKI is formally a linear-time algorithm with a user-chosen accuracy guarantee; above three dimensions, accuracy and linear time cannot both be fixed as $n$ grows.
- Hyperparameter learning is not silently corrupted: gradient ascent on the SKI log-likelihood visits, at an $O(1/K)$ rate over $K$ iterations, points whose true-gradient norm is small, within a ball whose radius is set by the SKI score error; with $m \propto n^{d/3}$ that score error grows only linearly in the sample size (aside from the response vector).
- Posterior mean error grows as $O((c^{2d}\max(T,n) + \sqrt{Tn}\,n)/m^{3/d})$, so the same inducing-point recipe keeps predictions accurate at the same rate; posterior covariance error is the fragile quantity, only guaranteed to decay with $m$ when $d < 3$.
Reading between the lines
- If the same analysis were run with a higher-order convolution kernel (error $O(h^q)$ per coordinate), the linear-time crossover would shift from $d = 3$ to $d = q$: the $d \le 3$ boundary is not a law of Gaussian processes but an artifact of cubic interpolation, and joint optimization of interpolation degree $L$ and grid size $m$ is the natural next target. (Editorial inference.)
- The exponential factor $c^{2d}$ and the covariance bound that only decays when $d < 3$ suggest SKI's predictive means stay accurate well past the point where its uncertainty estimates degrade — in higher dimensions users should trust point predictions more than variances. The paper proves the bounds; this prioritization is an editorial reading.
- The base case of the whole chain is the one-dimensional cubic convolution error bound, which needs a sufficiently smooth kernel; for rough stationary kernels such as Matérn with $\nu < 3/2$, the empirical interpolation order should drop below 3 and the $m^{-3/d}$ picture should break — a direct, testable boundary of the theory. (Editorial inference.)
- The paper's closing note reports that LLM assistants drafted most statements and attempted most proofs, with the authors supplying the initial kernel-error analysis and many model proofs being wrong; because every downstream bound inherits the appendix proofs, a careful independent re-derivation of Lemma 4.1 and Proposition 4.3 is the prudent reading step. (Editorial flag; the paper itself disclos
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops an error analysis for Structured Kernel Interpolation (SKI) with cubic convolutional interpolation. It derives elementwise and spectral-norm error bounds for the SKI kernel, Gram matrix, and cross-kernel matrix, and then applies these to bound the error of hyperparameter estimation (score function) and posterior mean and covariance. The central practical message is that the number of inducing points should scale as n^{d/3}, and the paper claims two dimensionality regimes: for d ≤ 3 any fixed spectral-norm error tolerance can be achieved in linear time for sufficiently large n, while for d > 3 the error must grow with n to maintain linear time. The appendix contains the proofs of the main lemmas and theorems.
Significance. If the results were correct, they would provide a rigorous foundation for a widely used scalable Gaussian-process approximation and give a concrete rule for choosing the number of inducing points. The paper also makes a useful connection to inexact-gradient optimization for the SKI log-likelihood. However, the main advertised claim fails at d = 3 because of an arithmetic error, and several downstream statements suffer from internal inconsistencies or incorrect invocations of external results. As it stands, the contribution cannot be relied upon without major corrections.
major comments (4)
- [§4.2, Corollary 4.7 and Eq. (4)] The assertion that the right-hand side of Eq. (4) decreases to 0 for d ≤ 3 is arithmetically wrong at d = 3. Setting d = 3 in Eq. (4) gives (1 + 2c^3) K' 8 c^6 D^3 · n (log n) / n = const · log n, which grows without bound as n → ∞. Therefore, for a fixed error tolerance ε smaller than the constant in front of log n, the condition ε ≥ RHS is eventually violated, and Theorem 4.5 then forces m = Θ(n/ε) = Θ(n), so m log m = Θ(n log n), not O(n). The claimed 'd ≤ 3' linear-time regime is false; only d < 3 has the stated property. This is a load-bearing error in the paper's headline result.
- [§5.1, Lemma 5.6] The first displayed inequality contains the factor ‖y‖√p max(...), while the final equality asserts the bound is ‖y‖²O(...). The proof of T1 bounds the derivative of the quadratic form y^T[(K̃+σ²I)^{-1} - (K+σ²I)^{-1}]y by a term proportional to ‖y‖² times the matrix derivative norm. Thus the correct statement should use ‖y‖² consistently, and the log-determinant term contributes without a ‖y‖ factor. This inconsistency changes the magnitude of the score-function error bound and therefore the neighborhood radius in Theorem 5.7.
- [§5.1, Theorem 5.7] The stated inequality bounds max_{k=0,...,N-1} ‖∇L(θ_k)‖² by 2µ(L* - L(θ_0))/K + ε_g²/(2µ). The right-hand side decreases with K, but for a nonconvex smooth function the maximum over all iterates cannot be forced below a small constant by increasing K: early iterates can have arbitrarily large gradients. The cited result of Stonyakin et al. (2023) gives a bound on the minimum, or on a suitable average, of the squared gradient norms over iterates, not on the maximum. As written, the theorem is false and does not support the 'at least one iterate approaches a neighborhood' conclusion stated in Section 5.1.
- [Appendix B.1.1, Lemma 4.1] The induction proof applies the one-dimensional Keys error bound to intermediate interpolants g_i, but these functions are only piecewise cubic after the first interpolation and do not inherit the smoothness (e.g., bounded third derivatives) required by Keys' theorem. The proof also asserts a uniform weight-sum bound c based on boundedness of u, but Definition 3.3's constant c bounds sums over arbitrary data points, not the fixed four-term sums appearing in the tensor-product recursion; a separate constant is needed. Because Lemma 4.1 supplies the O(c^{2d} m^{-3/d}) factor used in every subsequent bound, this proof must be repaired or the rates re-derived.
minor comments (5)
- [§6] The disclosure that 'LLM attempts at proofs were wrong' is an explicit limitation statement in the manuscript. The mathematical errors identified above are consistent with that disclosure, and all proofs should be independently verified in any revision.
- [Title and text] There are numerous typographical issues: 'Structured Kerne l Interpolation' on the title, 'primariliy' in Section 3.1, 'F ord' in Corollary 4.7, and 'constants' instead of 'constant' in Lemma 5.6.
- [§4.2, Corollary 4.6] The text introduces constants K and C, but Eq. (4) contains only C; the constant K is never used. Please remove or clarify.
- [§5.1, Theorem 5.7] The gradient error bound is denoted ε_g in the theorem but ε_G in Lemma 5.6. Unify the notation.
- [§3.3, Definition 3.3] The constant c is defined using a sum over an arbitrary set of points {x_i}_{i=1}^n, but later proofs use it as a bound for the fixed four-term sums of the cubic kernel. The relationship between these two uses should be stated explicitly.
Circularity Check
No significant circularity: core bounds are derived from external results (Keys 1981; Stonyakin et al. 2023; Horn & Johnson 2012) without fitted parameters relabeled as predictions. The only self-citation is non-load-bearing, and the d = 3 boundary issue is an arithmetic correctness problem, not a circular step.
full rationale
The derivation chain is self-contained and not circular. Lemma 4.1 is proved by induction from the one-dimensional Keys (1981) interpolation bound, with the constant c explicitly defined in Definition 3.3; it does not assume the target error rates. Proposition 4.3 and Lemma 4.4 follow from Lemma 4.2 via standard row/column-sum spectral-norm bounds. Theorem 4.5 and Corollary 4.6 are algebraic consequences of setting the Proposition 4.3 bound below epsilon and substituting h = O(m^{-1/d}); Corollary 4.7 then classifies the resulting exponents. The hyperparameter and posterior results rely on external results (Stonyakin et al. 2023; Horn & Johnson 2012; Bach 2013; Musco & Musco 2017) and reuse the Gram error bound rather than assuming the desired conclusion. No fitted parameter is renamed as a prediction. The only self-citation, Moreno et al. (2023), appears in Related Work and is not used as a premise in any proof, so it is not load-bearing. The paper's Section 6 disclosure that LLM proof attempts were often wrong is a transparency statement about correctness risk, not evidence of circularity. Separately, the proof of Corollary 4.7 appears to contain an arithmetic error at d = 3: the right-hand side of Eq. (4) is Theta(log n), not o(1), so the stated d <= 3 linear-time claim may fail at d = 3; this is a correctness issue, not circularity, and does not increase the circularity score.
Assumptions & free parameters
assumptions (4)
- standard math The one-dimensional cubic convolutional interpolation error is O(h^3) (Keys 1981), extended to tensor products with O(c^d h^3).
- domain assumption For any x, the sum of absolute interpolation weights over grid points is bounded by c^d for a constant c>1 (Lemma B.1, Definition 3.3).
- domain assumption The kernel and its hyperparameter partial derivatives are valid SPD kernels, bounded by M, and the log-likelihood is µ-smooth (Assumptions 5.1, 5.2, 5.3, 5.8).
- domain assumption The inducing points form a regular grid with equal spacing h and m=(2D/h)^d on a fixed domain [−D,D]^d with 2D divisible by h.
Cite this review
Pith. "Pith review of The Price of Linear Time: Error Analysis of Structured Kernel Interpolation." pith.science (2026). https://pith.science/paper/HEHXPEPA
@misc{pith2026250200298,
author = {Pith},
title = {Pith review of: The Price of Linear Time: Error Analysis of Structured Kernel Interpolation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HEHXPEPA}},
note = {Machine review of arXiv:2502.00298}
}
abstract
Structured Kernel Interpolation (SKI) (Wilson et al. 2015) helps scale Gaussian Processes (GPs) by approximating the kernel matrix via interpolation at inducing points, achieving linear computational complexity. However, it lacks rigorous theoretical error analysis. This paper bridges the gap: we prove error bounds for the SKI Gram matrix and examine the error's effect on hyperparameter estimation and posterior inference. We further provide a practical guide to selecting the number of inducing points under convolutional cubic interpolation: they should grow as $n^{d/3}$ for error control. Crucially, we identify two dimensionality regimes governing the trade-off between SKI Gram matrix spectral norm error and computational complexity. For $d \leq 3$, any error tolerance can achieve linear time for sufficiently large sample size. For $d > 3$, the error must increase with sample size to maintain linear time. Our analysis provides key insights into SKI's scalability-accuracy trade-offs, establishing precise conditions for achieving linear-time GP inference with controlled approximation error.
Reference graph
Works this paper leans on
-
[1]
Alaa, A. M. and van der Schaar, M. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances in Neural Information Processing Systems, 30, 2017
work page 2017
-
[2]
Sharp analysis of low-rank kernel matrix approximations
Bach, F. Sharp analysis of low-rank kernel matrix approximations. In Conference on learning theory, pp.\ 185--209. PMLR, 2013
work page 2013
-
[3]
Ban, H., Riemens, E. H., and Rajan, R. T. Malleable kernel interpolation for scalable structured gaussian process. In 2024 32nd European Signal Processing Conference (EUSIPCO), pp.\ 997--1001. IEEE, 2024
work page 2024
-
[4]
Burt, D., Rasmussen, C. E., and Van Der Wilk, M. Rates of convergence for sparse variational gaussian process regression. In International Conference on Machine Learning, pp.\ 862--871. PMLR, 2019
work page 2019
-
[5]
Burt, D. R., Rasmussen, C. E., and Van Der Wilk, M. Convergence of sparse variational inference in gaussian processes regression. Journal of Machine Learning Research, 21 0 (131): 0 1--63, 2020
work page 2020
-
[6]
Smooth optimization with approximate gradient
d'Aspremont, A. Smooth optimization with approximate gradient. SIAM Journal on Optimization, 19 0 (3): 0 1171--1183, 2008
work page 2008
-
[7]
Deisenroth, M. P., Fox, D., and Rasmussen, C. E. Gaussian processes for data-efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37 0 (2): 0 408--423, 2015
work page 2015
-
[8]
Deep gaussian processes for air quality inference
Desai, A., Gujarathi, E., Parikh, S., Yadav, S., Patel, Z., and Batra, N. Deep gaussian processes for air quality inference. In Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD), pp.\ 278--279, 2023
work page 2023
Show all 35 references
-
[9]
First-order methods of smooth convex optimization with inexact oracle
Devolder, O., Glineur, F., and Nesterov, Y. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146 0 (1): 0 37--75, 2014
2014
-
[10]
Frazier, P. I. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018
2018 arXiv
-
[11]
Q., Bindel, D., and Wilson, A
Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., and Wilson, A. G. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems, 31, 2018
2018
-
[12]
Q., and Murray-Smith, R
Girard, A., Rasmussen, C., Candela, J. Q., and Murray-Smith, R. Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Advances in neural information processing systems, 15, 2002
2002
-
[13]
Hensman, J., Fusi, N., and Lawrence, N. D. Gaussian processes for big data. In Uncertainty in Artificial Intelligence, 2013
2013
-
[14]
Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012
2012
-
[15]
diseasegps: auxiliary diagnostic system for genetic disorders based on genotype and phenotype
Huang, D., Jiang, J., Zhao, T., Wu, S., Li, P., Lyu, Y., Feng, J., Wei, M., Zhu, Z., Gu, J., et al. diseasegps: auxiliary diagnostic system for genetic disorders based on genotype and phenotype. Bioinformatics, 39 0 (9): 0 btad517, 2023
2023
-
[16]
A., and Wilson, A
Kapoor, S., Finzi, M., Wang, K. A., and Wilson, A. G. G. Skiing on simplices: Kernel interpolation on the permutohedral lattice for scalable gaussian processes. In International Conference on Machine Learning, pp.\ 5279--5289. PMLR, 2021
2021
-
[17]
Cubic convolution interpolation for digital image processing
Keys, R. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing, 29 0 (6): 0 1153--1160, 1981
1981
-
[18]
Kolmogorov, A. N. Wienersche spiralen und einige andere interessante kurven im hilbertschen raum. CR (Doklady) Acad. Sci. URSS (NS), 26: 0 115--118, 1940
1940
-
[19]
Leçons élémentaires sur les mathématiques
Lagrange, J.-L. Leçons élémentaires sur les mathématiques. Imprimerie de la République, 1795
-
[20]
A., Padhy, S., Antoran, J., Tripp, A., Terenin, A., Szepesvari, C., Hern \'a ndez-Lobato, J
Lin, J. A., Padhy, S., Antoran, J., Tripp, A., Terenin, A., Szepesvari, C., Hern \'a ndez-Lobato, J. M., and Janz, D. Stochastic gradient descent for gaussian processes done right. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevie...
2024
-
[21]
and Onnela, J.-P
Liu, G. and Onnela, J.-P. Bidirectional imputation of spatial gps trajectories with missingness using sparse online gaussian process. Journal of the American Medical Informatics Association, 28 0 (8): 0 1777--1784, 2021
2021
-
[22]
Entrywise error bounds for low-rank approximations of kernel matrices
Modell, A. Entrywise error bounds for low-rank approximations of kernel matrices. arXiv preprint arXiv:2405.14494, 2024
2024 arXiv
-
[23]
SKI to go faster: Accelerating toeplitz neural networks via asymmetric kernels
Moreno, A., Mei, J., and Walters, L. SKI to go faster: Accelerating toeplitz neural networks via asymmetric kernels. arXiv preprint arXiv:2305.09028, 2023
2023 arXiv
-
[24]
C., Burr, C., Cowls, J., Joshi, I., Taddeo, M., and Floridi, L
Morley, J., Machado, C. C., Burr, C., Cowls, J., Joshi, I., Taddeo, M., and Floridi, L. The ethics of ai in health care: a mapping review. Social Science & Medicine, 260: 0 113172, 2020
2020
-
[25]
and Musco, C
Musco, C. and Musco, C. Recursive sampling for the nystrom method. Advances in neural information processing systems, 30, 2017
2017
-
[26]
Rasmussen, C. E. and Williams, C. K. I. Gaussian processes for machine learning. MIT press, 2006
2006
-
[27]
and Ghahramani, Z
Snelson, E. and Ghahramani, Z. Sparse gaussian processes using pseudo-inputs. In Advances in neural information processing systems, pp.\ 1257--1264, 2006
2006
-
[28]
Stopping rules for gradient methods for non-convex problems with additive noise in gradient
Stonyakin, F., Kuruzov, I., and Polyak, B. Stopping rules for gradient methods for non-convex problems with additive noise in gradient. Journal of Optimization Theory and Applications, 198 0 (2): 0 531--551, 2023
2023
-
[29]
Titsias, M. K. Variational model selection for sparse gaussian process regression. Report, University of Manchester, UK, 2009
2009
-
[30]
Connections and equivalences between the nystr \"o m method and sparse variational gaussian processes
Wild, V., Kanagawa, M., and Sejdinovic, D. Connections and equivalences between the nystr \"o m method and sparse variational gaussian processes. arXiv preprint arXiv:2106.01121, 2021
2021 arXiv
-
[31]
and Nickisch, H
Wilson, A. and Nickisch, H. Kernel interpolation for scalable structured gaussian processes ( KISS-GP ). In International conference on machine learning, pp.\ 1775--1784. PMLR, 2015
2015
-
[32]
and Wild, V
Wynne, G. and Wild, V. Variational gaussian processes: A functional analysis view. In International Conference on Artificial Intelligence and Statistics, pp.\ 4955--4971. PMLR, 2022
2022
-
[33]
Q., and Wilson, A
Yadav, M., Pleiss, G., Gardner, J., Weinberger, K. Q., and Wilson, A. G. Faster kernel interpolation for gaussian processes. In Proceedings of the 38th International Conference on Machine Learning, pp.\ 11279--11288. PMLR, 2021
2021
-
[34]
Kernel interpolation with sparse grids
Yadav, M., Sheldon, D., and Musco, C. Kernel interpolation with sparse grids. In Advances in Neural Information Processing Systems, 2022
2022
-
[35]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.