REVIEW 6 minor 15 references
New Bounds for Sparse Variational Gaussian Processes
T0 review · 0 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Sparse GP training gets a strictly tighter bound for free
desk verdict A strictly tighter SVGP bound at unchanged cost, with correct proofs and honest experiments; worth refereeing and citing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the modified conditional variational distribution $q(f|u) = \mathcal{N}(f|K_{fu}K_{uu}^{-1}u, (K_{ff}-Q_{ff})^{1/2} V (K_{ff}-Q_{ff})^{1/2})$, where $V$ is a diagonal matrix of $N$ variational scaling parameters applied through the symmetric square root of the residual covariance $K_{ff}-Q_{ff}$. The identities doing the work are Lemma 3.1, which collapses the KL divergence between $q(f|u)$ and $p(f|u)$ to $\frac{1}{2}\sum_{i=1}^N (v_i - \log v_i - 1)$, and Lemma 3.2, which gives the expected log likelihood as $\log \mathcal{N}(y|K_{fu}K_{uu}^{-1}u, \sigma^2 I) - \frac{1}{2\sigma^2}\sum_{i=1}^N v_i(k_{ii}-q_{ii})$. Optimizing $V$ yields $V^* = \operatorname{diag}\left[\left(I + \sigma^{-2}(K_{ff}-Q_{ff})\right)^{-1}\right]$, which is exactly the diagonal approximation to the full inverse in Eq. (12); the new per-point log regularizer is tighter than the original trace term because $\log(1+a) \le a$.
What would settle it
On a regression with $M$ small and a kernel lengthscale much shorter than the spacing between inducing points, compute both ELBOs and compare the learned noise variance with the exact GP's: if the new bound does not reduce the overestimation of $\sigma^2$, the claim that off-diagonal residual correlations are unimportant for the empirical gain is refuted.
Extended reading notes
Core claim
The central claim is Proposition 3.3. After replacing $p(f|u)$ in the variational distribution by $q(f|u) = \mathcal{N}(f|K_{fu}K_{uu}^{-1}u, (K_{ff}-Q_{ff})^{1/2} V (K_{ff}-Q_{ff})^{1/2})$ with a diagonal matrix $V$ of $N$ positive parameters, the optimal setting is $v_i^* = \left(1 + \frac{k_{ii}-q_{ii}}{\sigma^2}\right)^{-1}$, and the collapsed ELBO becomes $F_{\mathrm{new}}$ as in Eq. (16). The bound satisfies $\log p(y) \ge F_{\mathrm{new}} \ge F$, and $F_{\mathrm{new}} > F$ whenever $K_{ff} \ne Q_{ff}$. The tightened bound keeps the DTC likelihood term unchanged and replaces the trace penalty $-\frac{1}{2\sigma^2}\operatorname{tr}(K_{ff}-Q_{ff})$ with a per-point log penalty; both the collapsed and minibatch forms are strictly tighter than their SVGP counterparts. The paper also derives a spherical-$v$ version for non-Gaussian likelihoods with $O(NM^2)$ cost, and the practical claim is that this reduces underfitting bias when the number of inducing points is limited, particularly the SVGP tendency to overestimate the noise variance $\sigma^2$.
Load-bearing premise
The empirical claim of meaningful bias reduction rests on the assumption that off-diagonal posterior correlations among training function values are small enough that replacing the full inverse by its diagonal captures most of the tightening, while the variational conditional keeps its mean at the prior value.
Editorial extensions
If this is right
- The collapsed bound $F_{\mathrm{new}}$ is strictly tighter than the standard SVGP bound whenever the Nyström approximation $Q_{ff}$ differs from $K_{ff}$, so training with limited inducing points is less biased.
- The minibatch ELBO is also strictly tighter than the uncollapsed bound of Hensman et al. (2013), and implementing it requires replacing one per-data-point term in existing sparse GP code.
- On the tested datasets, the learned noise variance moves closer to the exact-GP estimate, and test log likelihoods improve over SVGP and the Artemev et al. (2021) bound.
- The relaxation extends to non-Gaussian likelihoods through a single scalar $v$ per GP, preserving $O(NM^2)$ batch cost and $O(M^3)$ minibatch cost.
- For Bayesian GP-LVM, the same per-point log regularizer yields a new collapsed bound with only a minor modification to existing code.
Reading between the lines
- An implication the author leaves implicit is that the relaxation corrects covariance underfitting but not mean underfitting: because $q(f|u)$ keeps the prior conditional mean $E[f|u]$, residual bias in predictive means can persist when the posterior mean shift $\tilde{K}(\tilde{K}+\sigma^2 I)^{-1}(y-E[f|u])$ is large.
- The diagonal approximation in Remark 3.4 suggests that the practical gain over SVGP should grow as the residual covariance $K_{ff}-Q_{ff}$ becomes more diagonally dominant; a testable prediction is that the improvement is largest when inducing points are dense relative to the kernel correlation length.
- The per-point form of the new regularizer invites natural extensions to structured approximations, such as block-diagonal $V$ for localized off-diagonal correlations or per-output/per-layer scaling parameters in multi-output and deep GP models.
- The equality $V^* = \operatorname{diag}[(I + \sigma^{-2}(K_{ff}-Q_{ff}))^{-1}]$ suggests the bound could be tightened further by choosing a richer approximating family for the full inverse, with the present diagonal choice being the cheapest member of that family.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper revisits sparse variational Gaussian process regression (Titsias, 2009) and shows that the conditional-GP assumption in the variational distribution q(f,u) = p(f|u)q(u) can be relaxed during training. The author proposes q(f|u) = N(f | E[f|u], (Kff - Qff)^{1/2} V (Kff - Qff)^{1/2}) with a diagonal V = diag(v1,...,vN), proves closed-form expressions for the KL term (Lemma 3.1) and the expected log-likelihood (Lemma 3.2), and optimizes the ELBO analytically over q(u) and V to obtain a collapsed bound F_new = log N(y | 0, Qff + sigma^2 I) - (1/2) sum_i log(1 + (kii - qii)/sigma^2) (Proposition 3.3). Since log(1+a) <= a, F_new dominates the standard SVGP bound of Eq. (8) pointwise, and strictly when Kff differs from Qff, at unchanged O(N M^2) cost. The paper extends the result to stochastic minibatch training (Section 3.2) and to non-Gaussian likelihoods via a spherical-v variant (Proposition 3.5), connects the bound to Artemev et al. (2021) and Bui et al. (2017), and reports experiments on a 1-D toy problem, three medium-scale UCI datasets, eight large-scale datasets, and Poisson regression. Predictive inference intentionally reuses the standard SVGP predictive form, so the empirical gains arise from less biased hyperparameter learning.
Significance. I verified Lemma 3.1, Lemma 3.2, and Proposition 3.3 directly: the KL reduction, the trace identity tr(A^{1/2} V A^{1/2}) = sum_i v_i (kii - qii), the closed-form optimum v_i* = (1 + (kii - qii)/sigma^2)^{-1}, and the inequality F_new >= F are all correct, and the O(N M^2) cost claim follows from the Nystrom form of Qff. The tightness gain is parameter-free in the precise sense that v_i* is closed form and the domination F_new >= F holds for every hyperparameter setting, so the improvement is not an artifact of fitting. The stochastic bound (18)-(19) and the non-Gaussian spherical bound (23) inherit the tightness, and the connections to Artemev et al. (2021) (via Jensen), Bui et al. (2017) (no alpha-equivalence), and Zhu et al. (2023) (cost contrast) are accurate. The empirical section is honest: gains are clear on Pol and Bike and on several large-scale benchmarks, and the paper reports datasets (Elevators; KeggDirected, KEGGU, Song) where the new bound makes little difference.
minor comments (6)
- [Remark 3.4] The notation V* = diag[I + sigma^{-2}(Kff - Qff)]^{-1} is ambiguous: it should be read as the inverse of the diagonal matrix diag(I + sigma^{-2}(Kff - Qff)), i.e., the diagonal matrix with entries (1 + (kii - qii)/sigma^2)^{-1}, but as printed it could be mistaken for the diagonal of the inverse (I + sigma^{-2}(Kff - Qff))^{-1}, which is a different object. Please rewrite this expression to remove the ambiguity.
- [Remark 3.4] The statement that q(f|u) 'more accurately approximates the covariance' of p(f|u,y) goes beyond what is proven. The established facts are only V* <= I (so the covariance shrinks in the PSD order) and the ELBO domination in Proposition 3.3; closeness of the diagonal-truncated inverse to (I + sigma^{-2}(Kff - Qff))^{-1} in any matrix norm does not follow in general. Qualifying this clause as motivation rather than a proved fact would make the paper more precise.
- [Section 5.3] The claim that the results 'provide a clear improvement compared to training with the previous SVGP bound' is stronger than Tables 2 and 4 support: on KeggDirected, KEGGU, and Song the differences are within one standard error of zero, and on 3dRoad at M = 2048 (and Song at M = 1024) SVGP-new is marginally worse. I recommend rewording to state that gains are concentrated on several datasets with parity elsewhere.
- [Proposition 3.5] Proposition 3.5 is stated without a proof or a reference: the KL part follows from Lemma 3.1, but the marginal formula for q(fi) and the O(N M^2) cost argument deserve at least a sentence, or a pointer to the spherical-V derivation in Appendix B.4.
- [Appendix D.3 / Section 5.4] For the NYBikes experiments the learned v is reported to be below 0.01; at that value the KL penalty (N/2)(v - log v - 1) is on the order of hundreds of nats, so the reported improvements must come from large likelihood gains. Reporting the final ELBO values or the axis scale of Figure 3 (right) would make these results checkable.
- [Sections 3 and 5.3] Notation and typography: the conditional covariance matrix is rendered inconsistently as 'eKff' and as Ktilde_ff, and 'Matérn3/2' in Section 5.3 should be 'Matérn 3/2'; please make these consistent.
Circularity Check
No significant circularity: the tighter bound is derived by closed-form ELBO optimization, not by fitting or self-citation.
full rationale
The paper's central result is a closed-form tighter ELBO, not a fitted prediction. Lemma 3.1 and Lemma 3.2 are direct Gaussian identities: for same-mean Gaussians with covariances A^{1/2}VA^{1/2} and A, the KL is 1/2(-log det V - N + tr V), and E||y-f||^2 = ||y - mean(q)||^2 + tr(VA), which immediately yield Eq. (15). Proposition 3.3 then solves the ELBO optimization: the derivative of -1/2[v_i(1+d_i)-log v_i-1] with respect to v_i gives v_i*=(1+d_i)^{-1}; substituting and using log(1+d_i) <= d_i gives Eq. (16) with F_new >= F pointwise in all hyperparameters. The improvement is therefore a mathematical identity, not an artifact of fitting: v_i* is a closed-form function of k_ii-q_ii and sigma^2, and the bound inequality holds before any training. Test claims are measured on held-out data against exact GP and external baselines (ODVGP, SOLVE-GP), so there is no fitted input being renamed as a prediction. The only self-citation to Titsias (2009) is the baseline variational family being generalized; its derivation is reproduced in Appendix A and is not load-bearing for the new bound. The diagonal approximation of (I+sigma^{-2}(K-Q))^{-1} is an explicit modelling choice (Remark 3.4) and an admitted limitation, not a circular step. No pattern from the enumerated circularity kinds is present.
Assumptions & free parameters
free parameters (3)
- per-point variational scale v_i =
v_i* = (1 + (k_ii - q_ii)/sigma^2)^{-1}
- spherical scale v (non-Gaussian case) =
approx 0.675 (toy Poisson), < 0.01 (NYBikes)
- noise variance sigma^2 and kernel hyperparameters (sigma_f, lengthscales) =
dataset-dependent (e.g., sigma^2 = 0.087 for SGPR-new on the 1-D Snelson example)
assumptions (4)
- domain assumption Standard zero-mean GP regression with Gaussian homoscedastic noise (Eq. 1-2).
- standard math K_ff - Q_ff is positive semidefinite for a PSD kernel, so k_ii - q_ii >= 0 and its square root is well-defined (Eq. 12-13, Lemmas 3.1-3.2).
- ad hoc to paper The variational family restriction: q(f|u) shares the mean E[f|u] and uses a diagonal V (Eq. 13).
- standard math Standard Gaussian integration and matrix identities, plus concavity of the v-objective used to justify optimal v_i*.
Cite this review
Pith. "Pith review of New Bounds for Sparse Variational Gaussian Processes." pith.science (2026). https://pith.science/paper/KGOPUXT3
@misc{pith2026250208730,
author = {Pith},
title = {Pith review of: New Bounds for Sparse Variational Gaussian Processes},
year = {2026},
howpublished = {\url{https://pith.science/paper/KGOPUXT3}},
note = {Machine review of arXiv:2502.08730}
}
abstract
Sparse variational Gaussian processes (GPs) construct tractable posterior approximations to GP models. At the core of these methods is the assumption that the true posterior distribution over training function values ${\bf f}$ and inducing variables ${\bf u}$ is approximated by a variational distribution that incorporates the conditional GP prior $p({\bf f} | {\bf u})$ in its factorization. While this assumption is considered as fundamental, we show that for model training we can relax it through the use of a more general variational distribution $q({\bf f} | {\bf u})$ that depends on $N$ extra parameters, where $N$ is the number of training examples. In GP regression, we can analytically optimize the evidence lower bound over the extra parameters and express a tractable collapsed bound that is tighter than the previous bound. The new bound is also amenable to stochastic optimization and its implementation requires minor modifications to existing sparse GP code. Further, we also describe extensions to non-Gaussian likelihoods. On several datasets we demonstrate that our method can reduce bias when learning the hyperparameters and can lead to better predictive performance.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
o . 14 By maximizing wrt v we obtain v∗ = 1 +tr(Kff−Qff) N σ2 −1 , and by substituting this back into the bound we obtain Artemev et al. (2021)’s tighter bound on the initial trace regularization term. Overall this collapsed bound has the form log p(y) ≥ log N (y|0, Qff + σ2I) − N 2 log 1 +tr(Kff − Qff) N σ2 . (36) This collapsed bound is what the method ...
work page 2021
-
[2]
− 1 2σ2 tr yyT − 2y(KfuK−1 uu u)⊤ + (KfuK−1 uu u)(KfuK−1 uu u)⊤ + Kff − Qff = log N (y|KfuK−1 uu u, σ2I) − 1 2σ2 tr(Kff − Qff). (27) where we highlighted with blue a term in the third line to contrast it with a similar term when proving Lemma 3.2 in Appendix B.4. The ELBO in Equation (26) is written as log p(y) ≥ Z q(u) logN (y|KfuK−1 uu u, σ2I)p(u) q(u) ...
work page 2003
-
[3]
σ2 σ2 f ℓ2 Exact GP 0.0715 0.712 0.597 SVGP-new 0.087 0.485 0.615 SVGP 0.108 0.331 0.617 D
Hyperparameter values in 1-D Snelson example. σ2 σ2 f ℓ2 Exact GP 0.0715 0.712 0.597 SVGP-new 0.087 0.485 0.615 SVGP 0.108 0.331 0.617 D. Further experimental details and results For all regression experiments (apart from the toy Snelson 1-D dataset) we repeat the runs for five times using different random training and test splits. By following Wang et al...
work page 2019
-
[4]
Lloyd, C., Gunter, T., Osborne, M., and Roberts, S
doi: 10.1109/TNNLS.2019.2957109. Lloyd, C., Gunter, T., Osborne, M., and Roberts, S. Varia- tional inference for gaussian process modulated poisson processes. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pp. 1814–1822, Lille, France, 07–09 Jul
-
[9]
− 1 2σ2 tr h yyT − 2y(KfuK−1 uu u)⊤ + (KfuK−1 uu u)(KfuK−1 uu u)⊤ + (Kff − Qff) 1 2 V(Kff − Qff) 1 2 i = log N (y|KfuK−1 uu u, σ2I) − 1 2σ2 tr(V(Kff − Qff)), (35) where we used that tr((Kff − Qff) 1 2 V(Kff − Qff) 1 2 ) = tr(V(Kff − Qff)). Now since V is a diagonal matrix we have tr(V(Kff − Qff)) =PN i=1 vi(kii − qii) which completes the proof. B.3. Proof...
work page 2021
-
[14]
SGPR methods use M = 2048 inducing points initialized by k-means
The two plots in each column correspond to the same dataset: first row shows the ELBO (or log-likelihood) for all four methods (Exact GP, SGPR, SGPR-new and SGPR-artemev) with the number of iterations and the plot in the second row shows the corresponding values for σ2. SGPR methods use M = 2048 inducing points initialized by k-means. For these two first ...
-
[15]
Best mean values are highlighted
Test RMSE values of large scale regression datasets with standard errors in parentheses. Best mean values are highlighted. Kin40k Protein KeggDirected KEGGU 3dRoad Song Buzz HouseElectric N 25,600 29,267 31,248 40,708 278,319 329,820 373,280 1,311,539 d 8 9 20 27 3 90 77 9 From Shi et al. (2020) ODVGP 1024 + 10240.183(0.001) 0.625(0.004) 0.176(0.012) 0.15...
work page 2020
-
[18]
For the Bike dataset the initial train size (see e.g., Table 7 in Shi et al. (2020)) isN = 11122(with d =
work page 2020
Show all 15 references
-
[1024]
Large scale regression datasets The experimental settings are chosen to match the ones from Wang et al
D.2. Large scale regression datasets The experimental settings are chosen to match the ones from Wang et al. (2019) and Shi et al. (2020), where we used GPs with a Mat´ern32 kernel (with common lengthscale). Following these settings, for all datasets we train for 100 epochs us...
2019
-
[1978]
and Rasmussen, C
Qui˜nonero-Candela, J. and Rasmussen, C. E. A unifying view of sparse approximate Gaussian process regression. Journal of Machine Learning Research , 6:1939–1959,
1939
-
[2009]
A tutorial on sparse gaussian processes and variational inference
Leibfried, F., Dutordoir, V ., John, S., and Durrande, N. A tutorial on sparse gaussian processes and variational inference. arXiv preprint arXiv:2012.13962,
2012 arXiv
-
[2015]
Shi, J., Titsias, M
PMLR. Shi, J., Titsias, M. K., and Mnih, A. Sparse orthogonal variational inference for gaussian processes. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of ...
1932
-
[2018]
Hensman, J., Fusi, N., and Lawrence, N. D. Gaussian processes for big data. arXiv preprint arXiv:1309.6835,
-
[2020]
M., and Murillo-Fuentes, J
Havasi, M., Hern´andez-Lobato, J. M., and Murillo-Fuentes, J. J. Deep Gaussian processes with decoupled inducing inputs. arXiv preprint arXiv:1801.02939,
-
[2021]
A., Pleiss, G., Gardner, J
Wang, K. A., Pleiss, G., Gardner, J. R., Tyree, S., Wein- berger, K. Q., and Wilson, A. G. Exact Gaussian processes on a million data points. arXiv preprint arXiv:1903.08114,
1903 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.