REVIEW 5 major objections 5 minor 26 references
On the Usage of Gaussian Process for Efficient Data Valuation
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Data valuation for Gaussian Process models reduces to a utility/aggregation decomposition and can be computed by Schur-complement updates of the Integrated Variance instead of retraining on every subset.
desk verdict The GP update machinery is legitimate, but Equation (9) and Algorithm 1 do not compute Shapley values, so the paper's central claim fails as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the canonical decomposition of Definition 3.1, which writes a data valuation index as $DV_i(\Phi, Agg_i, f, D) = Agg_i(\{\Phi(f(x;A)), A \subset D\})$, separating the utility function $\Phi$ from the aggregation procedure $Agg_i$. The paper's concrete engine is the Integrated Variance $IV(D) = \int \mathrm{Var}(Z - \hat{Z}_D)\,dP_X$, a Bayesian measure of the GP predictor's leftover uncertainty, and the Schur-complement update formula of Theorem 2.1 that computes $IV(A \cup \{i\})$ from quantities already available for $A$. Algorithm 1 runs this update along random permutations of the data and accumulates Shapley-style increments, so each Monte Carlo pass costs a sequence of low-rank updates rather than full retraining.
What would settle it
Run Algorithm 1 on a regression dataset and, for a few thousand randomly sampled subsets, actually refit the GP, including kernel hyperparameters, on each subset, compute the Integrated Variance of the refitted model, and compare the resulting Shapley estimates with the paper's conditional-GP estimates; if the rankings diverge substantially, the cheap updates are not estimating the retrained-model valuations. A simpler check is to verify whether $IV(\varnothing)$ is well-defined in the Universal Kriging variant used in Algorithm 1, since the empty-set predictor is not specified.
Extended reading notes
Core claim
The central claim is that the computational bottleneck of data valuation is not the aggregation rule but the need to access a utility function on every sub-model, and that Gaussian Process regression removes this bottleneck for a natural utility. For a GP, a 'sub-model' trained on a subset is just the same Gaussian process conditioned on fewer observations, so its predictive distribution, and the Integrated Variance measuring remaining uncertainty, can be written explicitly from the covariance matrix of the full dataset. The paper proves an iterative update formula for the Integrated Variance when one datum is added, using the Schur complement on the covariance matrix, and packages this into a Monte Carlo estimator of Shapley values. The stated consequence is that the utility function becomes cheap enough that the practitioner can spend the computational budget on more aggregation samples, and that the valuations remain consistent with direct computation.
Load-bearing premise
The load-bearing premise is that a 'sub-model' is the same Gaussian Process with the same settings, just conditioned on fewer points, so no actual retraining happens when a subset is evaluated; the empty-set initialization in Algorithm 1 also relies on a definition the paper does not give.
Editorial extensions
If this is right
- If the approach holds, data valuations for GP-based models can be computed in settings where retraining on every coalition was previously infeasible.
- The utility/aggregation split gives practitioners a way to compare methods like Leave-One-Out, Data Shapley, Banzhaf and Beta Shapley as different choices in the same framework.
- Because the Schur update is iterative, adding or removing one point updates the valuation of all points without reinverting the covariance matrix.
- Data removal experiments suggest that removing points with low valuations degrades predictive performance less than random removal, especially when most of the dataset is discarded.
- The same machinery extends aggregation beyond a single datum to coalitions, since the update formula works for any added set, as the paper notes.
Reading between the lines
- A consequence the paper leaves implicit is that the valuations are conditional on the kernel settings fitted once on the full dataset; if a practitioner refits those settings on each subset, the Schur shortcut no longer applies.
- The same decomposition could be applied to other Bayesian models with closed-form predictive variances, such as Bayesian linear regression, as long as an analogous update formula exists.
- Kernel choice becomes a valuation-design parameter: the experiments show that adding noise to the kernel changes the range and ranking stability, which suggests practitioners could tune the kernel to emphasize global versus local influence.
- A testable extension would be to use the iterative Integrated Variance update with approximate or sparse GP methods, trading exactness for scalability beyond the few thousand points tried here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a decomposition of data valuation (DV) methods into a utility function and an aggregation procedure, and then argues that Gaussian Processes (GPs) with integrated variance as the utility can make Shapley-type data valuation computationally efficient via Schur-complement updates. The main algorithmic contribution is Algorithm 1, which estimates "Data Shapley" values by averaging marginal integrated-variance differences over random permutations of the training data. The paper validates the method on a synthetic sinusoid example, the Boston Housing dataset, and a Folktables-derived regression task.
Significance. If the central claim were correct, the paper would offer a useful practical bridge between GP modeling and data valuation, with the Schur-update machinery providing cheap access to sub-model utilities. The proposed decomposition is a reasonable expository lens for organizing DV methods, though it is a definition rather than a derived theorem. The paper also correctly identifies that Monte-Carlo permutation sampling can be combined with fast GP updates. However, the technical validity of the reported Shapley values is undermined by an incorrect aggregation formula and a flawed update rule, and the experimental evidence is internally inconsistent.
major comments (5)
- [§3.2, Eq. (9)] Equation (9) is not the Shapley value. The standard Shapley value is a weighted sum over subsets with coefficients |S|!(n-|S|-1)!/n!, or equivalently the average of marginal contributions over all permutations. Equation (9) instead uses a uniform weight of 1/n! for every subset, which gives equal weight to every coalition size and is not equivalent to any known Shapley or semi-value aggregation. For n=3, the Shapley weights are 1/3, 1/6, 1/6, 1/3, not 1/6, 1/6, 1/6, 1/6. The claim that Eq. (9) defines the Data Shapley indices is therefore mathematically incorrect, and all DSV results in the paper that rely on this formula are not Shapley values.
- [Algorithm 1] Algorithm 1 contains a second aggregation error. In the permutation loop, the marginal contribution IV_{π,j} - IV_{π,j-1} is divided by n! before being stored in Upd_j, and then Upd_j is averaged over random permutations. Averaging the marginal over permutations already estimates the Shapley value; the extra factor 1/n! scales the entire estimator by 1/n!. The correct update is Upd_j = IV_{π,j} - IV_{π,j-1}. As written, for the Boston Housing experiment with n=506, every reported valuation would be scaled by 1/506!, which cannot be the intended quantity. The paper does not provide any alternative interpretation of this factor, so the algorithm as stated does not compute the Shapley values it claims to compute.
- [Algorithm 1 and §2.2] Algorithm 1 initializes IV_{π,∅} := IV(∅), but IV(∅) is undefined in the setting the paper describes. Section 2.2 explicitly focuses on Universal Kriging, whose predictor (Eq. (2)) requires the matrix [K F; F^T 0] and the vector [k(x); f(x)]. With an empty training set, K is 0×0, F is p×0, and the kriging system has no well-defined solution, so the integrated variance in Eq. (5) is not defined. The paper does not define an empty-set predictor or explain how IV(∅) is computed. This is not a cosmetic issue: the first step of every permutation in Algorithm 1 depends on IV(∅).
- [§4.1 and §2.2] The paper never states the fixed-hyperparameter assumption that is necessary for the Schur-update formulas. The update in Eq. (11) of Appendix A is valid only if the covariance kernel and its hyperparameters (noise level, lengthscale, etc.) are the same for the full dataset and for every subset A. In standard data valuation, a "sub-model" trained on A would be obtained by refitting the model on A, which for a GP includes re-estimating hyperparameters. If hyperparameters are fitted on the full dataset and then fixed for all subsets, the object being evaluated is a conditional GP, not a retrained GP. This assumption must be stated explicitly and its effect on the meaning of the reported valuations must be discussed, since it changes the utility function from the one a practitioner would obtain by retraining on each subset.
- [§4.2 and Appendix B.2] The experimental claims are internally inconsistent. The main text states that "valuation-based methods consistently outperform random removal" and that the approach was "demonstrated ... on synthetic data as well as two others datasets," but Appendix B.2 explicitly reports that on the Folktables dataset "the data removal task ... did not conclusively show that removal based on value-based strategies had an advantage" (see Figure 7). This contradiction means the general claim of consistent outperformance over random removal is not supported by the evidence in the paper. In addition, Appendix B.1 states that the burn-in parameter was chosen to align the DSV computation with Leave-One-Out results; tuning a parameter to match a target baseline weakens the validation of the method as an independent measure of data value.
minor comments (5)
- [Section 1] There is a grammatical error in the sentence "we leverage the unique structure of GPs enables us to perform explicit and immediate computations of sub-models" (the sentence needs an auxiliary verb or a restructure such as "the unique structure of GPs enables").
- [Eq. (8)] The formula for the naive aggregation has mismatched parentheses: it reads "Φ[f (x; D] − Φ[f (x; D∼i]" instead of "Φ[f (x; D)] − Φ[f (x; D∼i)]".
- [Theorem 2.1 and Eq. (13)] Theorem 2.1 refers to "Equation 13 in Appendix A" for the explicit update formula, but Eq. (13) gives only the top-left element of the inverse of M_{A∪{i}}(x), not directly the integrated variance. The paper should spell out how this expression is integrated over x to obtain IV(A∪{i}), or at least clearly state the missing integration step.
- [Appendix B.1] The abbreviation "DSV" is used both for "Data Shapley Value" and for the truncated estimator with Schur complement; the text should explicitly distinguish "DSV computed without Schur" from "DSV computed with Schur" everywhere, as the experiments compare these two variants.
- [Figure 3] The caption "with a notable advantage as less of 20% is retained" is ambiguous; it should say "when less than 20% of the dataset is retained".
Circularity Check
Mild self-referentiality in the definitional decomposition and LOO-tuned burn-in; the GP update derivation itself is independent.
-
self definitional
[Definition 3.1, Eq. (6); Abstract]
"Then a general Data Valuation set of indices {DVi(Φ, Aggi, f,D)} is defined by its elements DVi(Φ, Aggi, f,D) := Aggi({Φ(f (x; A)), A⊂ D})."
The advertised 'novel canonical decomposition' is introduced by definition: any data valuation index is defined as an aggregation procedure applied to a utility function evaluated on sub-models. Therefore the claim that all DV methods can be analyzed as a utility function plus an aggregation procedure is true by construction and cannot fail. It is a formalization rather than a derived result, so presenting it as a finding is self-definitional, although it does not affect the subsequent GP computation.
-
fitted input called prediction
[Appendix B.1; Section 4.2]
"The Truncated DSV algorithm was configured to perform 1500 iterations for the Boston Housing dataset, with a burn-in period of 50 indices per iteration. The burn-in parameter allows the DSV computation to stabilize, aligning closer to Leave-One-Out (LOO) results. ... Our implementation yields the same rankings and LOO values no matter the approach considered, as confirmed by a perfect Spearman’s coefficient."
The reported agreement between DSV and LOO is used as a sanity check, but the burn-in parameter was explicitly chosen to make DSV align closer to LOO. Consequently, the perfect Spearman correlation between DSV and LOO is partially manufactured by the parameter choice rather than being an independent confirmation. This is a fitted-input-to-expected-output loop, though it is peripheral to the main Schur-update claim.
full rationale
The central computational derivation is not circular. The Integrated Variance update relies on the Schur complement, cited as an external algebraic result from Ginsbourger and Schärer, and the GP machinery is standard from Rasmussen and Williams; these do not assume the target valuations. The Shapley aggregation is also imported from the external cooperative-game literature, and the data-removal experiment against random removal provides an external, non-self-referential benchmark. The two mild circularities are the definitional character of the proposed decomposition and the burn-in tuned to LOO before LOO agreement is reported as sanity. Notably, the paper's Eq. (9) appears to omit subset-size weights and Algorithm 1 applies a 1/n! factor that biases the Monte Carlo estimator; this is an internal mathematical correctness issue, not circularity, so it does not increase the circularity score. Overall, the paper's load-bearing GP update argument remains independent, yielding a low circularity score.
Assumptions & free parameters
free parameters (3)
- kernel noise level =
0.01 and 0.38
- DSV burn-in =
50 indices per iteration
- covariance reset threshold =
not specified
assumptions (4)
- ad hoc to paper Every data valuation index can be written as an aggregation of a utility function over all subsets.
- domain assumption GP kernel hyperparameters remain fixed across all sub-models.
- domain assumption Integrated variance over the input distribution is computable.
- domain assumption IV(∅) is well-defined.
Cite this review
Pith. "Pith review of On the Usage of Gaussian Process for Efficient Data Valuation." pith.science (2026). https://pith.science/paper/UP3EEE6W
@misc{pith2026250604026,
author = {Pith},
title = {Pith review of: On the Usage of Gaussian Process for Efficient Data Valuation},
year = {2026},
howpublished = {\url{https://pith.science/paper/UP3EEE6W}},
note = {Machine review of arXiv:2506.04026}
}
read the original abstract
In machine learning, knowing the impact of a given datum on model training is a fundamental task referred to as Data Valuation. Building on previous works from the literature, we have designed a novel canonical decomposition allowing practitioners to analyze any data valuation method as the combination of two parts: a utility function that captures characteristics from a given model and an aggregation procedure that merges such information. We also propose to use Gaussian Processes as a means to easily access the utility function on ``sub-models'', which are models trained on a subset of the training set. The strength of our approach stems from both its theoretical grounding in Bayesian theory, and its practical reach, by enabling fast estimation of valuations thanks to efficient update formulae.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Basics and trends in sensitivity analysis: Theory and practice in R
S \'e bastien Da Veiga, Fabrice Gamboa, Bertrand Iooss, and Cl \'e mentine Prieur. Basics and trends in sensitivity analysis: Theory and practice in R. SIAM, ., 2021
work page 2021
-
[2]
Retiring adult: New datasets for fair machine learning
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021
2021
-
[3]
Jean H Gallier. Notes on the schur complement. University of Pennsylvania, 2010
work page 2010
-
[4]
Data shapley: Equitable valuation of data for machine learning
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2242--2251, ., 09--15 Jun 2019. PMLR. URL https://proceedings.mlr.press/v...
work page 2019
-
[5]
A distributional framework for data valuation
Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3535--3544, ., 13--18 Jul 2020. PMLR. URL https://proceedings.mlr.press/v119/ghorbani20a.html
work page 2020
-
[6]
David Ginsbourger and Cedric Schärer. Fast calculation of gaussian process multiple-fold cross-validation residuals and their covariances, 2023
work page 2023
-
[7]
Hedonic housing prices and the demand for clean air
David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5 0 (1): 0 81--102, March 1978. ISSN 0095-0696. doi:10.1016/0095-0696(78)90006-2. URL https://www.sciencedirect.com/science/article/pii/0095069678900062
arXiv 1978
- [8]
Show all 26 references
-
[9]
Efficient task-specific data valuation for nearest neighbor algorithms
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv:1908.08619, 2019 a
1908 arXiv
-
[10]
Towards efficient data valuation based on the shapley value
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve G \"u rel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics...
2019
-
[11]
Lava: Data valuation without pre-specified learning algorithms
Hoang Anh Just, Feiyang Kang, Jiachen T Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.00054, 2023
2023 arXiv
-
[12]
A statistical approach to some basic mine valuation problems on the witwatersrand
Daniel G Krige. A statistical approach to some basic mine valuation problems on the witwatersrand. Journal of the Southern African Institute of Mining and Metallurgy, 52 0 (6): 0 119--139, 1951
1951
-
[13]
Beta shapley: a unified and noise-reduced data valuation framework for machine learning
Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110.14049, 2021
2021 arXiv
-
[14]
An explicit link between gaussian fields and gaussian markov random fields: the stochastic partial differential equation approach
Finn Lindgren, Håvard Rue, and Johan Lindström. An explicit link between gaussian fields and gaussian markov random fields: the stochastic partial differential equation approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73 0 (4): 0 423--498...
2011
-
[15]
Calculations of Sobol indices for the Gaussian process metamodel
Amandine Marrel, Bertrand Iooss, B \'e atrice Laurent, and Olivier Roustant. Calculations of Sobol indices for the Gaussian process metamodel. Reliability Engineering & System Safety, 94 0 (3): 0 742--751, March 2009. ISSN 0951-8320. doi:10.1016/j.ress.2008.07.008
2009 doi
-
[16]
Principles of geostatistics
Georges Matheron. Principles of geostatistics. Economic geology, 58 0 (8): 0 1246--1266, 1963
1963
-
[17]
The bayesian bridge between simple and universal kriging
Henning Omre and Kjetil B Halvorsen. The bayesian bridge between simple and universal kriging. Mathematical Geology, 21: 0 767--786, 1989
1989
-
[18]
Rasmussen and C.K.I
C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning series. MIT Press, 2005. ISBN 9780262182539. URL https://books.google.ca/books?id=H3aMEAAAQBAJ
2005
-
[19]
L. S. Shapley. 17. A Value for n-Person Games, pages 307--318. Princeton University Press, Princeton, 1953. ISBN 9781400881970. URL https://doi.org/10.1515/9781400881970-018
1953 doi
-
[20]
Practical bayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. UR...
2012
-
[21]
Elementary mathematical theory of classification and prediction
Taffee T Tanimoto. Elementary mathematical theory of classification and prediction. Technical report, International Business Machines Corp., 1958
1958
-
[22]
Uncertainty Quantification and Experimental Design for Large - Scale Linear Inverse Problems under Gaussian Process Priors , August 2022
Cédric Travelletti, David Ginsbourger, and Niklas Linde. Uncertainty Quantification and Experimental Design for Large - Scale Linear Inverse Problems under Gaussian Process Priors , August 2022. URL http://arxiv.org/abs/2109.03457. arXiv:2109.03457 [cs, stat]
2022 arXiv
-
[23]
Tanimoto random features for scalable molecular machine learning
Austin Tripp, Sergio Bacallado, Sukriti Singh, and Jos \'e Miguel Hern \'a ndez-Lobato. Tanimoto random features for scalable molecular machine learning. In Thirty-seventh Conference on Neural Information Processing Systems, ., 2023. NeurIPS. URL https://openreview.net/forum?i...
2023
-
[24]
Exact gaussian processes on a million data points
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact gaussian processes on a million data points. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informat...
2019
-
[25]
Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity
Tianhao Wang and Ruoxi Jia. Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205.15466, 2022
2022 arXiv
-
[26]
The Schur complement and its applications, volume 4
Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006
2006
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.