Pith. sign in

REVIEW 3 major objections 5 minor 10 references

Uncertainty Quantification in Probabilistic Machine Learning Models: Theory, Methods, and Insights

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Predictive uncertainty in GPLVMs decomposes exactly into aleatoric and epistemic parts.

desk verdict The epistemic uncertainty estimator in Eq. (31) computes a second moment, not a variance, so the paper's central quantity is biased; the rest is a clean but standard GPLVM UQ exposition. read the letter →

arxiv 2509.05877 v2 pith:3RIPCQXV submitted 2025-09-07 stat.ML cs.AIcs.LG

Marzieh Ajirak , Anand Ravishankar , Petar M. Djuric This is my paper · ORCID
classification stat.MLcs.AIcs.LG
keywords uncertaintyquantificationepistemicaleatoricGaussianprocesslatentvariablemodelrandomFourierfeaturesMonteCarloestimationlawoftotalvariancepredictivedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that in a Gaussian process latent variable model equipped with random Fourier features, the total predictive variance of a missing output can be decomposed exactly into an aleatoric part, the expected irreducible noise variance, and an epistemic part that collects uncertainty in the training latent positions, the test latent position, and the Gaussian process weights. The decomposition follows from the law of total variance, applied twice, so that epistemic uncertainty is further split into a contribution from parameter covariance and a contribution from test-input variability. To make the decomposition computable, the paper proposes a Monte Carlo estimator that draws samples of the training latents and of the test latent position and evaluates the random-feature predictive mean and noise variance. If correct, the framework gives practitioners two actionable numbers for every imputed value: how much uncertainty is intrinsic to the data and how much could shrink with better inference or more data.

What carries the argument

The load-bearing device is the random Fourier feature map $\phi(x)$, which turns each Gaussian process output into a linear function $\phi(x)^\top \theta_d$. Because the conditional predictive mean is linear in the weights, the law of total variance can be applied at the level of $\phi(x_*)^\top E[\theta_d|X,Y]$, producing the expressions in Eqs. (20) through (24) that isolate contributions from posterior weight covariance, test-input variability, and training-latent variability. The same feature map makes the Monte Carlo estimator in Eq. (31) possible: samples $x_*^{(m,\ell)}$ plug into $\phi$, and the posterior weight moments $\mathrm{Cov}[\theta_d|X^{(m)},Y]$ come from the RFF-based Gaussian process fit.

What would settle it

In the synthetic linear output $y_1 = w_1^\top x + \epsilon$, compute the mean of $\phi(x_*)^\top E[\theta_1|X,Y]$ over the $L$ samples; if it is not close to zero, then Eq. (31)'s second term is a mean square rather than a variance, and the epistemic uncertainty reported for that output is overstated.

Watch

Extended reading notes

Core claim

The paper's central claim is that Eqs. (18), (24), (31), and (32) give a correct and systematic accounting of where predictive uncertainty comes from in an RFF-based GPLVM. Starting from the conditional model $y_{d,*} \sim N(\phi(x_*)^\top \theta_d, \sigma_d^2)$, the total variance $\mathrm{Var}[y_{d,*}]$ equals the expectation of $\sigma_d^2$ over the test-latent posterior, plus the variance of the conditional mean $\phi(x_*)^\top E[\theta_d|X,Y]$. The epistemic part is then broken down by the law of total variance into an expected term involving $\mathrm{Cov}[\theta_d|X,Y]$, a trace term tracking fluctuations of $\phi(x_*)$ around its mean, and a term tracking the variability of the conditional mean across training-latent posteriors. The proposed Monte Carlo scheme evaluates each of these terms using $M$ posterior draws of $X$ and $L$ draws of $x_*$ per $X$ draw. On the paper's own terms, this is a principled route to separate irreducible data noise from model uncertainty in a scalable Gaussian process approximation.

Load-bearing premise

The central estimator's second term is only a variance if the expected prediction, averaged over all plausible latent positions of the test point, is zero; the paper neither states nor enforces that condition.

Editorial extensions

If this is right

  • Each predicted missing output comes with a split: aleatoric uncertainty tells the floor set by data noise, and epistemic uncertainty tells how much remains available to shrink.
  • A practitioner can attribute epistemic uncertainty to training latents, test position, or GP weights and decide whether more data, better latent inference, or more random features is the right fix.
  • The RFF approximation makes the whole computation scalable, with cost controlled by the number of frequencies $J$ and the Monte Carlo sample sizes $M$ and $L$, so the method applies to larger datasets than exact GP GPLVMs.
  • Smooth outputs such as the periodic function keep epistemic uncertainty low, while a step-function output shows high epistemic uncertainty, reflecting the GP's known difficulty with discontinuities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A diagnostic the paper does not report: comparing the sample second moment in Eq. (31) with a properly centered sample variance would quantify how much the zero-mean assumption on $\phi(x_*)^\top E[\theta_d|X,Y]$ matters, and publishing that gap would let readers judge the estimator directly.
  • Because the decomposition only uses linearity of the conditional mean in $\theta_d$, it transfers directly to Bayesian linear regression, Bayesian neural network last layers, or other feature-based linear models, where the same total-variance identities apply.
  • A follow-up experiment could split epistemic uncertainty by source for each of the four synthetic outputs and check whether the weight-covariance term correlates with $J$, the input-variability term with the smoothness of the function, and the training-latent term with $M$.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework for decomposing predictive uncertainty in Gaussian process latent variable models (GPLVMs) with Random Fourier Feature (RFF) approximations into aleatoric and epistemic components. It derives an expression for the epistemic variance using the law of total variance, introduces Monte Carlo estimators for both components, and reports synthetic experiments with four output types (linear, quadratic, periodic, step) to illustrate how the estimates behave as the number of RFF features varies. The central claim is that Eqs. (18), (24), (31), and (32) provide a correct decomposition of total predictive variance and that the proposed Monte Carlo estimates are valid.

Significance. If the derivation and estimator were correct, the paper would offer a useful practical recipe for uncertainty decomposition in a scalable GPLVM setting, and the synthetic study would provide some insight into how uncertainty estimates behave across output types. The authors also deserve credit for explicitly acknowledging that Monte Carlo variation is not accounted for, which is a common omission. However, the significance is undercut by a load-bearing error in the Monte Carlo estimator: as written, Eq. (31) estimates a second moment rather than a variance, so the reported epistemic uncertainties are biased. The associated theoretical derivation in Eqs. (23)--(24) also contains a notational/identity problem that must be repaired. Because these issues affect the paper's central claim, the current version cannot be accepted as a reliable contribution to UQ methodology.

major comments (3)
  1. [§5, Eq. (31); §4, Eq. (24)] Eq. (31) does not estimate the variance appearing in Eq. (24). The final term of (31) is, up to averaging constants, (1/L) Σ_ℓ (φ(x*^{(ℓ)})^T E[θ_d | X, Y])^2, which is the sample second moment of a = φ(x*)^T E[θ_d | X, Y]. The corresponding expression in Eq. (24) is Var[a] = E[a^2] − (E[a])^2, and the centering term is never subtracted. The paper neither states nor enforces the zero-mean condition E[a] = 0; indeed, in the synthetic linear output y1 = w1^T x + ε, the posterior predictive mean is generally nonzero over the test-latent distribution, so the omitted term is non-negligible. This is not a cosmetic issue: the reported epistemic uncertainties in Fig. 2 are biased upward by the squared mean of the predictive function, which invalidates the central claim that Eqs. (31)--(32) provide correct epistemic and aleatoric uncertainty estimates.
  2. [§4, Eqs. (23)--(24)] Eq. (23) as printed is an identity only under an interpretation that conflicts with the notation used elsewhere. The left side, Var_{p(x*|...)}(φ(x*)^T E[θ_d|X,Y]), is identical to the second term on the right side, so the equation is circular as written. The intended identity presumably involves the joint variance Var_{x*,θ}(φ^T θ), in which case the correct decomposition is Var_{x*,θ}(φ^T θ) = E_{x*}[φ^T Cov[θ|X,Y] φ] + Var_{x*}(φ^T E[θ|X,Y]). The text does not state this interpretation, and Eq. (24) therefore does not follow from the equations as written. This derivation issue must be fixed for the theoretical formulation to be sound.
  3. [§6, Figs. 1--2] The experimental section does not validate the uncertainty estimates. The paper reports boxplots of estimated aleatoric and epistemic variances but provides no calibration metric (e.g., coverage of predictive intervals), no comparison against an exact GP or another baseline, and no numerical comparison with the known generative noise variance σ²_epsilon = 1, even though such a comparison is possible for the linear output. The statement that the aleatoric uncertainty for y1 was 'estimated accurately' is therefore not supported by the presented evidence. This matters because the estimator bias identified in Eq. (31) directly affects the plotted epistemic values.
minor comments (5)
  1. [§3, Eq. (10)] The prior notation p(Σ) should be p(Σ_y) for consistency with the earlier definition of Σ_y.
  2. [§3, after Eq. (16)] The text contains an unresolved placeholder reference 'Fig. ??', which should be fixed before submission.
  3. [§3, Eq. (17)] The quantities bθ_d and bσ²_o,*,d are described only as 'estimated values' without specifying whether they are MAP estimates, posterior means, or derived from a particular Monte Carlo sample; this should be clarified.
  4. [§4, Eqs. (20)--(21)] The prose around Eq. (21) contains a typographical repetition ('variations in x* variations') and missing punctuation; these should be cleaned up.
  5. [References] The reference list is not consistently formatted (e.g., volume/page information is incomplete for several entries); this should be standardized.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the UQ decomposition is a direct law-of-total-variance identity; the Monte Carlo estimators evaluate posterior quantities rather than fitting the target uncertainties, and no load-bearing self-citation is present.

full rationale

The paper's central UQ decomposition is an algebraic application of the law of total variance. Eq. (18) is Var[y] = E[Var(y|x*,theta)] + Var(E[y|x*,theta]); this is an identity, but not a circularity in the prohibited sense because the aleatoric and epistemic terms are not defined in terms of the final numerical estimates, and neither term is fitted to the target uncertainties. The epistemic expansion in Eqs. (20)-(24) continues to condition on posterior quantities Cov[theta_d|X,Y] and p(x*|...), which come from the GPLVM likelihood and priors, not from the uncertainty values being reported. The Monte Carlo estimators (31)-(32) are posterior expectations of already-derived expressions; sigma_d^2 samples are drawn from the noise posterior, not tuned to match the plotted uncertainties. References to RFF (Rahimi and Recht), GPLVM (Titsias and Lawrence), and GP (Rasmussen and Williams) are external, and no load-bearing claim depends on a self-citation. The main concern suggested by close reading, the possible confusion of a sample second moment with a variance in Eq. (31), would be a mathematical correctness or derivation error, not a circular structure, because the omitted centering term would not be an input to the method. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central framework rests on standard probability identities such as the law of total variance and Bochner's theorem, on modeling choices about the GPLVM and RFF approximation, and on one unstated zero-mean assumption in the Monte Carlo estimator. No new physical or statistical entities are introduced.

free parameters (4)
  • RBF kernel hyperparameters (lengthscale and variance) = not reported
    Random frequencies ω_j are sampled from the power-spectral density of the RBF kernel in Section 5, so the kernel hyperparameters must be chosen or estimated; they control the GP prior and can dominate the estimated predictive variances.
  • Number of RFF features J = 10, 25, 50, 100, 200, 500
    J is varied over discrete values in the experiments, but no criterion selects a working value, and the finite-J approximation error is not compared with the exact GP.
  • Monte Carlo sample counts M and L = M=100, L=100
    The estimators in Eqs. (31) and (32) depend on M and L, and the paper states in Section 7 that the resulting Monte Carlo uncertainty is not accounted for.
  • Variational posterior parameters for X = not specified
    Samples X^(m) in Eq. (31) are drawn "using the variational distribution of X" in Section 6, but the variational family, optimization objective, and convergence checks are not given.
assumptions (5)
  • domain assumption The predictive posterior for each output dimension factorizes as a Gaussian across dimensions (Eq. 13).
    Used to write the predictive distribution of y_u,* as a product of univariate Gaussians, discarding cross-output correlations that could carry uncertainty information.
  • domain assumption The observed data are generated from Gaussian GPs with diagonal noise Σ_y (Eq. 8), so the conditional predictive variance of y_{d,*} given x_* and θ_d is exactly σ_d^2.
    Underpins the aleatoric term in Eq. (18) and its estimator Eq. (32).
  • standard math Bochner's theorem guarantees the RFF approximation κ(x,x') ≈ φ(x)^T φ(x') for the chosen kernel.
    Used in Eqs. (14)-(15) to replace GP kernels with finite random feature maps.
  • domain assumption The variational posterior for X is a faithful approximation of p(X|Y), so M=100 samples from it are representative.
    All epistemic estimates in Eq. (31) are averages over samples X^(m) drawn from this variational distribution; no quality or convergence check is given.
  • ad hoc to paper Var[g(x*)] = E[g(x*)^2] in Eq. (31), i.e., the predictive mean g has zero mean over p(x*|...).
    The second term of Eq. (31) estimates the variance of φ(x*)^T E[θ_d|X,Y] by the sample second moment without subtracting the squared sample mean; this holds only under an unstated zero-mean assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty Quantification in Probabilistic Machine Learning Models: Theory, Methods, and Insights." pith.science (2026). https://pith.science/paper/3RIPCQXV

@misc{pith2026250905877,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Quantification in Probabilistic Machine Learning Models: Theory, Methods, and Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RIPCQXV}},
  note         = {Machine review of arXiv:2509.05877}
}
read the original abstract

Uncertainty Quantification (UQ) is essential in probabilistic machine learning models, particularly for assessing the reliability of predictions. In this paper, we present a systematic framework for estimating both epistemic and aleatoric uncertainty in probabilistic models. We focus on Gaussian Process Latent Variable Models and employ scalable Random Fourier Features-based Gaussian Processes to approximate predictive distributions efficiently. We derive a theoretical formulation for UQ, propose a Monte Carlo sampling-based estimation method, and conduct experiments to evaluate the impact of uncertainty estimation. Our results provide insights into the sources of predictive uncertainty and illustrate the effectiveness of our approach in quantifying the confidence in the predictions.

Figures

Figures reproduced from arXiv: 2509.05877 by the authors.

Figure 1
Figure 1. The estimated aleatoric uncertainties in each of the four predicted outputs of [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. The estimated epistemic uncertainties in each of the four predicted outputs of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages

  1. [1]

    Abdar, M., Khosravi, A., Islam, S. M. S., Acharya, U. R., and Vasilakos, A. V. (2022). The need for quantification of uncertainty in artificial intelligence for clinical data analysis: increasing the level of trust in the decision-making process. IEEE Systems, Man, and Cybernetics Magazine , 8(3):28--40

    work page 2022

  2. [2]

    Cheng, S., Quilodr \'a n-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., et al. (2023). Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review. IEEE/CAA Journal of Automatica Sinica , 10(6):1361--1387

    work page 2023

  3. [3]

    Ghanem, R., Higdon, D., Owhadi, H., et al. (2017). Handbook of Uncertainty Quantification , volume 6. Springer New York

    work page 2017

  4. [4]

    Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective . MIT press

    work page 2012

  5. [5]

    M., et al

    Papamarkou, T., Skoularidou, M., Palla, K., Aitchison, L., Arbel, J., Dunson, D., Filippone, M., Fortuin, V., Hennig, P., Hern \'a ndez-Lobato, J. M., et al. (2024). Position: B ayesian deep learning is needed in the age of large-scale AI . In Forty-first International Conference on Machine Learning

    work page 2024

  6. [6]

    and Recht, B

    Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in Neural Information Processing Systems , 20

    work page 2007

  7. [7]

    Rasmussen, C. E. and Williams, C. K. (2006). Gaussian Processes for Machine Learning , volume 2. MIT Press

    work page 2006

  8. [8]

    Rudin, W. (2017). Fourier Analysis on Groups . Courier Dover Publications

    work page 2017

Show all 10 references
  1. [9]

    and Lawrence, N

    Titsias, M. and Lawrence, N. D. (2010). Bayesian G aussian process latent variable model. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 844--851. JMLR Workshop and Conference Proceedings

    2010

  2. [10]

    Wang, T., Wang, Y., Zhou, J., Peng, B., Song, X., Zhang, C., Sun, X., Niu, Q., Liu, J., Chen, S., et al. (2025). From aleatoric to epistemic: E xploring uncertainty quantification techniques in artificial intelligence. arXiv preprint arXiv:2501.03282

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.