pith. sign in

arxiv: 2507.20941 · v4 · submitted 2025-07-28 · 📊 stat.ML · cs.AI· cs.LG· stat.ME· stat.OT

Multivariate Standardized Residuals for Conformal Prediction

Sacha Braun , Eug\`ene Berta , Michael I. Jordan , Francis Bach This is my paper

Pith reviewed 2026-05-19 03:24 UTC · model grok-4.3

classification 📊 stat.ML cs.AIcs.LGstat.MEstat.OT
keywords conformal predictionmultivariate regressionconditional coveragestandardized residualsMahalanobis distanceheteroskedasticityuncertainty quantificationnon-conformity score
0
0 comments X

The pith

Standardized residuals via local covariance achieve asymptotic conditional coverage for multivariate conformal prediction.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper extends univariate residual normalization to the multivariate setting by whitening residuals with a learned local covariance to account for output correlations and heteroskedasticity. It derives a sufficient condition on the data distribution under which this yields asymptotic conditional coverage in conformal prediction. The resulting non-conformity score is the Mahalanobis distance, which is closed-form and avoids sampling from estimated CDFs. This enables efficient construction of conformal sets while improving conditional coverage on heteroskedastic data. A sympathetic reader would care because marginal coverage alone is often insufficient for reliable decision-making in multi-output regression tasks.

Core claim

We propose a natural extension of normalizing non-conformity scores to the multivariate setting by whitening the residuals to decouple output correlations and standardize local variance. We derive a sufficient condition characterizing a broad class of distributions for which standardized residuals yield asymptotic conditional coverage. Using the Mahalanobis distance induced by a learned local covariance as a non-conformity score provides a closed-form, computationally efficient mechanism for capturing inter-output correlations and heteroskedasticity.

What carries the argument

Mahalanobis distance induced by a learned local covariance matrix, which whitens residuals to standardize variance and remove inter-output correlations.

If this is right

  • Conformal sets can be constructed in closed form without sampling from cumulative distribution functions.
  • Valid conformal sets extend to transformations of the multivariate output.
  • Conformal sets can be refined when partial output information is revealed at test time.
  • Missing output values can be handled directly within the conformal prediction procedure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The whitening approach could be combined with modern covariance estimators to scale to higher-dimensional outputs.
  • Similar residual standardization might improve conditional properties in non-conformal uncertainty methods such as quantile regression.
  • Empirical checks of the sufficient condition on new datasets could help practitioners decide when the method is reliable.

Load-bearing premise

The data distribution belongs to the broad class satisfying the sufficient condition for standardized residuals to achieve asymptotic conditional coverage.

What would settle it

Measure the empirical conditional coverage rate on held-out data drawn from a distribution known to violate the sufficient condition and verify whether it drops below the target level.

Figures

Figures reproduced from arXiv: 2507.20941 by Eug\`ene Berta, Francis Bach, Michael I. Jordan, Sacha Braun.

Figure 1
Figure 1. Figure 1: Illustrating a typical failure case of the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustrating the generalized CDF score 𝑆HDP. With this score, a portion of the total mass of the estimated distribution 𝑝^(·|𝑋) is conformalized to satisfy the required 1 − 𝛼 coverage level on the calibration set. The size of the conformal sets constructed cannot be null, as it naturally follows the variance of the estimated conditional distribution, making this score robust to the failure case exposed for… view at source ↗
Figure 3
Figure 3. Figure 3: Conformal sets obtained with different score functions: [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Conformal sets obtained for partially revealed outputs via the method described in Section [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparing conformal sets obtained for projected outputs [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Histograms of conditional coverage P(𝑌 ∈ 𝐶𝛼(𝑋𝑖)|𝑋𝑖) for different test samples 𝑋𝑖 , using different conformal prediction methods with required coverage 1 − 𝛼 = 0.90 (vertical red dashed line). Aggregated results for 100 test points per dataset on ten randomly generated synthetic datasets. Het￾eroskedastic gaussian noise 𝐵 = 𝒩 (0, 𝐼𝑘) (left) and heteroskedastic exponential noise 𝐵 = ℰ(1) (right). 6.3 Real d… view at source ↗
Figure 7
Figure 7. Figure 7: Failure mode of the optimal transport strategy. Dots on the left represent samples from a [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
read the original abstract

While split conformal prediction guarantees marginal coverage, approaching the stronger property of conditional coverage is essential for reliable uncertainty quantification. Naive conformal scores, however, suffer from poor conditional coverage in heteroskedastic settings. In univariate regression, this is commonly addressed by normalizing non-conformity scores using an estimated local score variance. In this work, we propose a natural extension of this normalization to the multivariate setting, effectively whitening the residuals to decouple output correlations and standardize local variance. Furthermore, we derive a sufficient condition characterizing a broad class of distributions for which standardized residuals yield asymptotic conditional coverage. We demonstrate that using the Mahalanobis distance induced by a learned local covariance as a non-conformity score provides a closed-form, computationally efficient mechanism for capturing inter-output correlations and heteroskedasticity, avoiding the expensive sampling required by previous methods based on cumulative distribution functions. This structure unlocks several practical extensions, including the handling of missing output values, the refinement of conformal sets when partial information is revealed, and the construction of valid conformal sets for transformations of the output. Finally, we provide extensive empirical evidence on both synthetic and real-world datasets showing that our approach yields conformal sets that improve upon the conditional coverage of existing multivariate baselines.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes using the Mahalanobis distance induced by a learned local covariance as a non-conformity score to standardize residuals in multivariate conformal prediction. It derives a sufficient condition under which this yields asymptotic conditional coverage for a broad class of distributions, provides a closed-form efficient alternative to sampling-based methods for capturing correlations and heteroskedasticity, and reports empirical improvements on synthetic and real-world datasets along with extensions for missing outputs and transformations.

Significance. If the sufficient condition holds with appropriate estimator convergence, the approach extends univariate residual normalization to the multivariate setting in a computationally efficient manner, avoiding expensive CDF sampling while improving conditional coverage. The empirical evidence and practical extensions (missing values, partial information, output transformations) strengthen its potential impact for reliable uncertainty quantification in multi-output regression.

major comments (2)
  1. [sufficient condition derivation] Derivation of the sufficient condition: the claim of asymptotic conditional coverage for the broad class requires the local covariance estimator to converge to the true conditional covariance at a rate that preserves quantile behavior, but no explicit convergence rate or separation from the calibration set is stated; without this the whitening step risks introducing dependence that breaks the asymptotic argument.
  2. [empirical evaluation] Empirical results section: reported gains in conditional coverage are presented without details on data exclusion rules, error-bar computation, or whether the local covariance estimator was trained on a separate split from calibration data; this leaves the support for the central claim only partially verifiable.
minor comments (2)
  1. [notation] Clarify in the notation section whether the learned local covariance is denoted distinctly from the population conditional covariance to prevent reader confusion.
  2. [method] Add a short discussion of computational complexity for the local covariance estimation step in high-dimensional output regimes.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. The comments highlight important aspects of the theoretical derivation and empirical presentation that we address point by point below. We propose targeted revisions to strengthen clarity and verifiability while preserving the core contributions.

read point-by-point responses

  1. Referee: [sufficient condition derivation] Derivation of the sufficient condition: the claim of asymptotic conditional coverage for the broad class requires the local covariance estimator to converge to the true conditional covariance at a rate that preserves quantile behavior, but no explicit convergence rate or separation from the calibration set is stated; without this the whitening step risks introducing dependence that breaks the asymptotic argument.

    Authors: We appreciate the referee's careful reading of the asymptotic argument. The sufficient condition (Theorem 3.1) is formulated for a broad class of distributions under which the Mahalanobis-based score yields asymptotic conditional coverage provided the local covariance estimator converges in probability to the true conditional covariance at a rate that leaves the limiting quantile unaffected. The manuscript already specifies that the estimator is fit on a training split held out from the calibration set, which ensures the required independence. We will add an explicit remark in the revised version stating that the estimator must satisfy ||hat{Sigma}(x) - Sigma(x)|| = o_p(1) in an appropriate matrix norm to preserve quantile convergence, thereby addressing the dependence concern directly. revision: yes

  2. Referee: [empirical evaluation] Empirical results section: reported gains in conditional coverage are presented without details on data exclusion rules, error-bar computation, or whether the local covariance estimator was trained on a separate split from calibration data; this leaves the support for the central claim only partially verifiable.

    Authors: We agree that additional experimental details will improve verifiability. The local covariance estimator was trained on a dedicated training split disjoint from both the calibration and test sets, as described in the experimental protocol. In the revision we will explicitly document this split, state that no data points were excluded beyond standard preprocessing for missing outputs, and clarify that error bars denote standard errors computed across 10 independent random seeds. These additions will make the reported improvements in conditional coverage fully reproducible and directly supportive of the central claims. revision: yes

Circularity Check

0 steps flagged

Derivation of sufficient condition remains independent of fitted covariance

full rationale

The paper states that it derives a sufficient condition under which standardized residuals achieve asymptotic conditional coverage for a broad class of distributions. The Mahalanobis non-conformity score is constructed from a learned local covariance, yet the coverage claim is presented as following from the derived distributional condition rather than reducing by construction to any fitted parameter or self-citation chain. No equations in the abstract equate the prediction directly to the input fit, and the theoretical step is separated from the empirical validation on synthetic and real data. The derivation is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on accurate estimation of a local covariance matrix from data and on the data distribution satisfying the sufficient condition for asymptotic conditional coverage.

free parameters (1)
  • local covariance estimator
    The covariance matrix is estimated locally from data to standardize residuals and capture correlations; this is a data-dependent parameter central to the non-conformity score.
axioms (1)
  • domain assumption The data distribution belongs to the broad class for which the derived sufficient condition guarantees asymptotic conditional coverage.
    Invoked to establish the theoretical guarantee for standardized residuals.

pith-pipeline@v0.9.0 · 5754 in / 1355 out tokens · 61687 ms · 2026-05-19T03:24:51.190291+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Super-Level-Set Regression: Conditional Quantiles via Volume Minimization

    stat.ML 2026-05 unverdicted novelty 7.0

    Super-level-set regression directly optimizes conditional level-set boundaries via volume minimization to achieve minimum-volume prediction regions with conditional coverage.

  2. A Kernel Nonconformity Score for Multivariate Conformal Prediction

    stat.ML 2026-04 unverdicted novelty 5.0

    A new kernel nonconformity score for multivariate conformal prediction that adapts to residual geometry, provides finite-sample coverage, and achieves convergence rates based on effective kernel rank rather than ambie...

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION article output.bibitem format.authors "author" output.check author format.key output output.year.check new.block format.title "title" output.check new.block crossref missing format.jour.vol output format.article.crossref output.nonnull format.pages output if new.block note output fin.entry FUNCTION b...

    work page

  2. [2]

    Charnock, S

    Alsing, J., T. Charnock, S. Feeney, and B. Wandelt (2019). Fast likelihood-free cosmology with neural density estimators and active learning. Monthly Notices of the Royal Astronomical Society\/ 488\/ (3), 4440--4458

    work page 2019

  3. [3]

    Angelopoulos, A. N. and S. Bates (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning\/ 16\/ (4), 494--591

    work page 2023

  4. [4]

    Aolaritei, M

    Braun, S., L. Aolaritei, M. I. Jordan, and F. Bach (2025). Minimum volume conformal sets for multivariate regression. arXiv preprint arXiv:2503.19068\/

    work page arXiv 2025

  5. [5]

    David, F. and N. Johnson (1948). The probability integral transformation when parameters are estimated from the sample. Biometrika\/ 35\/ (1/2), 182--190

    work page 1948

  6. [6]

    Fontana, Y

    Dheur, V., M. Fontana, Y. Estievenart, N. Desobry, and S. B. Taieb (2025). Multi-output conformal regression: A unified comparative study with new conformity scores. International Conference on Machine Learning\/

    work page 2025

  7. [7]

    English, E. and C. Lippert (2025). Japan: Joint adaptive prediction areas with normalising-flows. arXiv preprint arXiv:2505.23196\/

    work page arXiv 2025

  8. [8]

    Bates, and Y

    Feldman, S., S. Bates, and Y. Romano (2023). Calibrated multiple-output quantile regression with representation learning. Journal of Machine Learning Research\/ 24\/ (24), 1--48

    work page 2023

  9. [9]

    Conformal prediction with corrupted labels: Uncertain imputation and robust re-weighting,

    Feldman, S., S. Bates, and Y. Romano (2025). Conformal prediction with corrupted labels: Uncertain imputation and robust re-weighting. arXiv preprint arXiv:2505.04733\/

    work page arXiv 2025

  10. [10]

    Foygel Barber, R., E. J. Candes, A. Ramdas, and R. J. Tibshirani (2021). The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA\/ 10\/ (2), 455--482

    work page 2021

  11. [11]

    Shimizu, and R

    Izbicki, R., G. Shimizu, and R. B. Stern (2022). CD -split and HPD -split: Efficient conformal regions in high dimensions. Journal of Machine Learning Research\/ 23\/ (87), 1--32

    work page 2022

  12. [12]

    Johnstone, C. and B. Cox (2021). Conformal uncertainty sets for robust optimization. In Conformal and Probabilistic Prediction and Applications , Volume 152, pp.\ 72--90

    work page 2021

  13. [13]

    2025 , archivePrefix =

    Klein, M., L. Bethune, E. Ndiaye, and M. Cuturi (2025). Multivariate conformal prediction using optimal transport. arXiv preprint arXiv:2502.03609\/

    work page arXiv 2025

  14. [14]

    Liu, and G

    Kong, J., Y. Liu, and G. Yang (2025). Fair conformal prediction for incomplete covariate data. arXiv preprint arXiv:2504.12582\/

    work page arXiv 2025

  15. [15]

    Josse, E

    Le Morvan, M., J. Josse, E. Scornet, and G. Varoquaux (2021). What’s a good imputation to predict with missing values? In Advances in Neural Information Processing Systems , Volume 34, pp.\ 11530--11540. Curran Associates, Inc

    work page 2021

  16. [16]

    G’Sell, A

    Lei, J., M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association\/ 113\/ (523), 1094--1111

    work page 2018

  17. [17]

    Lei, J. and L. Wasserman (2014). Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology\/ 76\/ (1), 71--96

    work page 2014

  18. [18]

    Destercke, and S

    Messoudi, S., S. Destercke, and S. Rousseau (2022). Ellipsoidal conformal inference for multi-target regression. In Conformal and Probabilistic Prediction with Applications , Volume 179, pp.\ 294--306

    work page 2022

  19. [19]

    Neeven, J. and E. Smirnov (2018, 11--13 Jun). Conformal stacked weather forecasting. In Conformal and Probabilistic Prediction and Applications , Volume 91, pp.\ 220--233

    work page 2018

  20. [20]

    Proedrou, V

    Papadopoulos, H., K. Proedrou, V. Vovk, and A. Gammerman (2002). Inductive confidence machines for regression. In European conference on machine learning , pp.\ 345--356. Springer

    work page 2002

  21. [21]

    Varoquaux, A

    Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al. (2011). Scikit-learn: Machine learning in P ython. Journal of Machine Learning Research\/ 12 , 2825--2830

    work page 2011

  22. [22]

    Fishkov, M

    Plassier, V., A. Fishkov, M. Guizani, M. Panov, and E. Moulines (2025). Probabilistic conformal prediction with approximate conditional validity. The Thirteenth International Conference on Learning Representations\/

    work page 2025

  23. [23]

    Stoltz, Y

    Principato, G., G. Stoltz, Y. Amara-Ouali, Y. Goude, B. Hamrouche, and J.-M. Poggi (2024). Conformal prediction for hierarchical data. arXiv preprint arXiv:2411.13479\/

    work page internal anchor Pith review arXiv 2024

  24. [24]

    Robert, C. P. and G. Casella (1999). Monte Carlo Statistical Methods . Springer

    work page 1999

  25. [25]

    Patterson, and E

    Romano, Y., E. Patterson, and E. Candes (2019). Conformalized quantile regression. Advances in Neural Information Processing systems\/ 32

    work page 2019

  26. [26]

    Roth, M. (2012). On the multivariate t distribution . Link \"o ping University Electronic Press

    work page 2012

  27. [27]

    Lei, and L

    Sadinle, M., J. Lei, and L. Wasserman (2019). Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association\/ 114\/ (525), 223--234

    work page 2019

  28. [28]

    and K.-S

    Sampson, M. and K.-S. Chan (2024). Flexible conformal highest predictive conditional density sets. arXiv preprint arXiv:2406.18052\/

    work page arXiv 2024

  29. [29]

    Shafer, G. and V. Vovk (2008). A tutorial on conformal prediction. Journal of Machine Learning Research\/ 9\/ (3), 371--421

    work page 2008

  30. [30]

    Nadjahi, and C

    Thurin, G., K. Nadjahi, and C. Boyer (2025). Optimal transport-based conformal prediction. International Conference on Machine Learning\/

    work page 2025

  31. [31]

    Cleaveland, R

    Tumu, R., M. Cleaveland, R. Mangharam, G. Pappas, and L. Lindemann (2024). Multi-modal conformal prediction regions by optimizing convex shape templates. In 6th Annual Learning for Dynamics & Control Conference , pp.\ 1343--1356. PMLR

    work page 2024

  32. [32]

    Vovk, V. (2012). Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning , pp.\ 475--490. PMLR

    work page 2012

  33. [33]

    Gammerman, and G

    Vovk, V., A. Gammerman, and G. Shafer (2005). Algorithmic Learning in a Random World . Springer

    work page 2005

  34. [34]

    Wang, Z., R. Gao, M. Yin, M. Zhou, and D. Blei (2023). Probabilistic conformal prediction using conditional random samples. In International Conference on Artificial Intelligence and Statistics , Volume 206, pp.\ 8814--8836. PMLR

    work page 2023

  35. [35]

    Wieslander, H., P. J. Harrison, G. Skogberg, S. Jackson, M. Frid \'e n, J. Karlsson, O. Spjuth, and C. W \"a hlby (2020). Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images. IEEE Journal of Biomedical and Health Informatics\/ 25\/ (2), 371--380

    work page 2020

  36. [36]

    Dieuleveut, J

    Zaffran, M., A. Dieuleveut, J. Josse, and Y. Romano (2023). Conformal prediction with missing values. In International Conference on Machine Learning , Volume 202, pp.\ 40578--40604. PMLR

    work page 2023

  37. [37]

    Josse, Y

    Zaffran, M., J. Josse, Y. Romano, and A. Dieuleveut (2024). Predictive uncertainty quantification with missing covariates. arXiv preprint arXiv:2405.15641\/

    work page arXiv 2024

  38. [38]

    Zhou, X., B. Chen, Y. Gui, and L. Cheng (2025). Conformal prediction: A data perspective. ACM Computing Surveys\/

    work page 2025