REVIEW 3 major objections 5 minor 1 cited by
Multivariate Conformal Prediction using Optimal Transport
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read By mapping vector-valued conformity scores through an optimal transport map to a uniform ball and calibrating a radius on the resulting norms, this paper constructs conformal prediction sets for multivariate outputs with a…
desk verdict A promising OT-based multivariate conformal method whose headline coverage guarantee does not cover the implementation actually used in the experiments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the entropic Brenier map estimator $T_\varepsilon(z) = z - \nabla f_\varepsilon(z)$, built from Sinkhorn dual potentials, which gives a tractable, out-of-sample evaluable map from residual space to a stratified spherical uniform grid. The norm of its image is the scalar conformity score, and the discrete spherical uniform target—$n_S$ directions times $n_R$ radii plus $n_o$ origin copies—yields the exact radius formula in Proposition 3.2. The mechanism that carries the argument is the empirical quantile of transported norms, which absorbs any approximation error in the map into the calibration radius.
What would settle it
Take an i.i.d. multivariate regression dataset, set $\alpha=0.1$, fit the entropic map on the calibration residuals, compute the radius as the empirical quantile of the mapped calibration norms, score a fresh test point with the same fitted map, and repeat over many test points: if coverage ever falls systematically below 90 percent, the guarantee as stated is false for that protocol. A sharper test is the transductive version: refit the map on all $n+1$ points in each trial using an order-dependent optimization and then check marginal coverage, since exchangeability of the transformed scores is the only assumption the proof needs.
Extended reading notes
Core claim
The central claim is that multivariate conformal prediction reduces to a univariate conformal problem after a change of coordinates learned by optimal transport. Given vector-valued residuals $S(X_i,Y_i)$, the method learns a map $\hat T$ pushing their empirical distribution toward a uniform measure on the unit ball, defines the score of any candidate $y$ as $\|\hat T(S(X_{n+1}, y))\|$, and sets the prediction region as the preimage of a ball in transported space whose radius is the empirical $(1-\alpha)$-quantile of the observed transported norms. Proposition 3.6 asserts that this set contains $Y_{n+1}$ with probability at least $1-\alpha$ for any finite sample and any map $\hat T$, as long as the sequence of transformed scores is exchangeable. Because the radius is calibrated on the actual distribution of $\|\hat T(S)\|$, approximation error in the transport map affects only the size and shape of the region, not the coverage guarantee.
Load-bearing premise
The load-bearing premise is that the transformed scores $\|\hat T(S_i)\|$ for calibration and for the new point are exchangeable with one another, which requires the transport map to be fixed before calibration or to be a symmetric function of the whole exchangeable sequence; if the map is refit using the same data that determines the radius in a way that breaks that symmetry, the coverage guarantee can fail.
Editorial extensions
If this is right
- Calibration on the empirical quantile of transported norms absorbs transport-map approximation error, so coverage stays valid even with entropic regularization and finite target grids.
- OT-CP regions are preimages of balls in transported space, so they can be non-elliptic and shaped by the residual distribution, unlike ellipsoidal Merge-CP sets.
- When source and target distributions are Gaussian or uniform, the transport map is affine and OT-CP's score coincides with the Mahalanobis distance, making the method a strict generalization of Merge-CP with a Mahalanobis norm.
- On the 24-task benchmark, OT-CP reports smaller mean region sizes than the M-CP and Merge-CP baselines on most low-dimensional datasets, with the advantage shrinking for outputs of dimension 14 to 16.
- The computational cost is $O(nm)$ to fit the map and $O(m)$ per candidate score, so the number $m$ of target grid points and the regularization $\varepsilon$ trade region size against runtime, with $m=2^{15}$ and $\varepsilon=0.1$ working well in the experiments.
Reading between the lines
- Because the coverage guarantee is independent of map quality, the honest comparison metric for OT-CP becomes region size alone: a poor transport map still gives valid sets, just larger ones, so the method's practical value hinges on how well entropic maps capture residual geometry in modest dimensions.
- The scalar calibration step means OT-CP can inherit univariate conformal extensions, such as covariate-shift weighting, adaptive or conditional quantile calibration, or online exchangeability adjustments, by swapping the final ranking step without touching the transport machinery.
- A practitioner who reuses the calibration set to tune $\varepsilon$ or $m$ and then reports coverage on the same set is outside the exchangeability premise; a clean protocol is to choose hyperparameters on training data and reserve the calibration split for the radius.
- The framework suggests replacing the spherical-uniform target with a data-dependent or density-shaped target, which could trade away uniform-norm calibration simplicity for smaller regions at the cost of a new calibration rule.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OT-CP, a conformal prediction method for multivariate outputs in which vector-valued conformity scores are mapped to a univariate rank via an optimal transport map to a spherical uniform target, after which standard univariate conformal prediction is applied to the resulting norms. The authors claim a distribution-free finite-sample coverage guarantee for the resulting region (Proposition 3.6, Lemma B.2) and report smaller mean region sizes than M-CP and Merge-CP baselines on most low-dimensional benchmark datasets, with ablations over the entropic regularization ε and the number of target points m.
Significance. The conceptual idea of using optimal transport to define a multivariate depth or rank and then inheriting conformal guarantees is attractive and timely, and the paper includes a broad empirical study (24 datasets, ablations, runtime comparisons). The theoretical core, however, is a restatement of the standard exchangeability argument for conformal prediction; the paper's distinctive claim is that the guarantee holds for any approximated transport map, and that claim is where the proof and the implementation diverge. If the protocol issues are resolved, the method could be a useful addition to the multivariate conformal toolbox; in its current form, the finite-sample coverage guarantee is not established for the implemented algorithm.
major comments (3)
- [§3.3 / Lemma B.2 / §3.4] The coverage guarantee in Proposition 3.6 and Lemma B.2 requires the empirical quantile region bR_{α,n+1} to be a symmetric function of the full exchangeable sequence (Z1,...,Z_{n+1}). This holds only when the transport map is either a fixed map independent of the calibration data (split conformal) or a permutation-invariant function of all n+1 points (transductive). The implementation in Section 3.4 fits the entropic map Tε on the n calibration residuals and then calibrates the radius r̂ as the empirical quantile of the norms of those same residuals; the test score is transported by a map that was not fitted on it, so the transformed norms are not exchangeable. The averaging identity in Lemma B.2 therefore fails for the reported algorithm, and Proposition 3.6 does not cover the implementation. Remark 3.3's statement that the coverage guarantee 'does not hold if the transport map is computed solely on a hold-out independent dataset' conflates the calibration set with the split; the fix is to train the map on a separate training split (or use a pre-trained map) and calibrate on a disjoint set, or to recompute the map for each candidate transductively.
- [Definition 2.3 and Eq. (7), Eq. (14)] Definition 2.3 and Eq. (7): the claim that ||T(Z)|| is uniform on (0,1) when T pushes P to the 'uniform distribution on the unit ball' is only correct for the radial-uniform (spherically uniform) target measure, not for the Lebesgue-uniform measure on the ball, for which ||U|| has CDF r^d. The paper's own discrete construction in Section 3.1 is radial-uniform, so Definition 2.3 and Eq. (7) are internally inconsistent. Similarly, Eq. (14) states that ||T_{n+1}(Z_{n+1})|| follows U_{n+1}({0,1/nR,...,1}), but with nS directions per shell and equal mass per grid point, the radius distribution assigns mass nS/(n+1) to each positive shell (plus no/(n+1) at the origin), which is not uniform over the listed atoms. The definitions should be corrected to use the spherical uniform measure explicitly and the displayed distributional statements should be fixed accordingly.
- [Section 3.3, Eq. (19)] Proposition 3.6 is stated for an arbitrary approximation map T̂ with no condition, while the proof in Lemma B.2 requires the symmetry of bR_{α,n+1} in the full data vector. This is a nontrivial condition that fails for the entropic-map implementation described in Section 3.4. The proposition should state explicitly whether T̂ is fixed (split conformal) or permutation-invariant in the full data vector, and the claim that the guarantee holds 'for any approximation map' should be withdrawn or appropriately restricted.
minor comments (5)
- [Section 3.2] The phrase 'any approximation ˆT that preserves the permutation invariance of the score function' is undefined; a fixed pre-trained map is not a function of the data, while a map fitted on the calibration set is not permutation-invariant in the sense required by Lemma B.2, so the intended condition should be spelled out.
- [Section 3.4] The text 'Learning the entropic map estimator as in Section 3.4' is a self-reference; it should refer to Section 2.4.
- [Section 4.3] The low-dimensional claim 'Results in Figure 4' should reference Figure 1; the current figure cross-references between Figures 1, 2, 4, 6, and 8 are confusing.
- [Table 1] On several high-dimensional datasets the reported region sizes are astronomically large (e.g., 2E+13), which makes the region-size comparison across methods hard to interpret; a brief comment on this regime is needed.
- [Throughout] There are minor typos: 'discritbution' (Section 3.1), 'encouter' (Section 3.3), 'untractable' (Remark 3.3), and 'is is' (Proposition B.1).
Circularity Check
No significant circularity: the coverage guarantee follows from conformal exchangeability and empirical quantiles, not from fitted transport-map parameters.
full rationale
The central coverage claim (Proposition 3.6, Eq. 19) is obtained by applying univariate conformal calibration to the scalarized scores ||T-hat(S(X_i,Y_i))||. The radius r-hat_{α,n+1} is defined as the empirical quantile of exactly those norms, and Lemma B.2 proves coverage by averaging indicators over the exchangeable sequence Z_1,...,Z_{n+1}. This is the standard conformal construction, so the target coverage is not an input to the definition of the map or the radius. The entropic map (Eq. 11) is an estimator imported from the external OT literature (Pooladian & Niles-Weed 2021; Cuturi 2013), and the hyperparameters ε and m affect region size but not the proof of coverage. Self-citations (Cuturi 2013; Cuturi et al. 2022; Peyré & Cuturi 2019; Guha et al. 2024) are implementation or related-work references and are not load-bearing for Proposition 3.6. The main caveat is a protocol/validity concern, not circularity: Remark 3.3 explicitly warns that 'the coverage guarantee does not hold if the transport map is computed solely on a hold-out independent dataset,' and the implementation in Section 3.4 fits the entropic map on the n calibration residuals and calibrates on the same norms, while Lemma B.2 requires either a map fixed before calibration or one that is a permutation-invariant function of the full n+1 sequence. That gap concerns whether the finite-sample guarantee transfers to the reported protocol; it does not reduce the claimed result to its own inputs, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Entropic regularization ε =
0.1
- Number of target points m (sphere discretization) =
32768 = 2^15
- Target grid shell structure (nR radii, nS directions, no origin copies) =
implied by n+1 = nR nS + no
assumptions (4)
- domain assumption Exchangeability of the data points (Z_1,...,Z_{n+1})
- standard math Existence of the Brenier map T* pushing the source measure to the target measure (source has a density)
- ad hoc to paper Permutation invariance or symmetry of the approximation map T̂ for the transductive guarantee
- ad hoc to paper The discrete spherical target with equal mass per point yields the uniform rank distribution in Equation (14)
Cite this review
Pith. "Pith review of Multivariate Conformal Prediction using Optimal Transport." pith.science (2026). https://pith.science/paper/6PQCYIWI
@misc{pith2026250203609,
author = {Pith},
title = {Pith review of: Multivariate Conformal Prediction using Optimal Transport},
year = {2026},
howpublished = {\url{https://pith.science/paper/6PQCYIWI}},
note = {Machine review of arXiv:2502.03609}
}
read the original abstract
Conformal prediction (CP) quantifies the uncertainty of machine learning models by constructing sets of plausible outputs. These sets are constructed by leveraging a so-called conformity score, a quantity computed using the input point of interest, a prediction model, and past observations. CP sets are then obtained by evaluating the conformity score of all possible outputs, and selecting them according to the rank of their scores. Due to this ranking step, most CP approaches rely on a score functions that are univariate. The challenge in extending these scores to multivariate spaces lies in the fact that no canonical order for vectors exists. To address this, we leverage a natural extension of multivariate score ranking based on optimal transport (OT). Our method, OTCP, offers a principled framework for constructing conformal prediction sets in multidimensional settings, preserving distribution-free coverage guarantees with finite data samples. We demonstrate tangible gains in a benchmark dataset of multivariate regression problems and address computational \& statistical trade-offs that arise when estimating conformity scores through OT maps.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Manifold Constrained Conformal Prediction for Spatial Events
Manifold-constrained conformal prediction with sliced Wasserstein scores yields near-nominal coverage and lower energy/manifold distances for spatial event clouds than HDR or generative baselines.
Reference graph
Works this paper leans on
-
[1]
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511,
-
[9]
doi: 10.1214/20-AOS1996. URL https://doi.org/10. 1214/20-AOS1996. Hallin, M., La Vecchia, D., and Liu, H. Center-outward r-estimation for semiparametric varma models.Journal of the American Statistical Association, 117(538):925–938,
-
[10]
Hallin, M., Hlubinka, D., and Hudecov´a, ˇS. Efficient fully distribution-free center-outward rank tests for multiple- output regression and manova. Journal of the American Statistical Association, 118(543):1923–1939,
work page 1923
-
[14]
Semiparametric conformal prediction
10 Multivariate Conformal Prediction using Optimal Transport Park, J. W., Tibshirani, R., and Cho, K. Semiparametric conformal prediction. arXiv preprint arXiv:2411.02114,
-
[16]
Quach, V ., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. Conformal language modeling. arXiv preprint arXiv:2306.10193,
-
[17]
URL https:// arxiv.org/abs/2501.18991. Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. Conformal prediction under covariate shift. Advances in neural information processing systems, 32,
-
[19]
Conformalized adaptive forecasting of heterogeneous trajectories
Zhou, Y ., Lindemann, L., and Sesia, M. Conformalized adaptive forecasting of heterogeneous trajectories. arXiv preprint arXiv:2402.09623,
-
[1990]
Guha, E., Natarajan, S., M ¨ollenhoff, T., Khan, M. E., and Ndiaye, E. Conformal prediction via regression- as-classification. arXiv preprint arXiv:2404.08168,
Show all 19 references
-
[1991]
3160440402
doi: 10.1002/cpa. 3160440402. Cella, L. and Ryan, R. Valid distribution-free inferential models for prediction. arXiv preprint arXiv:2001.09225,
2001 arXiv
-
[2005]
Wang, Z., Gao, R., Yin, M., Zhou, M., and Blei, D. M. Prob- abilistic conformal prediction using conditional random samples. arXiv preprint arXiv:2206.06584,
-
[2008]
Holland, M. J. Making learning more transparent using conformalized performance prediction. arXiv preprint arXiv:2007.04486,
2007 arXiv
-
[2016]
Testing for outliers with conformal p-values
Bates, S., Cand `es, E., Lei, L., Romano, Y ., and Sesia, M. Testing for outliers with conformal p-values. arXiv preprint arXiv:2104.08279,
-
[2017]
URL https://doi.org/10
doi: 10.1214/16-AOS1450. URL https://doi.org/10. 1214/16-AOS1450. Chernozhukov, V ., W¨uthrich, K., and Zhu, Y . Exact and ro- bust conformal inference methods for predictive machine learning with dependent data. Conference On Learning Theory,
-
[2019]
and Niles-Weed, J
Pooladian, A.-A. and Niles-Weed, J. Entropic estimation of optimal transport maps. arXiv preprint arXiv:2109.12004,
-
[2021]
Statistical opti- mal transport
Chewi, S., Niles-Weed, J., and Rigollet, P. Statistical opti- mal transport. arXiv preprint arXiv:2407.18163,
-
[2022]
Dheur, V ., Fontana, M., Estievenart, Y ., Desobry, N., and Taieb, S
URL https://arxiv.org/abs/2201.12324. Dheur, V ., Fontana, M., Estievenart, Y ., Desobry, N., and Taieb, S. B. Multi-output conformal regression: A unified comparative study with new conformity scores,
-
[2023]
Adap- tive inference with random ellipsoids through confor- mal conditional linear expectation
Henderson, I., Mazoyer, A., and Gamboa, F. Adap- tive inference with random ellipsoids through confor- mal conditional linear expectation. arXiv preprint arXiv:2409.18508,
-
[2024]
Kumar, B., Lu, C., Gupta, G., Palepu, A., Bellamy, D., Raskar, R., and Beam, A
URL https://proceedings.mlr.press/ v230/katsios24a.html. Kumar, B., Lu, C., Gupta, G., Palepu, A., Bellamy, D., Raskar, R., and Beam, A. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404,
-
[2025]
Fisch, A., Schuster, T., Jaakkola, T., and Barzilay, R
URL https://arxiv.org/abs/2501.10533. Fisch, A., Schuster, T., Jaakkola, T., and Barzilay, R. Few- shot conformal prediction with auxiliary tasks. ICML,
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.