REVIEW 3 major objections 3 minor 31 references
The paper claims that prediction intervals valid conditional on the point prediction itself are achievable with a single isotonic recalibration fit, and that a transductive variant adds exact finite-sample self-calibration without per-candi
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 20:17 UTC pith:QQEILSJR
load-bearing objection SICP is a genuine, lightweight contribution to prediction-conditional coverage; TICP's advertised exact self-calibration is an oracle artifact and should not be accepted as stated. the 3 major comments →
Isotonic Conformal Prediction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims two things. Split Isotonic Conformal Prediction obtains finite-sample coverage conditional on the calibrated prediction (Theorem 1) by fitting one isotonic recalibration map and computing the conformal quantile only among calibration points sharing the test point's recalibrated value; its calibrator is exactly self-calibrated on the fitting split and asymptotically on the population (Theorem 2). Transductive Isotonic Conformal Prediction obtains both exact finite-sample self-calibration and prediction-conditional coverage (Theorems 3 and 4) by replacing the global refit with a scalar within-bin mean update. Both procedures need only one isotonic fit, so SC-CP-style guarantee
What carries the argument
The central object is the isotonic recalibration map θ̂ = argmin over nondecreasing functions of Σ(Y_i − θ(f(X_i)))^2, composed with the base model to give f̂ = θ̂ ∘ f. Because θ̂ is a nondecreasing step function, f̂ takes finitely many distinct values and partitions the feature space into level sets; membership in a level set is exactly the event that the calibrated prediction equals a given value. Conditioning on the prediction therefore reduces to conditioning on a Mondrian-style category, and the conformal quantile can be computed from the calibration scores inside that category. The transductive variant's second key object is the local mean μ̂^(y)(X_{n+1}) = (Σ_{i∈I} Y_i + y)/(m+1), the
Load-bearing premise
The load-bearing premise is that at least one held-out calibration point shares the test point's calibrated prediction value; when none does, both procedures fall back to a marginal interval and the prediction-conditional guarantee lapses for that test point.
What would settle it
Run either algorithm on data where the isotonic calibrator produces many singleton level sets (or where test points frequently land in empty level sets), and measure coverage only on test points whose level set is empty in the calibration split. The fallback to a marginal interval will yield coverage below the nominal level on that stratum, showing that the finite-sample conditional guarantee requires the non-empty level-set condition.
If this is right
- Prediction-conditional coverage is achievable at split-conformal cost: SICP's per-test-point work is only a within-level-set quantile, so latency-sensitive applications can get conditional validity.
- Exact finite-sample self-calibration is decoupled from candidate-wise refitting: TICP recovers SC-CP's dual guarantee with a single isotonic fit plus an inner loop over candidate labels.
- Calibration before interval construction isolates aleatoric variability from model bias, so interval width reflects genuine outcome uncertainty rather than systematic miscalibration; experiments show narrower intervals at matched coverage when the base model is miscalibrated.
- Because level sets are learned rather than fixed in advance, the method yields group-conditional coverage as a by-product when a sensitive attribute is correlated with the calibrated prediction, as observed in the healthcare-utilization and bike-sharing experiments.
- The guarantees inherit split-sample trade-offs: partitioning the calibration data into D1 and D2 reduces the effective sample size for each stage, and SICP's asymptotic self-calibration requires continuity and strict monotonicity of the true calibration curve.
Where Pith is reading between the lines
- Editorial inference: the framework suggests a general recipe—any recalibrator that outputs finitely many distinct values (e.g., histogram binning or tree-based models) could be substituted for isotonic regression and would inherit the same prediction-conditional coverage argument, provided level sets are populated.
- Editorial inference: the m=0 fallback means the guarantee is conditional on the level set being non-empty; in settings where calibrated predictions are near-continuous or data are sparse, the practical value depends on how often level sets contain enough calibration points, a quantity the paper does not quantify.
- Editorial inference: a natural testable extension is an adaptive grid for candidate labels in TICP; because the local mean update is scalar, one could replace a uniform grid with a bisection or root-finding scheme to locate interval endpoints without enumerating K labels, potentially removing the factor K.
- Editorial inference: the self-calibration identity in Theorem 3 holds for any symmetric within-bin statistic, not only the mean; using a median or quantile-based local center would trade exact mean calibration for robustness to heavy tails while preserving the exchangeability argument.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Isotonic Conformal Prediction (ICP), a framework for obtaining prediction-conditional coverage and self-calibration at lower computational cost than Self-Calibrating Conformal Prediction (SC-CP). Two procedures are introduced: Split Isotonic Conformal Prediction (SICP), which fits a single isotonic calibrator on one split and applies split conformal prediction within the level sets of the resulting step function; and Transductive Isotonic Conformal Prediction (TICP), which replaces fixed residuals by a transductive local mean that includes the candidate outcome and claims exact finite-sample self-calibration and prediction-conditional coverage. The paper reports synthetic and real-data experiments showing coverage comparable to SC-CP with large runtime gains, and reports zero calibration error for TICP.
Significance. The motivation is timely and important: prediction-conditional coverage is a practically relevant relaxation of the impossible full conditional coverage, and reducing the computational burden of SC-CP is a worthwhile goal. SICP's prediction-conditional coverage theorem appears valid and its computational advantage over SC-CP is real. If the TICP claims were supported, the paper would make a substantial contribution. However, as stated, the exact finite-sample guarantees for TICP are established only for an oracle object that depends on the true test outcome, not for any prediction that the method outputs. This affects the central advertised advantage of TICP and the interpretation of the calibration-error results. The SICP contribution remains sound, but the paper's headline claims about TICP need substantial revision.
major comments (3)
- [Section 3.2.1, Theorem 3] The theorem defines mu_hat(X_{n+1}) = mu_hat^(Y_{n+1})(X_{n+1}), i.e. the within-bin mean computed with the true outcome included. This quantity is not a function of the covariates and calibration data alone; it cannot be evaluated before Y_{n+1} is observed. Algorithm 2 outputs only a prediction interval and never defines a point predictor, so the self-calibration guarantee E[Y_{n+1}|mu_hat(X_{n+1})] = mu_hat(X_{n+1}) does not attach to any object the method actually delivers. The only computable fixed point of Eq. (4) is the within-bin mean (1/m) sum_{i in I} Y_i, and for exchangeable data E[Y_{n+1} | (1/m)sum Y_i] need not equal that mean. Therefore Theorem 3 is an algebraic identity for an oracle mean, not a property of a deployable predictor. Consequently, the zero calibration error reported for TICP in Tables 2–11 and the abstract's claim that TICP attains exact finite-sample self-
- [Section 3.2.1, Theorem 4] The same oracle issue arises for the coverage guarantee. The theorem conditions on mu_hat^(Y_{n+1})(X_{n+1}), which depends on the unknown test outcome. Thus the 'prediction-conditional coverage' established in Theorem 4 is not conditional on any prediction that is a function of X_{n+1} and the training data. The interval produced by Algorithm 2 is not shown to satisfy coverage conditional on any computable point prediction. This undermines the comparison with SC-CP, whose guarantees are stated for its output prediction. The proof of Theorem 4 is internally correct for the conditioning variable used, but that variable is not available at prediction time.
- [Section 3.1 and 3.2, m = 0 cases] Theorems 1, 3, and 4 do not state any lower bound on m = |I(X_{n+1})|, but the text and algorithms specify a fallback to a marginal split-conformal interval when m = 0. The proofs effectively cover only the trivial case where S = {n+1} and the conformal quantile is infinite under the algorithm's definition (R ∪ {∞}); the actual fallback procedure is not analyzed in the theorem statements. Section 6 acknowledges the limitation, but the theorem statements and algorithm descriptions need to be reconciled. This is a presentation/consistency issue rather than a fundamental one for the populated-strata guarantees.
minor comments (3)
- [Throughout] There are several typographical errors: 'Prediciton' in the introduction, 'T ransductive' in Section 3.2 heading, 'under that fall under that fall' in Section 3, 'A rigourous framework' in Section 2.2, and 'Conferance' in the references. The references for Zadrozny and Elkan and Guo et al. lack years and complete venue information.
- [Algorithm 1 and 2] Algorithm 1 line 6 computes q as Quantile(1-α, R ∪ {∞}); when R is empty this returns ∞, so the algorithm returns the whole output space, not the marginal split-conformal interval described in Section 3.1. The same issue appears in Algorithm 2 when m = 0. The fallback described in the text is not implemented by the pseudocode.
- [Section 4 and 5] The calibration error definition is not stated precisely. For TICP the reported zero values are attributed to the algebraic identity in Theorem 3, but since Theorem 3 concerns an oracle object, the table entries are not measuring a property of any output prediction. Please clarify the metric and the object to which it is applied.
Circularity Check
TICP's exact finite-sample self-calibration and prediction-conditional coverage are defined in terms of the true outcome, making the guarantees algebraic identities rather than properties of a computable predictor.
specific steps
-
self definitional
[Section 3.2.1, Theorem 3 and Eq. (4)]
"Define ˆµ(Xn+1) = ˆµ(Yn+1)(Xn+1). Then E[Y n+1 |ˆµ(Xn+1)] = ˆµ(Xn+1) a.s."
The object being calibrated is the within-bin mean that includes the true outcome Y_{n+1}. For any finite bin, the sum of deviations from the bin mean is identically zero, which is exactly the algebraic identity used in the proof. Thus the theorem states a property of the outcome's own group average, not a property of a fitted or usable predictor. Algorithm 2 never outputs such a point prediction, so the exact finite-sample self-calibration claim is true by definition rather than by estimation.
-
self definitional
[Section 3.2.1, Theorem 4]
"P Yn+1 ∈ bC(X n+1) ˆµ(Yn+1)(Xn+1) ≥1−α."
The conditioning statistic ˆµ^(Yn+1)(Xn+1) is a function of the unknown test outcome. Thus the coverage guarantee is not conditional on a prediction available before observing Y_{n+1}; it does not match objective (ii) in Eq. (2), which conditions on ˆf(Xn+1). The interval's validity conditional on this outcome-dependent oracle is a conformal exchangeability artifact, centered on the same outcome-dependent mean, rather than a property of a computable prediction. The claimed finite-sample prediction-conditional validity is therefore a statement about an oracle, not about a deployable interval.
-
other
[Section 5.1, footnote 2 / Figure 2 caption]
"Transductive Isotonic CP achieves zero calibration error by construction2 ... 2 The first-order conditions of the isotonic regression objective imply E[Y|f (X)] = f (X) on the augmented calibration set; see Theorem 4.1 of van der Laan and Alaa (2024)."
The zero calibration error reported for TICP in Tables 2–11 is acknowledged to be 'by construction'. It is therefore not an empirical validation of a model's calibration; it is an algebraic consequence of the oracle construction used in the method. Presenting this forced value as evidence of exact self-calibration is a fitted-input-called-prediction pattern: the construction itself guarantees the zero, so the reported calibration error carries no independent empirical support.
full rationale
SICP's Theorem 1 is a legitimate finite-sample coverage result derived from exchangeability of the calibration and test points within a level set; that portion is not circular. The circularity is concentrated in TICP, the paper's headline method for recovering both objectives exactly. Theorem 3 defines its 'prediction' as the within-bin mean including the true test outcome, so the self-calibration equation is an algebraic identity (the average residual is zero by construction) rather than a property of a computable predictor. Theorem 4 conditions on the same outcome-dependent oracle, so the claimed prediction-conditional coverage is not conditional on a prediction a user could know. The experiments further present zero calibration error for TICP, which the paper itself admits is 'by construction'. These are not mere omissions: the central advertised advantage of TICP—exact finite-sample self-calibration at low computational cost—reduces to the definition of the oracle statistic. The score reflects that this is a partial but serious circularity: one of the two proposed procedures (SICP) is non-circular, but the stronger claims of the other procedure (TICP) are by-construction identities. No self-citation chains or imported uniqueness theorems are involved.
Axiom & Free-Parameter Ledger
free parameters (2)
- α (miscoverage level)
- K (grid size for discretizing continuous Y)
axioms (5)
- domain assumption Exchangeability of the n+1 data points (standard split conformal assumption).
- domain assumption Assumption 1: i.i.d. draws, E[Y^2]<∞, compact support of base predictions f(X).
- domain assumption Assumption 2: true calibration function θ0(s)=E[Y|f(X)=s] is continuous and strictly increasing.
- standard math Isotonic regression is uniformly consistent under Assumptions 1-2 (Robertson et al., 1988).
- standard math For any exchangeable random variables Z_1,...,Z_N, E[Z_i | (1/N)∑Z_j] = (1/N)∑Z_j.
read the original abstract
A point prediction that is well calibrated on average can still be systematically biased conditional on its own value, undermining its use in downstream decision-making. We consider two objectives for reliable uncertainty quantification: self-calibration, requiring a point prediction to be unbiased conditional on its own value, and prediction-conditional validity, requiring a prediction interval to attain nominal coverage conditional on the prediction. Self-Calibrating Conformal Prediction (SC-CP) attains both objectives exactly in finite samples, but requires refitting its calibrator for every candidate outcome, which is computationally prohibitive for continuous outcomes. We propose Isotonic Conformal Prediction (ICP), a framework that decouples calibration from prediction-set construction by fitting a single isotonic recalibration map and constructing prediction intervals within strata of similar recalibrated predictions. Within this framework we develop two procedures. Split Isotonic Conformal Prediction (SICP) attains prediction-conditional validity in finite samples and self-calibration asymptotically, at the computational cost of split conformal prediction. Transductive Isotonic Conformal Prediction (TICP) attains both objectives exactly in finite samples through a per-test-point inner loop that avoids refitting the isotonic calibrator. On synthetic heteroscedastic regression problems and a real-world healthcare-utilization dataset, both procedures match the coverage of SC-CP at substantially lower computational cost.
Figures
Reference graph
Works this paper leans on
-
[1]
1988 , publisher=
Order restricted statistical inference , author=. 1988 , publisher=
1988
-
[2]
Technical Report , year=
Mondrian confidence machine , author=. Technical Report , year=
-
[3]
Information and Inference: A Journal of the IMA , volume=
The limits of distribution-free conditional predictive inference , author=. Information and Inference: A Journal of the IMA , volume=. 2021 , publisher=
2021
-
[4]
The Annals of Statistics , volume=
Conformal prediction beyond exchangeability , author=. The Annals of Statistics , volume=. 2023 , publisher=
2023
-
[5]
Advances in neural information processing systems , volume=
Conformal prediction under covariate shift , author=. Advances in neural information processing systems , volume=
-
[6]
Inductive Confidence Machines for Regression
Papadopoulos, Harris and Proedrou, Kostas and Vovk, Volodya and Gammerman, Alex. Inductive Confidence Machines for Regression. Machine Learning: ECML 2002. 2002
2002
-
[7]
Self-Calibrating Conformal Prediction
van der Laan, Lars and Alaa, Ahmed M. , year = 2024, month = oct, number =. Self-. doi:10.48550/arXiv.2402.07307 , urldate =. arXiv , keywords =:2402.07307 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.07307 2024
- [8]
-
[9]
Lei, Jing and Wasserman, Larry , year = 2014, month = jan, journal =. Distribution-Free. doi:10.1111/rssb.12021 , urldate =
-
[10]
Gibbs, Isaac and Cherian, John J. and Cand. Conformal. doi:10.48550/arXiv.2305.12616 , urldate =. arXiv , keywords =:2305.12616 , primaryclass =
-
[11]
Conformal Prediction with Local Weights: Randomization Enables Local Guarantees , shorttitle =
Hore, Rohan and Barber, Rina Foygel , year = 2024, month = oct, number =. Conformal Prediction with Local Weights: Randomization Enables Local Guarantees , shorttitle =. doi:10.48550/arXiv.2310.07850 , urldate =. arXiv , keywords =:2310.07850 , primaryclass =
-
[12]
and Wasserman, Larry , year = 2017, month = mar, number =
Lei, Jing and G'Sell, Max and Rinaldo, Alessandro and Tibshirani, Ryan J. and Wasserman, Larry , year = 2017, month = mar, number =. Distribution-. doi:10.48550/arXiv.1604.04173 , urldate =. arXiv , keywords =:1604.04173 , primaryclass =
-
[13]
Koenker, Roger and Bassett, Gilbert , year = 1978, month = jan, journal =. Regression. doi:10.2307/1913643 , urldate =. 1913643 , eprinttype =
doi:10.2307/1913643 1978
-
[14]
Causal isotonic calibration for heterogeneous treatment effects
Causal Isotonic Calibration for Heterogeneous Treatment Effects , author =. doi:10.48550/arXiv.2302.14011 , urldate =. arXiv , keywords =:2302.14011 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2302.14011
-
[15]
Distribution-free calibration guarantees for histogram binning without sample splitting
Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting , author =. doi:10.48550/arXiv.2105.04656 , urldate =. arXiv , keywords =:2105.04656 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2105.04656
-
[16]
Barlow, R. E. and Brunk, H. D. , year = 1972, journal =. The. doi:10.2307/2284712 , urldate =. 2284712 , eprinttype =
-
[17]
Lichtenstein, Sarah and Fischhoff, Baruch and Phillips, Lawrence D. , editor =. Calibration of. Decision. doi:10.1007/978-94-010-1276-8_19 , urldate =
-
[18]
Obtaining Calibrated Probability Estimates from Decision Trees and Naive
Zadrozny, Bianca and Elkan, Charles , journal =. Obtaining Calibrated Probability Estimates from Decision Trees and Naive
-
[19]
Calibration of
Bella, Antonio and Ferri, C. Calibration of
-
[20]
, date =
Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q. , date =. On. Proceedings of the 34th
-
[21]
Information and Inference: A Journal of the IMA , volume =
The Limits of Distribution-Free Conditional Predictive Inference , author =. Information and Inference: A Journal of the IMA , volume =. doi:10.1093/imaiai/iaaa017 , urldate =
-
[22]
Cohen, Joel W. and Cohen, Steven B. and Banthin, Jessica S. , year = 2009, month = jul, journal =. The Medical Expenditure Panel Survey: A National Information Resource to Support Healthcare Cost Research and Inform Policy and Practice , shorttitle =. doi:10.1097/MLR.0b013e3181a23e3a , abstract =
-
[23]
and Barber, Rina Foygel and Bates, Stephen , year = 2026, month = mar, number =
Angelopoulos, Anastasios N. and Barber, Rina Foygel and Bates, Stephen , year = 2026, month = mar, number =. Theoretical. doi:10.48550/arXiv.2411.11824 , urldate =. arXiv , keywords =:2411.11824 , primaryclass =
-
[24]
Advances in Neural Information Processing Systems , volume =
Conformalized Quantile Regression , author =. Advances in Neural Information Processing Systems , volume =. 2019 , publisher =
2019
-
[25]
Vovk, Vladimir and Petej, Ivan , year = 2014, month = jun, number =. Venn-. doi:10.48550/arXiv.1211.0025 , urldate =. arXiv , keywords =:1211.0025 , primaryclass =
-
[26]
Facilitating Structure-Based Drug Discovery with an Artificial Intelligence-Driven Virtual Screening Platform , author =. Nature Protocols , pages =. doi:10.1038/s41596-026-01389-z , urldate =
-
[27]
Nature Reviews Drug Discovery , volume =
Applications of Machine Learning in Drug Discovery and Development , author =. Nature Reviews Drug Discovery , volume =. doi:10.1038/s41573-019-0024-5 , urldate =
-
[28]
, year = 2019, month = jan, journal =
Topol, Eric J. , year = 2019, month = jan, journal =. High-Performance Medicine: The Convergence of Human and Artificial Intelligence , shorttitle =. doi:10.1038/s41591-018-0300-7 , urldate =
-
[29]
Large Language Models Encode Clinical Knowledge , author =. Nature , volume =. doi:10.1038/s41586-023-06291-2 , urldate =
-
[30]
Distribution-free binary classification: prediction sets, confidence intervals and calibration
Gupta, Chirag and Podkopaev, Aleksandr and Ramdas, Aaditya , year = 2022, month = feb, number =. Distribution-Free Binary Classification: Prediction Sets, Confidence Intervals and Calibration , shorttitle =. doi:10.48550/arXiv.2006.10564 , urldate =. arXiv , keywords =:2006.10564 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2006.10564 2022
-
[31]
, year = 2007, month = apr, journal =
Gneiting, Tilmann and Balabdaoui, Fadoua and Raftery, Adrian E. , year = 2007, month = apr, journal =. Probabilistic. doi:10.1111/j.1467-9868.2007.00587.x , urldate =
arXiv 2007
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.