REVIEW 11 references
On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals
T0 review · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Training-conditional conformal prediction does not yield valid binomial proportion confidence intervals or safety guarantees: its PAC bound can be satisfied by trivial whole-space predictions.
desk verdict A correct and sharp N=2 counterexample showing training-conditional CP doesn't yield binomial proportion confidence intervals, but the Section 4.1 leap from continuous to binary scores is asserted, not proved. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pair is Theorem 1's PAC guarantee for training-conditional conformal prediction together with an indicator nonconformity measure (equation 8) that turns calibration scores into i.i.d. Bernoulli random variables. The argument works by enumerating the three possible calibration outcomes for N=2 (both points outside $Q$, one inside, both inside) and computing the resulting predicted set $\Gamma_\epsilon$: for $\epsilon\in[2/3,1)$, the only way to reach coverage $1-E$ when $b>E$ is the event $Q\times Q$, which forces $\Gamma_\epsilon=Z$. That event has probability $b^2$, so the bound $P^2(S_E)\ge E^2$ is met by trivial whole-space predictions, not by estimating the Bernoulli parameter $b$.
What would settle it
Repeat the appendix's N=2 experiment with $b>E$ and $\epsilon\in[2/3,1)$, recording on each calibration draw whether the predicted set is $Z$ and whether the new score is covered. If the bound $P^2(S_E)\ge E^2$ holds only through draws with $\Gamma_\epsilon=Z$, and the interval $[0,E]$ never contains $b$ when $b>E$, the paper's conclusion is confirmed; a draw with $\Gamma_\epsilon=Q$ that still covers the new score at the required level would refute it.
Extended reading notes
Core claim
The paper's central claim is that training-conditional conformal prediction, as used in recent safety-certification work, does not produce valid binomial proportion confidence intervals and therefore does not provide the statistical safety guarantees claimed. The demonstration is Example 1: with N=2 calibration points and an indicator nonconformity measure $A(z)=1_{\{z\in Q\}}$, the calibration scores are i.i.d. Bernoulli with parameter $b=P(Q)$. For any $\epsilon\in[2/3,1)$ the predicted set $\Gamma_\epsilon$ is the whole space $Z$ exactly when both calibration points fall in $Q$, an event of probability $b^2$, and is $Q$ otherwise. The PAC statement $P^2(S_E)\ge E^2$ then holds for $b>E$ only because $b^2\ge E^2$ on those trivial whole-space predictions, while the non-trivial prediction $Q$ fails to meet the coverage requirement; when $b\le E$ the coverage requirement is met automatically. Hence the confidence level does not estimate $b$, and the authors conclude that conformal prediction is unsuitable for binomial proportion problems and that direct binomial interval methods such as Clopper-Pearson should be used for statistical safety certification.
Load-bearing premise
The load-bearing assumption is that the nonconformity score is a fixed 0/1 indicator of membership in a set $Q$ with a fixed training set, so each calibration score is a Bernoulli coin flip; if scores were continuous or $Q$ depended on calibration data, the trivial-prediction failure for $N=2$ would not automatically generalize and a different proof would be needed.
Editorial extensions
If this is right
- Safety certificates built on Theorem 1 in the style of Lin & Bansal (2024) and Chilakamarri et al. (2024) are not valid guarantees of the probability of safe operation.
- The claimed equivalence by Vincent et al. (2024), that training-conditional conformal prediction with Bernoulli scores reduces to the Clopper-Pearson interval, is false.
- For safety certification, a Clopper-Pearson interval computed directly from Bernoulli evaluations of a trajectory cost function gives a valid binomial proportion confidence interval, whereas Theorem 1 does not.
- The mathematical content of Theorem 1 remains intact; what changes is its interpretation: it bounds how often the set predictor has coverage $1-E$, not how accurately any class or score probability is estimated.
Reading between the lines
- A testable extension: for N>2 with the same indicator score, the fraction of calibration sets that achieve coverage $1-E$ by predicting the whole space should still account for essentially all of the confidence when $b>E$; this could be quantified by simulation.
- The same vacuous-satisfaction mechanism should appear in any discrete-label conformal setting where the nonconformity score takes finitely many values, because a loose prediction (all labels) contributes to coverage without estimating label probabilities.
- If safety certification needs an upper confidence bound on the probability of entering an unsafe region, the Bernoulli calibration scores can be fed directly into a binomial proportion method; the paper's example suggests the conformal route cannot be repaired merely by choosing different $\epsilon$ and $E$ values.
Editorial analysis
A structured set of objections, weighed in public.
Circularity Check
No significant circularity: the paper derives its counterexample from the definitions of training-conditional conformal prediction and an indicator nonconformity measure, using an external theorem from Vovk (2012) and no fitted parameters.
full rationale
The paper contains no self-citations and no fitted-input-called-prediction steps. Its central derivation in Example 1 is a direct computation from the definitions of the INP (equations 3–5), the indicator nonconformity measure (equation 8), and the coverage event S_E (equation 6). The claim that training-conditional CP does not provide a BPCI is obtained by comparing the event S_E with the BPCI coverage probability in equation 2; the comparison is explicit and can be checked from the computed cases. Theorem 1 is cited from Vovk (2012), an external source, and is not used to justify the paper's own conclusion beyond providing the bound P^2(S_E) ≥ E^2, which is then examined by direct calculation. The appendix simulation confirms a known bound rather than fitting a parameter to produce a new prediction. The arguably debatable step in Section 4.1, equating a continuous safety cost J(x) with an indicator nonconformity measure, is an analytical modeling assumption rather than a circular reduction: it does not presuppose the target claim that CP is unsuitable for BPCI. Even if that equivalence is contestable on correctness grounds, the paper's main logical chain is not circular. The paper does not derive its conclusion from itself or from prior work by the same authors; it is self-contained against an external theorem and explicit definitions.
Assumptions & free parameters
assumptions (4)
- standard math Theorem 1 of Vovk (2012): for training-conditional CP, P^N(S_E) >= 1 - delta with delta = Bin_{N,E}(J), J=floor(epsilon(N+1)-1).
- domain assumption Indicator nonconformity measure A(z)=1_{z in Q} with fixed training set yields i.i.d. Bernoulli calibration scores with parameter b=P(Q).
- domain assumption Calibration and test samples are i.i.d. under an unknown distribution P, as required by Theorem 1 and the BPCI setup in Section 2.
- standard math Definition of conservative validity of a confidence interval: coverage probability rho(b, b_check, b_hat) >= 1-alpha for all b in [0,1] (equation 1).
Cite this review
Pith. "Pith review of On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals." pith.science (2026). https://pith.science/paper/27BUSVW5
@misc{pith2026250207497,
author = {Pith},
title = {Pith review of: On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals},
year = {2026},
howpublished = {\url{https://pith.science/paper/27BUSVW5}},
note = {Machine review of arXiv:2502.07497}
}
read the original abstract
Estimating the expectation of a Bernoulli random variable based on N independent trials is a classical problem in statistics, typically addressed using Binomial Proportion Confidence Intervals (BPCI). In the control systems community, many critical tasks-such as certifying the statistical safety of dynamical systems-can be formulated as BPCI problems. Conformal Prediction (CP), a distribution-free technique for uncertainty quantification, has gained significant attention in recent years and has been applied to various control systems problems, particularly to address uncertainties in learned dynamics or controllers. A variant known as training-conditional CP was recently employed to tackle the problem of safety certification. In this note, we highlight that the use of training-conditional CP in this context does not provide valid safety guarantees. We demonstrate why CP is unsuitable for BPCI problems and argue that traditional BPCI methods are better suited for statistical safety certification.
Figures
Reference graph
Works this paper leans on
-
[1]
Conformal prediction: A gentle introduction
Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning , 16 0 (4): 0 494--591, 2023
work page 2023
-
[2]
Reachability Analysis for Black-Box Dynamical Systems
Vamsi Krishna Chilakamarri, Zeyuan Feng, and Somil Bansal. Reachability analysis for black-box dynamical systems. arXiv preprint arXiv:2410.07796, 2024
work page Pith review arXiv 2024
-
[3]
The use of confidence or fiducial limits illustrated in the case of the binomial
Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26 0 (4): 0 404--413, 1934
work page 1934
-
[4]
Evaluating confidence interval methods for binomial proportions in clustered surveys
Natalie Dean and Marcello Pagano. Evaluating confidence interval methods for binomial proportions in clustered surveys. Journal of Survey Statistics and Methodology, 3 0 (4): 0 484--503, 2015
work page 2015
-
[5]
Conformal prediction: a unified review of theory and new challenges
Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29 0 (1): 0 1--23, 2023
2023
-
[6]
Verification of neural reachable tubes via scenario optimization and conformal prediction
Albert Lin and Somil Bansal. Verification of neural reachable tubes via scenario optimization and conformal prediction. In 6th Annual Learning for Dynamics & Control Conference, pp.\ 719--731. PMLR, 2024
work page 2024
-
[7]
A tutorial on conformal prediction
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9 0 (3), 2008
2008
-
[8]
Guarantees on robot system performance using stochastic simulation rollouts
Joseph A Vincent, Aaron O Feldman, and Mac Schwager. Guarantees on robot system performance using stochastic simulation rollouts. IEEE Transactions on Robotics, 2024
work page 2024
Show all 11 references
-
[9]
Conditional validity of inductive conformal predictors
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pp.\ 475--490. PMLR, 2012
2012
-
[10]
Algorithmic learning in a random world, volume 29
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005
2005
-
[11]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.