Conformal prediction without knowledge of labeled calibration data

Jonas Flechsig; Maximilian Pilz

REVIEW 3 major objections 4 minor 13 references

Conformal prediction without knowledge of labeled calibration data

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that conformal prediction can still guarantee coverage of the true label using only unlabeled calibration data, paying a penalty β equal to the model's exactness gap.

desk verdict True theorem, false lemma: the coverage bound survives but the written proof and the numeric example both need fixing. read the letter →

arxiv 2509.10321 v1 pith:NDJO6U3Y submitted 2025-09-12 stat.ME

Jonas Flechsig , Maximilian Pilz This is my paper

classification stat.ME MSC 62G1562H30

keywords conformalpredictiondistribution-freeuncertaintyquantificationunlabeledcalibrationdatacoverageguaranteemodelaccuracyclassificationregressionexactnessmeasure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conformal prediction normally needs a labeled calibration set to set its cutoff. This paper removes that requirement: it replaces true calibration labels with labels predicted by the model, then proves a coverage guarantee that depends on how exact the model is. For a classifier with accuracy at least 1−β, the conformity set covers the true label with probability at least 1−α−β; for regression, the same bound holds when exactness is measured by a quantile of the absolute error. The result matters because models can often be released without any labeled calibration data, and this shows that uncertainty quantification is still possible with a known performance number.

What carries the argument

The central object is the estimated score ŝ_k = s(X_k, Ŷ_k), where Ŷ_k is chosen from the β̃-neighborhood T_β̃(X_k) = {y : |s(X_k, f̂(X_k)) − s(X_k,y)| ≤ β̃} as the label maximizing the score. The assumption of (β̃,β)-exactness says the true label falls in that neighborhood with probability at least 1−β; for classification this is accuracy, for regression it is a quantile of the absolute error. The mechanism is dominance: with probability at least 1−β, ŝ_k ≥ s_k for every calibration point, so the empirical quantile of estimated scores dominates the true-score quantile with that probability; this transfers the standard conformal coverage bound 1−α down to 1−α−β.

What would settle it

Simulate a binary classifier whose true accuracy is 0.7 but set β=0.1 (as if accuracy were 0.9); with α=0.05 and a large unlabeled calibration set, compute C(X_test) from predicted labels for many test points. The empirical coverage will fall well below the claimed 0.85 lower bound, demonstrating that the guarantee fails when β is underestimated.

Watch

Extended reading notes

Core claim

Theorem 2 is the core claim: if a model is (β̃,β)-exact — the true label's score lies within β̃ of the predicted label's score with probability at least 1−β — then a conformity set built from unlabeled calibration data covers the test label with probability at least 1−α−β. In classification with score 1−p, (0,β)-exactness is exactly accuracy 1−β. In regression with absolute-error score, median absolute error makes the model (MedAE, 0.5)-exact. The proof uses estimated scores computed from predicted labels and shows, via a quantile-dominance lemma, that the cutoff from estimated scores is at least the cutoff from true scores with probability at least 1−β, converting the standard conformal gua

Load-bearing premise

The load-bearing premise is that the exactness values β̃ and β are known; the paper itself states that in practice they can only be estimated from training data, and if β is underestimated the stated 1−α−β coverage bound can fail.

Editorial extensions

If this is right

A classifier with known accuracy at least 1−β can produce valid conformal sets from unlabeled data, with coverage at least 1−α−β.
A regression model with median absolute error MedAE yields the same style of guarantee with β=0.5, using only unlabeled calibration data.
Larger β̃ gives larger conformity sets; the smallest known β̃ is therefore the right choice in practice.
The method is model-agnostic and post-hoc: it needs only the trained model and a performance number such as accuracy or median absolute error.
A leave-one-out procedure over an unlabeled test set gives a conformal set for every test observation, which is what makes the approach practical when no labeled data exist.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural hardening of the approach is to replace the point estimate of accuracy by a lower confidence bound; then the coverage guarantee holds with the confidence level folded into β, hedging the estimation uncertainty the paper flags in its discussion.
For well-calibrated classifiers, the score dominance in Lemma 1 may hold with β substantially smaller than the misclassification rate, so a refined exactness measure could yield tighter empirical sets than accuracy alone suggests.
The exactness framework could be adapted to covariate shift by letting β vary with the shift magnitude; the quantile-dominance argument would carry through if a shift-dependent exactness bound can be estimated.
A simulation across accuracy levels (say 0.6, 0.8, 0.95) would show the bound is approximately tight when exactness is exact, and would quantify how sensitive empirical coverage is to underestimating β.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: the central coverage guarantee is true, but the proof as written is not, and the numerical example looks off. The paper is still worth a serious referee because the gap it addresses is real and the fix is simple.

What's new: the (β̃,β)-exactness framework and the coverage bound 1−α−β for conformal prediction using only unlabeled calibration data. The idea of substituting predicted labels is natural, but I don't know prior work that states this specific guarantee. The classification case reduces cleanly to accuracy, and the bound is honest about the price you pay. The paper also clearly flags the main practical caveat in Section 5: exactness is estimated, not known, which adds uncertainty to the guarantee.

The soft spots, in order. Lemma 2 is false; the stress-test counterexample works. The quantile of U can be below the quantile of V with probability larger than β even when P(U≥V)≥1−β, because the index used is data-dependent. The good news is that Theorem 2 survives: a direct union bound gives miscoverage ≤ P(s_test > ŝ_test) + P(ŝ_test > q̂) ≤ β + α. So the result stands, but the proof needs to be rewritten. The numerical illustration also looks wrong: with accuracy 88.7%, the 95% quantile of true scores can't be 0.174 unless the error cases are bizarre; q̂=0.575 is more plausible. The authors should recheck that example. The regression case is also weaker than the classification case: the (β̃,β)-exactness assumption essentially says you know the error ball radius, which makes the 'unlabeled' part less impressive there. And the references miss the prediction-powered inference literature and conformal prediction with noisy labels; both are relevant and should be cited.

Who this is for: researchers in uncertainty quantification who need conformal guarantees in privacy-sensitive or label-scarce deployment. They'll get a usable baseline result, but they should treat the current proof as a sketch. I'd send it to peer review, with a request for a corrected proof and a cleaned-up example.

Referee Report

3 major / 4 minor

Summary. The paper proposes an extension of split conformal prediction for settings where labeled calibration data are unavailable. The authors define a notion of (β̃,β)-exactness for a model, construct pseudo-scores by replacing true calibration labels with predicted labels chosen from a score ball (Eqs. (7)–(9)), and claim that the usual conformal quantile applied to these pseudo-scores yields coverage at least 1−α−β for both classification and regression. The classification result is a correct and reasonably simple observation: under an accuracy lower bound, the cost of using predicted instead of true labels is an additive β penalty. The regression construction, however, degenerates to a fixed ball whose radius is the exactness parameter, so the conformal step is vacuous. The written proof of the main theorem relies on a false lemma, although the theorem can be repaired by a short union-bound argument. The numerical illustration in Section 4 contains an internal inconsistency.

Significance. If properly repaired, the classification result is a useful, if modest, contribution: it gives a clean statement that replacing true calibration labels by model predictions incurs only an additive coverage penalty equal to the model's error rate, provided that error rate is known. The argument is transparent and the paper honestly acknowledges in Section 5 that the exactness parameters are not known in practice and must be estimated. The regression claim is much weaker: the construction yields the fixed β̃-ball and the conformal quantile plays no role. The numerical example needs correction. Overall, the paper is clearly written and the central classification theorem is defensible after revision, but the current proof is invalid and the regression framing overstates the novelty.

major comments (3)

[Section 3.2, Eqs. (7)–(9)] Lemma 2 is false. Let n=2, α=0.5, U≡0, and V=1 with probability 0.4 and V=0 with probability 0.6. Then P(U≥V)=0.6≥1−β for β=0.4, but with the empirical quantile equal to the maximum, P(q̂_U≥q̂_V)=P(V1=V2=0)=0.36<0.6. The proof's flaw is that the index j in the claim is data-dependent; conditioning on its selection invalidates the step 'P(U_j≥V_j)≥1−β'. Since Theorem 2's proof uses Lemma 2 to assert P(B)≥1−β, the written proof is invalid. The theorem itself is salvageable: the union bound {s_test>q̂}⊆{s_test>ŝ_test}∪{ŝ_test>q̂} gives P(miscoverage)≤β+α by Lemma 1 and the standard conformal guarantee applied to the i.i.d. pseudo-scores. Please replace Lemma 2 and revise the proof of Theorem 2 accordingly.
[Section 3.2, Eqs. (7)–(9)] In the regression case with s(X,Y)=d(Y,fhat(X)), the set T_β̃(X) is the metric ball of radius β̃ around fhat(X). Any maximizer Ŷ of s over this ball satisfies d(Ŷ,fhat(X))=β̃, so every pseudo-score ŝ_k equals β̃ deterministically. Hence q̂=β̃ and C(X_test) is the fixed ball {y:d(y,fhat(X_test))≤β̃}. The conformal step is vacuous and the coverage statement is exactly the (β̃,β)-exactness assumption. The paper should state this explicitly; as written, the regression claim overstates the role of conformal prediction.
[Section 4] The numerical illustration is internally inconsistent. With an estimated accuracy of 88.7% on 1000 calibration points, about 113 points are misclassified. For a probability-based classifier, a misclassified point has true-class probability at most 1/2, hence score at least 1/2. Therefore at least 11.3% of the calibration scores exceed 0.5, so the 95% quantile q must exceed 0.5. The reported q=0.174 is incompatible with this accuracy. Please re-examine the calculation; the discrepancy suggests either the accuracy estimate or the reported quantiles are miscalculated.

minor comments (4)

[Section 2] The definition of q as the '⌈(n+1)(1−α)⌉/n-quantile' is nonstandard. It would be clearer to define q as the ⌈(n+1)(1−α)⌉-th smallest score, as is conventional in split conformal prediction.
[Theorem 2] The notation for C(X_test) as a subset of (I×T)^n×T is confusing. Since q̂ depends only on X_1,...,X_n, the conformity set can be written simply as C(X_test)={y:s(X_test,y)≤q̂(X_1,...,X_n)}, and the event can be defined without carrying the calibration labels.
[Section 5] The paper correctly notes that β̃ and β are not known in practice and must be estimated. This limitation should also be stated in the abstract or conclusion: the coverage guarantee is conditional on known exactness parameters; otherwise practitioners may overstate the finite-sample guarantee.
[Throughout] Minor typos: 'Without lose of generality' (Lemma 2), 'oﬀicial' (Section 4), 'suﬀicient' (Section 5). Also, the phrase 'if at least in ⌈(n+1)α⌉ of the n calibration data' in Remark 2 is awkward and should be rephrased.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the coverage bound follows from the exactness assumption plus standard conformal prediction on pseudo-scores; the regression special case is a trivial corollary but not a circular derivation.

full rationale

The paper's central claim is conditional: if the model is (β̃,β)-exact, then the conformity set built from unlabeled calibration data covers the true label with probability at least 1−α−β. The derivation chain is: (i) by exactness, with probability ≥1−β the true score s(X,Y) is no larger than the estimated score ŝ(X) = s(X,Ŷ) (Lemma 1); (ii) the estimated scores ŝ_1,...,ŝ_n,ŝ_test are i.i.d. because they are functions only of the i.i.d. inputs X_k; (iii) standard conformal prediction (Theorem 1, cited to external work) gives P(ŝ_test ≤ q̂) ≥ 1−α. A union bound then gives P(s_test ≤ q̂) ≥ 1−α−β. The theorem is therefore not obtained by assuming its conclusion; it is a consequence of an independent quantile guarantee on a constructed score plus the exactness input. No load-bearing self-citation occurs: Theorem 1 is cited to Angelopoulos and Bates and Papadopoulos et al., not to the authors' own work. In the regression special case the construction collapses: all estimated scores equal β̃, so q̂=β̃ and the conformity set reduces to the error ball appearing in Definition 1; the coverage guarantee then follows immediately from the exactness definition, weakening 1−β to 1−α−β. This is a lack of novelty in that special case, not a circular derivation, because the conformity set is not defined as the exactness ball—it degenerates to it. Section 5 honestly states that β̃ and β are not known in practice and must be estimated, introducing another source of uncertainty; this is a limitation, not a circular step. The written proof contains a defective Lemma 2, but the theorem itself is independently provable as above, so the proof defect is a correctness issue, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The main assumptions are the standard conformal exchangeability, the exactness definition (which is essentially the model accuracy), and the implicit need to know β̃ and β exactly rather than estimating them.

free parameters (2)

β (classification accuracy gap) = 0.113 (from estimated accuracy 88.7% in Section 4)
The coverage guarantee requires the model's accuracy or error rate as an input. In practice this is estimated from a labeled subset, introducing estimation uncertainty not captured by the theorem.
β̃ (score ball radius, regression) = median absolute error (estimated)
For regression, β̃ is the median absolute error or another quantile of the prediction error, which must be estimated from data. The paper's Corollary 2 uses MedAE as β̃.

assumptions (3)

standard math Standard conformal coverage guarantee (Theorem 1) for i.i.d. calibration scores
Invoked in the proof of Theorem 2 and implicitly needed for the pseudo-score exchangeability argument that yields P(ŝ_test ≤ q̂) ≥ 1-α.
ad hoc to paper The model f̂ is (β̃,β)-exact with known β̃ and β
The coverage bound depends entirely on this assumption. The paper states it in Definition 1 and uses it in Theorem 2. In practice β̃ and β are estimated, not known.
domain assumption Calibration data (X_k,Y_k) and test point (X_test,Y_test) are i.i.d. and independent of the trained model
Standard in conformal prediction; stated in Theorem 2. The pseudo-score exchangeability relies on the X_k being i.i.d. after conditioning on the fixed model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conformal prediction without knowledge of labeled calibration data." pith.science (2026). https://pith.science/paper/NDJO6U3Y

@misc{pith2026250910321,
  author       = {Pith},
  title        = {Pith review of: Conformal prediction without knowledge of labeled calibration data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NDJO6U3Y}},
  note         = {Machine review of arXiv:2509.10321}
}

abstract

We extend the method of conformal prediction beyond the case relying on labeled calibration data. Replacing the calibration scores by suitable estimates, we identify conformity sets $C$ for classification and regression models that rely on unlabeled calibration data. Given a classification model with accuracy $1-\beta$, we prove that the conformity sets guarantee a coverage of $P(Y \in C) \geq 1-\alpha-\beta$ for an arbitrary parameter $\alpha \in (0,1)$. The same coverage guarantee also holds for regression models, if we replace the accuracy by a similar exactness measure. Finally, we describe how to use the theoretical results in practice.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 4 linked inside Pith

[1]

Angelopoulos, Anastasios N., Rina Foygel Barber, and Stephen Bates. 2025. ``Theoretical Foundations of Conformal Prediction.'' https://arxiv.org/abs/2411.11824

arXiv 2025
[2]

Angelopoulos, Anastasios N., and Stephen Bates. 2022. ``A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.'' https://arxiv.org/abs/2107.07511

arXiv 2022
[3]

Arif, Majid. 2018. `` weight and height.csv .'' https://www.kaggle.com/datasets/majidarif17/weight-and-heightcsv/data

2018
[4]

Gibbs, Isaac, and Emmanuel Candès. 2021. ``Adaptive Conformal Inference Under Distribution Shift.'' https://arxiv.org/abs/2106.00170

arXiv 2021
[5]

Guan, Leying. 2022. ``Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction.'' Biometrika 110 (1): 33--50. https://doi.org/10.1093/biomet/asac040

doi:10.1093/biomet/asac040 2022
[6]

Molnar, Christoph. 2023. Introduction to Conformal Prediction with Python. https://christophmolnar.com/books/conformal-prediction

2023
[7]

Papadopoulos, Harris, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. 2002. ``Inductive Confidence Machines for Regression.'' In Machine Learning: ECML 2002, 345--56. Berlin, Heidelberg: Springer Berlin Heidelberg

2002
[8]

R Core Team. 2024. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/

2024

Show all 13 references

[9]

Candes, and Aaditya Ramdas

Tibshirani, Ryan J., Rina Foygel Barber, Emmanuel J. Candes, and Aaditya Ramdas. 2020. ``Conformal Prediction Under Covariate Shift.'' https://arxiv.org/abs/1904.06019

2020 arXiv
[10]

Vovk, Vladimir. 2015. ``Cross-Conformal Predictors.'' Annals of Mathematics and Artificial Intelligence 74: 9--28. https://doi.org/10.1007/s10472-013-9368-4

2015 doi
[11]

Vovk, Vladimir, Alexander Gammerman, and Craig Saunders. 1999. ``Machine-Learning Applications of Algorithmic Randomness.'' In Proceedings of the Sixteenth International Conference on Machine Learning, 444--53. Morgan Kaufmann

1999
[12]

Vovk, Vladimir, Ilia Nouretdinov, Valery Manokhin, and Alexander Gammerman. 2018. ``Cross-Conformal Predictive Distributions.'' In Proceedings of the Seventh Workshop on Conformal and Probabilistic Prediction and Applications, 91:37--51. Proceedings of Machine Learning Researc...

2018
[13]

V.fߜ h-J1> ,aK٫§6ǩ ezsC 86ZيyuHzF9(7YR67>MNW+| ?dzo)98?F ;; (Ǟ'uxǳBYR67>M D.Ιs76ǩ ez | 9xVo롎 m 8\ 6 > L^& g; gT ޿:MCWh l g =< s|Y j #_e ۨܥGqEٷ 3\ J7 v |

World Health Organisation. 2000. ``Obesity: Preventing and Managing the Global Epidemic. Report of a WHO Consultation.'' World Health Organization Technical Report Series 894: i--253. CSLReferences document fig-data-1.pdf0000664000000000000000000130211115061031551012057 0ustar...

2000

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Angelopoulos, Anastasios N., Rina Foygel Barber, and Stephen Bates. 2025. ``Theoretical Foundations of Conformal Prediction.'' https://arxiv.org/abs/2411.11824

arXiv 2025

[2] [2]

Angelopoulos, Anastasios N., and Stephen Bates. 2022. ``A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.'' https://arxiv.org/abs/2107.07511

arXiv 2022

[3] [3]

Arif, Majid. 2018. `` weight and height.csv .'' https://www.kaggle.com/datasets/majidarif17/weight-and-heightcsv/data

2018

[4] [4]

Gibbs, Isaac, and Emmanuel Candès. 2021. ``Adaptive Conformal Inference Under Distribution Shift.'' https://arxiv.org/abs/2106.00170

arXiv 2021

[5] [5]

Guan, Leying. 2022. ``Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction.'' Biometrika 110 (1): 33--50. https://doi.org/10.1093/biomet/asac040

doi:10.1093/biomet/asac040 2022

[6] [6]

Molnar, Christoph. 2023. Introduction to Conformal Prediction with Python. https://christophmolnar.com/books/conformal-prediction

2023

[7] [7]

Papadopoulos, Harris, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. 2002. ``Inductive Confidence Machines for Regression.'' In Machine Learning: ECML 2002, 345--56. Berlin, Heidelberg: Springer Berlin Heidelberg

2002

[8] [8]

R Core Team. 2024. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/

2024