pith. sign in

arxiv: 2605.18354 · v1 · pith:YEUXMRUOnew · submitted 2026-05-18 · 💻 cs.LG

Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration

Fanyi Wu , Lihua Niu , Samuel Kaski , Michele Caprio This is my paper

Pith reviewed 2026-05-20 11:54 UTC · model grok-4.3

classification 💻 cs.LG
keywords conformal predictionprediction setsdecoupled optimizationmarginal coveragesplit conformalefficiency calibrationrisk control
0
0 comments X

The pith

Decoupling tuning and calibration splits in conformal optimization produces efficient prediction sets with finite-sample marginal coverage.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes Decoupled Conformal Optimisation to separate the search for efficient prediction-set structures from the step that certifies coverage. An independent tuning split selects the structure while a fresh calibration split computes the conformal quantile. Conditional on the chosen structure, standard split-conformal exchangeability then supplies finite-sample marginal coverage for any candidate class without a confidence parameter or multiple-testing correction. This approach targets marginal coverage rather than high-probability risk control, yet the two converge to the same population threshold under consistency assumptions. Readers would care because the method often shrinks average set sizes or interval widths on benchmarks while still tracking the nominal coverage level.

Core claim

Conditional on the tuned structure, standard split-conformal exchangeability yields finite-sample marginal coverage for any candidate class, without a confidence parameter or multiple-testing correction. DCO therefore targets a different finite-sample guarantee from PAC-style methods: marginal conformal coverage rather than high-probability risk control. Under consistency assumptions on the coupled risk bound, the two approaches nevertheless converge to the same population threshold.

What carries the argument

The train-tune-calibrate design principle that assigns an independent tuning split to efficiency-oriented structural selection and a separate calibration split to the final conformal quantile.

If this is right

  • Standard split-conformal exchangeability applies directly after tuning to deliver the coverage guarantee for any chosen prediction-set class.
  • DCO tracks the nominal coverage level closely across classification and regression tasks.
  • Average prediction-set size or interval width decreases relative to PAC-style calibration on benchmarks such as ImageNet-A, CIFAR-100, Diabetes, California Housing, and Concrete.
  • The decoupled and coupled approaches converge to the same population threshold under consistency assumptions on the risk bound.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The separation may allow more elaborate efficiency optimizations inside conformal pipelines without triggering extra correction factors.
  • Direct comparisons between DCO and other efficiency-focused conformal variants could expose regime-specific trade-offs.
  • Extending the method to streaming or dependent-data settings would first require verifying that split independence can still be maintained.

Load-bearing premise

The tuning split and calibration split are drawn independently so that structure selection on the tuning data leaves exchangeability intact for the coverage guarantee on the calibration data.

What would settle it

An experiment that draws the tuning and calibration splits dependently or from overlapping data and checks whether empirical coverage on fresh test points falls materially below the nominal level.

Figures

Figures reproduced from arXiv: 2605.18354 by Fanyi Wu, Lihua Niu, Michele Caprio, Samuel Kaski.

Figure 1
Figure 1. Figure 1: Coupled calibration versus DCO-Warmstart. In CRC/BQ-style calibration, the same [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Coverage (a) and interval width (b) on the Diabetes dataset over 50 random splits [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Classification results on ImageNet-A over 50 random splits. (a) Mean prediction set size. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of test coverage (a) and prediction set size (b) on ImageNet-A across 50 [PITH_FULL_IMAGE:figures/full_fig_p031_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Coverage (a) and average interval width (b) for Split CP and BQ on the Diabetes dataset. [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Coverage (a) and average interval width (b) under prior scales [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
read the original abstract

Bayesian conformal optimisation methods often use the same held-out data both to search for efficient prediction sets and to certify coverage or risk. This coupling is natural for high-probability risk-control guarantees, but it is not necessary when the target is standard finite-sample marginal conformal coverage. We propose Decoupled Conformal Optimisation (DCO), a train-tune-calibrate design principle that uses an independent tuning split for efficiency-oriented structural selection and a fresh calibration split for the final conformal quantile. Conditional on the tuned structure, standard split-conformal exchangeability yields finite-sample marginal coverage for any candidate class, without a confidence parameter or multiple-testing correction. DCO therefore targets a different finite-sample guarantee from PAC-style methods: marginal conformal coverage rather than high-probability risk control. Under consistency assumptions on the coupled risk bound, the two approaches nevertheless converge to the same population threshold. Across classification and regression benchmarks, including ImageNet-A, CIFAR-100, Diabetes, California Housing, and Concrete, DCO tracks the nominal coverage level closely while often reducing average prediction-set size or interval width relative to PAC-style calibration. On ImageNet-A, for example, the average set size decreases from $26.52$ to $25.26$ and the 95th-percentile set size from $58.95$ to $53.73$; on Diabetes, the average interval width decreases from $2.098$ to $1.914$.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The manuscript proposes Decoupled Conformal Optimisation (DCO), a train-tune-calibrate protocol that selects prediction-set structures (e.g., nonconformity scores or candidate families) on an independent tuning split and then computes the conformal quantile on a fresh calibration split. Conditional on the chosen structure, standard split-conformal exchangeability yields finite-sample marginal coverage at the nominal level without multiple-testing corrections or extra confidence parameters. The paper contrasts this marginal guarantee with PAC-style high-probability risk control, notes convergence to the same population threshold under consistency assumptions, and reports empirical results on ImageNet-A, CIFAR-100, Diabetes, California Housing, and Concrete showing nominal coverage tracking together with reductions in average set size or interval width (e.g., ImageNet-A average set size 26.52 to 25.26).

Significance. If the independence of the tuning and calibration splits is maintained, DCO supplies a simple, theoretically grounded route to efficiency-oriented structural selection while retaining the exact finite-sample marginal coverage property of classical split conformal prediction. The distinction between marginal coverage and high-probability risk control is clearly drawn, and the empirical size reductions on standard benchmarks illustrate practical benefit. The derivation re-uses existing exchangeability arguments rather than introducing new axioms, which strengthens the contribution.

minor comments (3)
  1. [§3] §3 (method): include an explicit pseudocode or diagram of the train-tune-calibrate splitting protocol so that readers can verify the required independence of the tuning and calibration sets.
  2. [empirical section] Table 1 / empirical section: report standard errors or results over multiple random splits for the size and coverage metrics; single-run point estimates make it harder to judge the stability of the observed reductions.
  3. [§4] Abstract and §4: the consistency assumption under which DCO and PAC-style thresholds converge should be stated more formally (e.g., as a limit on the risk estimator) rather than left at the level of informal discussion.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the accurate and positive summary of our manuscript, including the clear distinction drawn between marginal conformal coverage and PAC-style high-probability risk control. We appreciate the recommendation for minor revision and note that the empirical improvements on benchmarks such as ImageNet-A are correctly highlighted.

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper's central finite-sample marginal coverage guarantee is obtained by applying standard split-conformal exchangeability to a fresh calibration split whose scores remain exchangeable with the test point once the structure has been fixed on an independent tuning split. This follows directly from the classical theory of split conformal prediction under i.i.d. sampling and does not reduce to any quantity fitted or defined on the tuning data; the decoupling is an explicit protocol choice rather than a self-referential construction. No load-bearing self-citations, ansatzes, or renamings of known results appear in the derivation chain.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The claim rests on the standard exchangeability assumption of conformal prediction and user-chosen data splits; no new entities are introduced.

free parameters (1)
  • data split proportions
    The division of available data into training, tuning, and calibration portions is a modeling choice left to the practitioner.
axioms (1)
  • domain assumption Calibration data points are exchangeable with test points
    Invoked to obtain the finite-sample marginal coverage guarantee after structure selection.

pith-pipeline@v0.9.0 · 5796 in / 1130 out tokens · 42339 ms · 2026-05-20T11:54:39.787441+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages

  1. [1]

    Vovk, Vladimir and Gammerman, Alex and Shafer, Glenn , title =

    work page

  2. [2]

    Journal of Machine Learning Research , volume =

    Shafer, Glenn and Vovk, Vladimir , title =. Journal of Machine Learning Research , volume =. 2008 , url=

    work page 2008

  3. [3]

    and Wasserman, Larry , title =

    Lei, Jing and G'Sell, Max and Rinaldo, Alessandro and Tibshirani, Ryan J. and Wasserman, Larry , title =. Journal of the American Statistical Association , volume =

    work page

  4. [4]

    2022 , eprint=

    A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification , author=. 2022 , eprint=

    work page 2022

  5. [5]

    and Bates, Stephen and Malik, Jitendra and Jordan, Michael I

    Angelopoulos, Anastasios N. and Bates, Stephen and Malik, Jitendra and Jordan, Michael I. , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  6. [6]

    and Lei, Lihua and Malik, Jitendra and Jordan, Michael I

    Bates, Stephen and Angelopoulos, Anastasios N. and Lei, Lihua and Malik, Jitendra and Jordan, Michael I. , title =. Journal of the ACM , volume =

    work page

  7. [7]

    Angelopoulos and Stephen Bates and Emmanuel J

    Anastasios N. Angelopoulos and Stephen Bates and Emmanuel J. Cand. The Annals of Applied Statistics , number =. 2025 , doi =

    work page 2025

  8. [8]

    , title =

    Fong, Edwin and Holmes, Chris C. , title =. Advances in Neural Information Processing Systems (NeurIPS) , volume =. 2021 , url=

    work page 2021

  9. [9]

    and Cemgil, A

    Stutz, David and Dvijotham, Krishnamurthy D. and Cemgil, A. Taylan and Doucet, Arnaud , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  10. [10]

    2025 , eprint=

    Conformal Risk Control , author=. 2025 , eprint=

    work page 2025

  11. [11]

    International Conference on Machine Learning (ICML) , year=

    Conformal Prediction as Bayesian Quadrature , author=. International Conference on Machine Learning (ICML) , year=

    work page

  12. [12]

    2026 , eprint=

    Bayesian Conformal Prediction as a Decision Risk Problem , author=. 2026 , eprint=

    work page 2026

  13. [13]

    The Annals of Mathematical Statistics , volume =

    Dvoretzky, Aryeh and Kiefer, Jack and Wolfowitz, Jacob , title =. The Annals of Mathematical Statistics , volume =

    work page

  14. [14]

    2013 , note =

    Jansen, Laurens , title =. 2013 , note =

    work page 2013

  15. [15]

    2021 , eprint=

    Conformal Bayesian Computation , author=. 2021 , eprint=

    work page 2021

  16. [16]

    Biometrika , volume=

    Fast exact conformalization of the lasso using piecewise linear homotopy , author=. Biometrika , volume=. 2019 , publisher=

    work page 2019

  17. [17]

    Annals of Statistics , volume=

    Least angle regression , author=. Annals of Statistics , volume=. 2004 , publisher=

    work page 2004

  18. [18]

    Proceedings of the National Academy of Sciences , volume=

    Multisurface method of pattern separation for medical diagnosis applied to breast cytology , author=. Proceedings of the National Academy of Sciences , volume=. 1990 , publisher=

    work page 1990

  19. [19]

    The Annals of Statistics , volume=

    Predictive inference with the jackknife+ , author=. The Annals of Statistics , volume=. 2021 , publisher=

    work page 2021

  20. [20]

    Far East Journal of Mathematical Sciences , volume=

    The concept of exchangeability and its applications , author=. Far East Journal of Mathematical Sciences , volume=

    work page

  21. [21]

    2009 , publisher=

    Bayesian theory , author=. 2009 , publisher=

    work page 2009

  22. [22]

    2018 , howpublished=

    JAX: composable transformations of Python+NumPy programs , author=. 2018 , howpublished=

    work page 2018

  23. [23]

    Conference on Learning Theory , pages=

    Efficiency of conformalized ridge regression , author=. Conference on Learning Theory , pages=. 2014 , organization=

    work page 2014

  24. [24]

    arXiv preprint arXiv:2103.09763 , year=

    Conformalized survival analysis , author=. arXiv preprint arXiv:2103.09763 , year=

    work page arXiv

  25. [25]

    Transactions on Machine Learning Research , issn=

    Conformalized Credal Regions for Classification with Ambiguous Ground Truth , author=. Transactions on Machine Learning Research , issn=. 2025 , url=

    work page 2025

  26. [26]

    2025 , eprint=

    The Joys of Categorical Conformal Prediction , author=. 2025 , eprint=

    work page 2025

  27. [27]

    Journal of Statistical Software , volume=

    Stan: a probabilistic programming language , author=. Journal of Statistical Software , volume=

    work page

  28. [28]

    Transactions on Machine Learning Research , issn=

    Credal Bayesian Deep Learning , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    work page 2024

  29. [29]

    Advances in neural information processing systems , volume=

    Simple and scalable predictive uncertainty estimation using deep ensembles , author=. Advances in neural information processing systems , volume=

    work page

  30. [30]

    international conference on machine learning , pages=

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning , author=. international conference on machine learning , pages=. 2016 , organization=

    work page 2016

  31. [31]

    Advances in neural information processing systems , volume=

    Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift , author=. Advances in neural information processing systems , volume=

    work page

  32. [32]

    Artificial Intelligence Review , volume=

    A review of predictive uncertainty estimation with machine learning , author=. Artificial Intelligence Review , volume=. 2024 , publisher=

    work page 2024

  33. [33]

    2022 , issn =

    Valid inferential models for prediction in supervised learning problems , journal =. 2022 , issn =. doi:https://doi.org/10.1016/j.ijar.2022.08.001 , url =

    work page doi:10.1016/j.ijar.2022.08.001 2022

  34. [34]

    Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=

    Conformal prediction with conditional guarantees , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , pages=. 2025 , publisher=

    work page 2025

  35. [35]

    Rina Foygel Barber and Emmanuel J. Cand. The Annals of Statistics , number =. 2023 , doi =

    work page 2023

  36. [36]

    , title =

    Jansen, L. , title =. 2013 , note =

    work page 2013

  37. [37]

    arXiv preprint arXiv:2511.05746v1 , year =

    Conformalized Bayesian Inference, with Applications to Random Partition Models , author =. arXiv preprint arXiv:2511.05746v1 , year =

    work page arXiv

  38. [38]

    2009 , publisher =

    A First Course in Bayesian Statistical Methods , author =. 2009 , publisher =

    work page 2009

  39. [39]

    2023 , eprint=

    Dirichlet-based Uncertainty Quantification for Personalized Federated Learning with Improved Posterior Networks , author=. 2023 , eprint=

    work page 2023

  40. [40]

    Journal of Statistical Planning and Inference , volume =

    Bayes--Hermite Quadrature , author =. Journal of Statistical Planning and Inference , volume =

    work page

  41. [41]

    URL https://doi.org/10.1080/ 01621459.2017.1307116

    Jing Lei and Max G’Sell and Alessandro Rinaldo and Ryan J. Tibshirani and Larry Wasserman , title =. Journal of the American Statistical Association , volume =. 2018 , publisher =. doi:10.1080/01621459.2017.1307116 , URL =

    work page doi:10.1080/01621459.2017.1307116 2018

  42. [42]

    Valiant, L. G. , title =. 1984 , issue_date =. doi:10.1145/1968.1972 , journal =

    work page doi:10.1145/1968.1972 1984

  43. [43]

    CoRR , volume =

    Anthony Bellotti , title =. CoRR , volume =. 2021 , url =. 2105.11255 , timestamp =

    work page arXiv 2021

  44. [44]

    2024 , eprint=

    On the Expected Size of Conformal Prediction Sets , author=. 2024 , eprint=

    work page 2024

  45. [45]

    URL https://doi.org/10.1080/ 01621459.2017.1307116

    Sadinle, Mauricio and Lei, Jing and Wasserman, Larry , year=. Least Ambiguous Set-Valued Classifiers With Bounded Error Levels , volume=. Journal of the American Statistical Association , publisher=. doi:10.1080/01621459.2017.1395341 , number=

    work page doi:10.1080/01621459.2017.1395341 2017

  46. [46]

    2026 , eprint=

    Optimal Decision-Making Based on Prediction Sets , author=. 2026 , eprint=

    work page 2026