Pith. sign in

REVIEW 3 major objections 5 minor 77 references

Conformal Prediction for Zero-Shot Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that CLIP's zero-shot conformal prediction sets can be made up to 20% smaller, with the coverage guarantee intact, by re-weighting the pooled calibration and query logits through a fast entropic optimal transport step.

desk verdict Conf-OT's coverage guarantee is asserted, not proven, and using calibration labels to set the OT marginal breaks exchangeability; the empirical study is strong but the central claim fails. read the letter →

arxiv 2505.24693 v1 pith:PMMJWW5Z submitted 2025-05-30 cs.CV

Julio Silva-Rodríguez , Ismail Ben Ayed , Jose Dolz This is my paper
classification cs.CV
keywords conformalpredictionzero-shotclassificationvision-languagemodelsCLIPoptimaltransportSinkhornalgorithmtransductivelearningsetefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to make conformal prediction sets small and useful when the underlying model is a zero-shot vision-language model such as CLIP, whose pre-training domain differs from the target task. Split conformal prediction guarantees coverage for any black-box predictor, but that domain drift inflates the prediction sets. The paper claims the drift can be reduced inside the conformal pipeline itself, with no extra labeled data and no training: solve an entropic optimal transport problem on the pooled calibration and query similarities, using the calibration labels only to fix the class proportions, and run the non-conformity scores on the transported codes. Across 15 datasets and the LAC, APS, and RAPS scores, the authors report relative set-size reductions of up to about 20%, improved class-conditional coverage, and a small accuracy gain, at an added cost of well under a second on the largest datasets.

What carries the argument

The load-bearing object is the entropic optimal transport problem $$\max_{Q \in \mathcal{Q}} \operatorname{tr}(Q^\top S) + \varepsilon H(Q)$$ over the combined calibration-plus-query similarity matrix $S \in \mathbb{R}^{K \times (N+M)}$, where $Q$ is constrained to the transportation polytope: row sums equal the calibration label marginal $m$, and column sums equal a uniform sample marginal. The mechanism that carries the argument is the Sinkhorn–Knopp iteration, which alternates row and column renormalization of the softmax-scaled similarity matrix and is run for three iterations (Algorithm 1). The resulting column-normalized codes are the new non-conformity inputs; computing scores on these transported codes instead of the raw zero-shot probabilities is what shifts the classifier toward the target domain while keeping calibration and query scores on the same jointly transformed scale.

What would settle it

Build a task with a known label shift between calibration and query: calibrate on a balanced subset, query on a subset where one class is several times more frequent, run Conf-OT with the calibration-derived marginal $m$, and measure empirical coverage over many re-splits at $\alpha = 0.10$. Coverage falling systematically below $1-\alpha$, or changing when the true query marginal replaces $m$, would show that the claimed preservation of exchangeability, and with it the validity of the coverage guarantee, fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the domain gap between a pre-trained vision-language model and a downstream zero-shot task can be corrected inside the conformal prediction pipeline, using only the calibration and query data already in hand. Concretely, Conf-OT solves an entropic optimal transport problem on the joint similarity matrix of calibration and query samples, producing a matrix of soft codes whose columns are class assignments and whose row marginal is fixed to the label distribution observed in the calibration set. The codes are then treated exactly like raw probabilities: any non-conformity score (LAC, APS, or RAPS) is computed on them, the usual split conformal threshold is found from calibration, and prediction sets are built for the queries. On 15 datasets, three scores, and several CLIP and MetaCLIP backbones, the authors report that this consistently shrinks average set size by up to roughly 20% relative to the untransported zero-shot scores, improves class-conditional coverage, and modestly raises top-1 accuracy, while the Sinkhorn step adds about 0.6–0.75 seconds of runtime on the largest datasets.

Load-bearing premise

The load-bearing premise is that the transport re-weighting, which uses the calibration labels to fix the class proportions $m$, leaves calibration and query non-conformity scores exchangeable enough for the split conformal coverage guarantee to hold; the paper asserts this in Section 4.1 and Algorithm 1 without a proof, and the dependence on calibration labels is exactly the kind of asymmetry that could break exchangeability.

Editorial extensions

If this is right

  • Applied on top of LAC, APS, or RAPS, Conf-OT reduces average prediction set size by roughly 15–20% relative to base zero-shot scores while empirical coverage stays at the nominal level across 15 datasets.
  • Gains persist across backbones (CLIP ResNet-50/101, ViT-B/32, ViT-B/16, ViT-L/14; MetaCLIP ViT-B/16, ViT-H/14) and at both error rates $\alpha = 0.10$ and $\alpha = 0.05$.
  • Efficiency gains survive scarce calibration (10% of data) and tiny query batches (8–32 images), so the transductive computation does not require large test batches.
  • The calibration label marginal is an active ingredient: replacing it with a uniform marginal shrinks the benefits, and top-1 accuracy rises about 2.6–2.9 points alongside the set-size reduction.
  • Conf-OT adds only about 0.6–0.75 seconds on the largest datasets, roughly 15 times less runtime than the TIM and TransCLIP baselines, while delivering smaller or equal sets and better coverage behavior than those baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because Conf-OT operates only on black-box class-similarity vectors and is training-free, it should transfer to other foundation-model families that emit per-class scores, not just contrastive vision-language models.
  • The ablations show the transport step is complementary to temperature scaling, which suggests stacking Conf-OT with other score post-processings, including conformal training objectives defined on the transported codes, could compound the efficiency gains.
  • The paper does not stress-test label shift between calibration and query; since the marginal $m$ is estimated from calibration labels, a natural prediction is that Conf-OT's efficiency and coverage degrade when the query class frequencies differ from calibration, and a re-split experiment with a deliberately skewed query subset would settle this.
  • The size of the reported gains (near-20% set shrinkage with a 2–3 point accuracy rise) suggests most of the zero-shot inefficiency in conformal sets comes from systematic, transportable logit bias rather than per-sample noise, so similar reductions should be obtainable by any calibration-aware re-weighting of the score distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies split conformal prediction for zero-shot vision-language models (CLIP). It argues that the domain gap between CLIP's pre-training and downstream tasks degrades conformal set efficiency, and that adapting the classifier on the calibration set would break exchangeability and hence destroy coverage guarantees. As a remedy, the authors propose Conf-OT, a transductive optimal transport procedure applied to the combined calibration and query logit matrix. The OT problem uses a label-marginal constraint estimated from the calibration labels, is solved with a few Sinkhorn iterations, and the resulting soft codes are then fed into a standard split conformal pipeline using LAC, APS, or RAPS. The paper claims that Conf-OT is unsupervised, maintains coverage guarantees, improves set efficiency by up to 20%, and is about 15 times faster than existing transductive baselines. The empirical study covers 15 datasets, 7 backbones, and 3 non-conformity scores, with ablations on data ratios, query batch sizes, temperature scaling, and runtime.

Significance. If the coverage guarantee claim were established, Conf-OT would be a practically valuable contribution: it is training-free, computationally cheap, and the experimental evaluation is unusually broad, spanning multiple CLIP and MetaCLIP backbones, 15 datasets, three non-conformity scores, 20 seeds, and detailed ablations. The code is promised. The paper also correctly identifies a real gap in the literature, namely that most conformal prediction work for vision uses in-distribution models whereas foundation models are transferred across domains. However, the central theoretical claim—that Conf-OT maintains the split conformal coverage guarantee—is not proven, and the algorithm as stated uses the calibration labels in a way that breaks the standard exchangeability argument. Since the paper's headline contribution is precisely 'maintaining coverage guarantees' while improving efficiency, this issue is load-bearing and cannot be waved away by empirical coverage rates.

major comments (3)
  1. [Sec. 4.2, Eq. (6); Algorithm 1, Steps 3 and 11] The coverage guarantee in Eq. (2) is asserted but not proven for Conf-OT. Step 3 of Algorithm 1 sets the label marginal as m = (1/N) sum_i y_i^ohe, using the calibration labels. The Sinkhorn solution Q* in Eq. (8) therefore depends on the calibration labels through the row marginal m. The subsequent conformal scores are computed from Q*, so the calibration and test scores are not exchangeable under permutations of the combined calibration/test indices: permuting the label vector changes m and changes the transformation applied to all scores. Split conformal validity requires the nonconformity scores of calibration and test points to be exchangeable, and no theorem or argument in the paper establishes that this symmetry survives the label-dependent reweighting. The paper's own Fig. 1(b) and Sec. 4.1 show that using calibration labels for adaptation (Adapt+SCP) breaks exchangeability; Conf-OT uses those same labels, albeit through an OT marginal rather than a classifier fit. Empirical coverage close to nominal in Tables 1-2 is a finite-sample observation, not a distribution-free guarantee. This is a central claim of the paper and needs either a rigorous proof of exchangeability (or a weighted conformal correction), or the claim 'maintaining coverage guarantees' must be withdrawn.
  2. [Sec. 4.2, Algorithm 1, Block 2] The conformal quantile in Eq. (3) is computed from scores s_i = S(q_i^*, y_i), where q_i^* are themselves produced by an optimization problem that ingests the calibration labels. In standard split conformal prediction, the score function is fixed before seeing the calibration labels; here the score function is a function of the calibration labels, so the rank of each s_i is not distribution-free. The manuscript does not provide a proposition, theorem, or reference that covers this kind of data-dependent score transformation. Without such a result, the statement in Sec. 5.2 that Conf-OT maintains coverage guarantees is unsupported.
  3. [Sec. 5.2, Tables 1 and 2] The empirical coverage values reported for Conf-OT are all close to 1-alpha, but these are averages over 20 random splits and are not accompanied by confidence intervals or a demonstration that the deviations are within the finite-sample conformal tolerance. More importantly, even exact empirical coverage on 15 datasets would not establish the distribution-free validity claim, because the central value proposition of conformal prediction is finite-sample validity without distributional assumptions. The manuscript should either prove the guarantee or explicitly reframe the contribution as an empirical method whose coverage is only approximate.
minor comments (5)
  1. [Supplementary, Appendix A] The sentence 'First, Conf-OT [59] proposes to train the optimum temperature scaling...' appears to be a typo: reference [59] is Stutz et al., and the method being described is temperature scaling, not the paper's own Conf-OT. This creates confusion about the related-work discussion.
  2. [Fig. 1(b), Sec. 4.1, Appendix E] The density plots in Fig. 1(b) are not fully explained: it is unclear whether the densities are over seeds, over datasets, or over calibration samples, and the exact notion of 'breaks exchangeability' being visualized should be stated in the caption or main text.
  3. [Algorithm 1] The symbol S is used both for the similarity matrix in Step 2 and for the non-conformity score function in the input line and in Steps 14-16; this overloading makes the algorithm harder to read and should be disambiguated.
  4. [Tables 2 and 8] The baseline notation TIMKL(cm||uK) and TIMKL(cm||m) uses 'cm' where the text in Appendix F defines the predicted label marginal as m-hat; the notation should be made consistent with the formal definition in Eqs. (19)-(20).
  5. [Sec. 5.3, Fig. 3(b)] The claim that Fig. 3(b) illustrates 'an inverse trend' between accuracy and set size when optimizing the entropic constraint is not backed by a precise explanation of what is plotted; the supplement (Fig. 5) clarifies part of this, but the main-text figure should be self-contained.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the coverage-guarantee argument: the label-dependent OT marginal is relabeled 'unsupervised' to claim exchangeability, while the reported set-size gains remain empirical and non-circular.

  1. self definitional [Sec. 4.2, Eq. (6) and Algorithm 1, Step 1.1 (definition of m)]
    "We propose a transfer learning strategy, which is: i) unsupervised, i.e., does not directly rely on label supervision... Thus, the proposed setting avoids introducing any distributional shifts that could potentially break the exchangeability assumption required in conformal prediction. ... in our scenario, we constrain the solution to respect the observed label-marginal distribution on the calibration set, such that m = 1/N \sum_i y_i^ohe, where y_i^ohe is the one-hot encoding of y_i."

    The paper's only argument that the OT transform preserves exchangeability is the word 'unsupervised.' But Algorithm 1 sets the marginal m from calibration one-hot labels, so Q* is a function of those labels. The claim 'no distributional shifts that could break exchangeability' therefore follows from the definition of 'unsupervised' rather than from any proof about the label-conditioned transform. Since the coverage guarantee in Eq. (2) is never derived for scores produced by a label-dependent reweighting, the theoretical output rests on a self-definitional label. The set-size and accuracy improvements are measured on held-out test data, so they are not circular; only the guarantee argument is.

full rationale

The reported efficiency improvements are empirical: Conf-OT is evaluated on disjoint calibration/test partitions across 15 datasets, and the test labels are never used to build the sets. No fitted parameter is renamed as a prediction, and no equation reduces to its own input. The self-citations in the paper (e.g., templates from Silva-Rodríguez et al., LP++, or the TransCLIP baseline) are not load-bearing for the central conformal claim. The one genuine circularity-adjacent step is the coverage argument: the method is called 'unsupervised' to conclude exchangeability, yet the only non-uniform input to the Sinkhorn problem is the calibration-label marginal m. The paper never proves that the resulting calibration and test non-conformity scores remain exchangeable, nor does it provide a weighted-conformal correction; invoking the standard split-conformal guarantee is therefore an unsupported inference. This is a rigor gap as much as a circularity, so it is weighted as partial rather than full: the central efficiency claims retain independent empirical content, while the theoretical 'maintaining coverage guarantees' claim is effectively assumed by calling the label-dependent procedure unsupervised.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim of coverage guarantee rests on the unproven assumption that the transductive OT code assignment preserves exchangeability of calibration and test scores. The calibration label marginal is a fitted statistic, and the Sinkhorn hyperparameters are hand-chosen. No new entities are introduced.

free parameters (3)
  • label_marginal_m = per-dataset label frequency vector from calibration set
    Computed as m = (1/N) sum onehot(y_i) in Eq. (6) and Algorithm 1 step 1.1. It is fitted to the calibration labels and used to constrain the optimal transport solution, affecting the conformal scores.
  • entropic_weight_tau = 1.0
    Fixed to 1.0 for all tasks (Sec 5.1). Controls the strength of the entropic constraint in Eq. (7). Chosen by hand, not tuned per dataset.
  • sinkhorn_iterations_T = 3
    Fixed to 3 in Algorithm 1. The convergence study in Appendix G.5 supports this choice, but it is still a hand-set hyperparameter.
assumptions (4)
  • domain assumption Calibration and test data are exchangeable in the target domain.
    Required for split conformal prediction coverage, stated in Sec 3.2 and Sec 4.1. This is standard but may be violated under real domain drift.
  • ad hoc to paper The optimal transport code transformation preserves exchangeability of the conformal scores.
    Sec 4.2 and Algorithm 1. The paper asserts that the transductive OT avoids breaking exchangeability, but no proof is given. This is the load-bearing assumption for the coverage guarantee.
  • standard math The Sinkhorn algorithm converges to the regularized optimal transport solution.
    Sec 4.2 uses Sinkhorn iterations with T=3, relying on known convergence properties of the entropic OT problem. This is standard but approximate in practice.
  • standard math The standard split conformal quantile theorem (Eq. (3)) applies to the computed scores.
    Sec 3.2. This is the classical result from Vovk et al. [66], but its validity here depends on the exchangeability of the scores, which is the unproven assumption above.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conformal Prediction for Zero-Shot Models." pith.science (2026). https://pith.science/paper/PMMJWW5Z

@misc{pith2026250524693,
  author       = {Pith},
  title        = {Pith review of: Conformal Prediction for Zero-Shot Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMMJWW5Z}},
  note         = {Machine review of arXiv:2505.24693}
}
read the original abstract

Vision-language models pre-trained at large scale have shown unprecedented adaptability and generalization to downstream tasks. Although its discriminative potential has been widely explored, its reliability and uncertainty are still overlooked. In this work, we investigate the capabilities of CLIP models under the split conformal prediction paradigm, which provides theoretical guarantees to black-box models based on a small, labeled calibration set. In contrast to the main body of literature on conformal predictors in vision classifiers, foundation models exhibit a particular characteristic: they are pre-trained on a one-time basis on an inaccessible source domain, different from the transferred task. This domain drift negatively affects the efficiency of the conformal sets and poses additional challenges. To alleviate this issue, we propose Conf-OT, a transfer learning setting that operates transductive over the combined calibration and query sets. Solving an optimal transport problem, the proposed method bridges the domain gap between pre-training and adaptation without requiring additional data splits but still maintaining coverage guarantees. We comprehensively explore this conformal prediction strategy on a broad span of 15 datasets and three non-conformity scores. Conf-OT provides consistent relative improvements of up to 20% on set efficiency while being 15 times faster than popular transductive approaches.

Figures

Figures reproduced from arXiv: 2505.24693 by the authors.

Figure 1
Figure 1. How to transfer black-box VLMs without breaking exchangeability? In this work, we explore split conformal prediction (SCP) for VLMs (see (a)) to provide trustworthiness guarantees. These zero-shot models typically undergo adaptation to enhance their performance. However, leveraging the SCP calibration data for adaptation breaks the exchangeability assumption [66], which produces miss-coverage during inference (see (… view at source ↗
Figure 2
Figure 2. Entropic constraint (τ ). Conf-OT is compatible with recent observations [69] regarding the positive effect on set size of temperature scaling (τ < 1) on adaptive scores (b,c). However, such behavior does not generalize to non-adaptive scores, i.e., LAC (a), whereas Conf-OT improves the performance atop all non-conformity scores. Results using CLIP ViT-B/16 on 15 datasets with α = 0.10. 0 2.5 5 7.5 10 Accuracy -8 -6… view at source ↗
Figure 3
Figure 3. Accuracy vs. set size change (∆) using Conf-OT. (a) Correlation among datasets for LAC [39]. (b) Effect of the entropic constraint for RAPS [2]. Results using CLIP ViT-B/16 on 15 datasets with α = 0.10. More information in Appendix G.4. Conf-OT components. The proposed approach presents a small number of tunable elements. First, as previously dis￾cussed, we fixed the entropic constraint weight to its stan￾dard value… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: an extended version of [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Relation of accuracy against set size improvement on adaptive scores, i.e., APS [51] (a) and RAPS [2] (b), resulting from modifying the distribution sharpness, via temperature scal￾ing. Results using ViT-B/16 on 15 datasets with α = 0.10. These results complement [PIT…
Figure 6
Figure 6. Figure 6: presents the convergence of the Sinkhorn algorithm in Conf-OT regarding the number of iterations by measur￾ing set size. These results demonstrate that such algorithms reach a satisfactory convergence after three iterations. This observation is consistent with typical …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 73 canonical work pages

  1. [1]

    Learn- ing by transduction

    Vladimir Vapnik Alex Gammerman, V olodya V ovk. Learn- ing by transduction. In Conference on Uncertainty in Artifi- cial Intelligence, pages 148–156, 1998. 1, 3, 12

    work page 1998

  2. [2]

    Uncertainty sets for image clas- sifiers using conformal prediction

    Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image clas- sifiers using conformal prediction. In International Confer- ence on Learning Representations (ICLR) , 2020. 1, 3, 4, 6, 7, 8, 12, 13, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26

    work page 2020

  3. [3]

    Self-labelling via simultaneous clustering and representation learning

    YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In In- ternational Conference on Learning Representations (ICLR),

    work page

  4. [4]

    Food-101 – mining discriminative components with ran- dom forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with ran- dom forests. In European Conference on Computer Vision (ECCV), 2014. 6, 13

    work page 2014

  5. [5]

    Transductive infor- mation maximization for few-shot learning

    Malik Boudiaf, Ziko Imtiaz Masud, J ´erˆome Rony, Jose Dolz, Pablo Piantanida, and Ismail Ben Ayed. Transductive infor- mation maximization for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 1– 13, 2020. 3, 6, 7, 14, 15, 19

    work page 2020

  6. [6]

    Malik Boudiaf, Hoel Kervadec, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation with- out meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 13979– 13988, 2021. 8

    work page 2021

  7. [7]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vi- sion (ECCV), pages 132–149, 2018. 5, 16

    work page 2018

  8. [8]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 3606– 3613, 2014. 6, 13

    work page 2014

Show all 77 references
  1. [9]

    Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi

    Alvaro H.C. Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. An information theoretic per- spective on conformal prediction. In Advances in Neural In- formation Processing Systems (NeurIPS), 2024. 3, 12

    2024

  2. [10]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. InAdvances in Neural Information Pro- cessing Systems (NeurIPS), 2013. 4, 5

    2013

  3. [11]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009. 1, 6, 13

    2009

  4. [12]

    A baseline for few-shot image clas- sification

    Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichan- dran, and Stefano Soatto. A baseline for few-shot image clas- sification. In International Conference on Learning Repre- sentations (ICLR), 2020. 3

    2020

  5. [13]

    Class-conditional conformal prediction with many classes

    Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-conditional conformal prediction with many classes. InAdvances in Neu- ral Information Processing Systems (NeurIPS) , 2023. 1, 4, 6, 12, 13

    2023

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  7. [15]

    Training uncertainty-aware classifiers with conformalized deep learning

    Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty-aware classifiers with conformalized deep learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022. 3, 12

    2022

  8. [16]

    Fergus, and P

    Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Worskshops (CVPRW), page...

    2004

  9. [17]

    Discriminative ensemble meta-learning with co-regularization for rare fundus diseases diagnosis

    Mengdi Gao, Hongyang Jiang, Lei Zhu, Zhe Jiang, Mufeng Geng, Qiushi Ren, and Yanye Lu. Discriminative ensemble meta-learning with co-regularization for rare fundus diseases diagnosis. Medical Image Analysis, 89:102884, 2023. 2, 12

    2023

  10. [18]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision (IJCV),

  11. [19]

    Transductive conformal inference with adaptive scores

    Ulysse Gazin, Gilles Blanchard, and Etienne Roquain. Transductive conformal inference with adaptive scores. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 348–360, 2024. 3, 12

    2024

  12. [20]

    Visual-language prompt tuning with knowledge-guided context optimization

    Changsheng Xu Hantao Yao, Rui Zhang. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2

    2023

  13. [21]

    Deep residual learning for image recognition.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6

    2016

  14. [22]

    Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover clas- sification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover clas- sification. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , pages 3606–3613, 2018. 6, 13

    2018

  15. [23]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 15262–15271,

  16. [24]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. InP...

    2021

  17. [25]

    Conformal prediction for deep clas- sifier via label ranking

    Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep clas- sifier via label ranking. In International Conference on Ma- chine Learning (ICML), pages 20331–20347, 2024. 1, 4

    2024

  18. [26]

    Lp++: A surprisingly strong linear probe for few-shot clip

    Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. Lp++: A surprisingly strong linear probe for few-shot clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23773–23782, 2024. 1, 2

    2024

  19. [27]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , page 3498–3505,

  20. [28]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Preprint, 2009. 1

    2009

  21. [29]

    Tibshi- rani, and Larry Wasserman

    Jing Lei, Max G’Sell, Alessandro Rinaldo, Ryan J. Tibshi- rani, and Larry Wasserman. Distribution-free predictive in- ference for regression. Journal of the American Statistical Association, 113(523):1094–1111, 2018. 4

    2018

  22. [30]

    Difficulty-aware meta-learning for rare disease diagnosis

    Xiaomeng Li, Lequan Yu, Yueming Jin, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Difficulty-aware meta-learning for rare disease diagnosis. In Medical Image Computing and Computer Assisted Intervention (MICCAI) , pages 357–366,

  23. [31]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVP...

    2023

  24. [32]

    Effective adaptation in multi- task co-training for unified autonomous driving

    Xiwen Liang, Yangxin Wu, Jianhua Han, Hang Xu, Chun- jing Xu, and Xiaodan Liang. Effective adaptation in multi- task co-training for unified autonomous driving. Advances in Neural Information Processing Systems (NeurIPS) , 35: 19645–19658, 2022. 1

    2022

  25. [33]

    Learning object-language alignments for open-vocabulary object de- tection

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gho- lamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object de- tection. In International Conference on Learning Represen- tations (ICML), 2023. 1

    2023

  26. [34]

    Multimodality helps unimodality: Cross- modal few-shot learning with multimodal models

    Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross- modal few-shot learning with multimodal models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2

    2023

  27. [35]

    Clip-driven universal model for organ segmentation and tumor detection

    Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Compute...

    2023

  28. [36]

    Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In In- ternational Conference on Machine Learning (ICML), pages 23033–23044. PMLR, 2023. 1

    2023

  29. [37]

    S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. In arXiv preprint arXiv:1306.5151, 2013. 6, 13

    2013 arXiv

  30. [38]

    Transductive zero-shot and few-shot clip

    Segolene Martin, Yunshi Huang, Fereshteh Shakeri, Jean- Christophe Pesquet, and Ismail Ben Ayed. Transductive zero-shot and few-shot clip. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28816–28826, 2024. 3

    2024

  31. [39]

    Least ambiguous set-valued classifiers with bounded error levels

    Jing Lei Mauricio Sadinle and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association , 114(525): 223–234, 2019. 1, 3, 4, 6, 7, 8, 12, 15, 18, 19, 20, 21, 22, 23, 24, 25, 26

    2019

  32. [40]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. In International Conference on Learning Representations (ICLR) , pages 1– 17, 2023. 2

    2023

  33. [41]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neu- ral Information Processing Systems (NeurIPS), 36, 2024. 1

    2024

  34. [42]

    Robust calibration of large vision- language adapters

    Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large vision- language adapters. In European Conference on Computer Vision (ECCV), pages 1–19, 2024. 1, 2

    2024

  35. [43]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In In- dian Conference on Computer Vision, Graphics and Image Processing, 2008. 6, 13

    2008

  36. [44]

    Chils: Zero-shot image classification with hierarchical label sets

    Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning (ICML), pages 26342–26362, 2023. 1

    2023

  37. [45]

    To- wards calibrated robust fine-tuning of vision-language mod- els

    Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyungwoo Song. To- wards calibrated robust fine-tuning of vision-language mod- els. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1

    2024

  38. [46]

    Inductive confidence machines for re- gression

    Harris Papadopoulos, Kostas Proedrou, V olodya V ovk, and Alex Gammerman. Inductive confidence machines for re- gression. In European Conference on Machine Learning (ECML), pages 345–356, 2002. 1, 3, 4, 12

    2002

  39. [47]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 3498–3505, 2012. 6, 13

    2012

  40. [48]

    Transductive confidence machines for pattern recognition

    Kostas Proedrou, Ilia Nouretdinov, V olodya V ovk, and Alex Gammerman. Transductive confidence machines for pattern recognition. In European Conference on Machine Learning (ECML), pages 381–390, 2002. 1, 3, 12

    2002

  41. [49]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  42. [50]

    Do imagenet classifiers generalize to im- 10 agenet? In International Conference on Machine Learning (ICML), pages 5389–5400, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- 10 agenet? In International Conference on Machine Learning (ICML), pages 5389–5400, 2019. 6, 13

    2019

  43. [51]

    Clas- sification with valid and adaptive coverage

    Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Clas- sification with valid and adaptive coverage. In Advances in Neural Information Processing Systems (NeurIPS), pages 3581–3591, 2020. 1, 3, 4, 6, 7, 8, 12, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26

    2020

  44. [52]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 1

    2022

  45. [53]

    Transduction with confidence and credibility

    Craig Saunders, Alexander Gammerman, and V olodya V ovk. Transduction with confidence and credibility. In Interna- tional Joint Conference on Artificial Intelligence (IJCAI) , pages 722–726, 1999. 1, 3, 12

    1999

  46. [54]

    Few-shot adaptation of medical vision-language models

    Fereshteh Shakeri, Yunshi Huang, Julio Silva-Rodr ´ıguez, Houda Bahig, An Tang, Jose Dolz, and Ismail Ben Ayed. Few-shot adaptation of medical vision-language models. In Medical Image Computing and Computer-Assisted Interven- tion (MICCAI), pages 553–563, 2024. 1, 2, 12

    2024

  47. [55]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems (NeurIPS), 35:14274–14289, 2022. 1, 2

    2022

  48. [56]

    A closer look at the few-shot adaptation of large vision-language models

    Julio Silva-Rodr ´ıguez, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few-shot adaptation of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23681–23690, 2024. 1, 2, 4, 13

    2024

  49. [57]

    A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision

    Julio Silva-Rodr ´ıguez, Hadi Chakor, Riadh Kobbi, Jose Dolz, and Ismail Ben Ayed. A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision. Medical Image Analysis, 99:103357, 2025. 1

    2025

  50. [58]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. In arXiv preprint arXiv:1212.0402, 2012. 6, 13

    2012 arXiv

  51. [59]

    Learning optimal conformal classifiers

    David Stutz, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. In International Conference on Learning Repre- sentations (ICLR), 2022. 1, 3, 4, 6, 12, 13

    2022

  52. [60]

    An empirical study into what matters for calibrating vision-language models

    Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. In International Confer- ence on Machine Learning (ICML), 2024. 1

    2024

  53. [61]

    Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No ”zero-shot” without exponential data: Pretraining concept frequency determines multimodal model performance. In Advances in Neural Information Pro...

    2024

  54. [62]

    Realistic evaluation of transductive few-shot learning

    Olivier Veilleux, Malik Boudiaf, Pablo Piantanida, and Is- mail Ben Ayed. Realistic evaluation of transductive few-shot learning. In Advances in Neural Information Processing Sys- tems (NeurIPS), pages 9290–9302, 2021. 3

    2021

  55. [63]

    Optimal transport: old and new

    C ´edric Villani. Optimal transport: old and new . Springer,

  56. [64]

    Conditional validity of inductive conformal predictors

    Vladimir V ovk. Conditional validity of inductive conformal predictors. In Proceedings of the Asian Conference on Ma- chine Learning, pages 475–490, 2012. 3, 12

    2012

  57. [65]

    Transductive conformal predictors

    Vladimir V ovk. Transductive conformal predictors. In Arti- ficial Intelligence Applications and Innovations, pages 348– 360, 2013. 3

    2013

  58. [66]

    Algo- rithmic Learning in a Random World

    Vladimir V ovk, Alex Gammerman, and Glenn Shafer. Algo- rithmic Learning in a Random World. Springer, 2005. 1, 2, 3, 4, 8, 12

    2005

  59. [67]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021. 3

    2021

  60. [68]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 6, 13

    2019

  61. [69]

    Does confidence calibration improve confor- mal prediction? In arXiv preprint arXiv:2402.04344, 2024

    Huajun Xi, Jianguo Huang, Kangdao Liu, Lei Feng, and Hongxin Wei. Does confidence calibration improve confor- mal prediction? In arXiv preprint arXiv:2402.04344, 2024. 3, 4, 6, 7, 8, 13, 16

    2024 arXiv

  62. [70]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485–3492, 2010. 6, 13

    2010

  63. [71]

    Demystifying CLIP data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In International Conference on Learning Rep- resentations (ICLR), 2024. 6

    2024

  64. [72]

    Task residual for tuning vision-language models

    Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision-language models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10899–10909, 2023. 1

    2023

  65. [73]

    Boost- ing vision-language models with transduction

    Maxime Zanella, Beno ˆıt G´erin, and Ismail Ben Ayed. Boost- ing vision-language models with transduction. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 3, 6, 7, 14, 15, 19

    2024

  66. [74]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. In European Conference on Computer Vision (ECCV), pages 1–19, 2022. 1, 2

    2022

  67. [75]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. Inter- national Journal of Computer Vision (IJCV) , 2022. 1, 2, 3, 6, 13

    2022

  68. [76]

    Ease: Unsupervised discrim- inant subspace learning for transductive few-shot learning

    Hao Zhu and Piotr Koniusz. Ease: Unsupervised discrim- inant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9068–9078,

  69. [2022]

    ↓” indicates smaller values are better. Bold numbers are superior results. These results complement Tab. 2 in the main manuscript. Grayish marginal coverage (“Cov

    3 11 Conformal Prediction for Zero-Shot Models Supplementary Material A. Extended related works In the following, we provide extended remarks about prior literature on conformal prediction. Concretely, we delve into its historical transductive nature and two different fam- ili...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.