Pith. sign in

REVIEW 3 major objections 5 minor 57 references

SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SoC replaces the aggressive full-orthogonality penalty in test-time prompt tuning with a Huber-based capped regularizer, producing better-calibrated vision-language predictions without sacrificing accuracy.

desk verdict The empirical story is credible and useful, but the theoretical centerpiece is a false theorem and should be removed or fixed before you rely on it. read the letter →

arxiv 2601.08617 v1 pith:PBLBF5OR submitted 2026-01-13 cs.CV

Leo Fillioux , Omprakash Chakraborty , Ismail Ben Ayed , Paul-Henry Cournède , Stergios Christodoulidis , Maria Vakalopoulou , Jose Dolz This is my paper · ORCID
classification cs.CV
keywords test-timeprompttuningcalibrationexpectederrorvision-languagemodelsCLIPorthogonalityconstraintsHuberlosssemanticsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that test-time prompt tuning of vision-language models has been optimized mainly for accuracy, leaving their confidence estimates badly miscalibrated, and that the recent fix of forcing pairwise orthogonality between class text embeddings makes matters worse by repelling semantically similar classes until the model becomes overconfident. It proposes Semantic Orthogonal Calibration (SoC), which replaces the quadratic orthogonality penalty with a Huber-based regularizer that caps the repulsion applied to similar class prototypes. On 11 fine-grained datasets, SoC cuts average expected calibration error from 7.7 (for O-TPT) to 5.4 while slightly improving average accuracy, and it also lowers calibration error on the four ImageNet natural distribution-shift benchmarks. A first-order gradient analysis supports the intuition: full orthogonality contracts worst-case prototype similarity roughly twice as fast per gradient step as the capped penalty, and this sharper contraction pushes the confidence floor higher.

What carries the argument

The central object is the pairwise cosine-similarity matrix of the unit-normalized text class prototypes. O-TPT penalizes the squared Frobenius distance from the identity, so the gradient on a highly similar pair is proportional to the similarity and grows without bound. SoC instead applies a Huber loss to each off-diagonal similarity, quadratic for similarities below margin δ and linear above it, after normalizing similarities and setting δ to the 20th percentile of the empirical distribution. The argumentative engine is a one-step first-order analysis showing that O-TPT contracts worst-case similarity μ roughly twice as fast as SoC, combined with the claimed bound pmax(v) ≥ 1/(1 + (K−1)exp

What would settle it

Check Proposition 1 with a unit image embedding orthogonal to all K class prototypes: all logits are zero, so pmax = 1/K, which for μ < 1 lies strictly below the claimed bound 1/(1 + (K−1)exp(−α(1−μ))). That single calculation refutes the proposition as stated. A weaker test of the general mechanism: run O-TPT on a dataset whose mistakes are dominated by semantically distant class pairs; if calibration error still rises sharply, high-similarity repulsion is not the sole driver.

Watch

Extended reading notes

Core claim

If the paper is right, overconfidence in calibrated test-time prompt tuning is caused not by under-separated prototypes but by over-separation: the quadratic orthogonality penalty gives the largest gradients to the class pairs with the highest cosine similarity, which are precisely the semantically related classes that should remain close. SoC replaces that penalty with a Huber loss on pairwise prototype similarities, quadratic below a margin and linear above it, so that high-similarity pairs receive a fixed, capped repulsion instead of one that grows with similarity. Empirically, across two ViT backbones, 18 prompt initializations, one and two gradient steps, and CoOp-initialized prompts, S

Load-bearing premise

The theory assumes that the worst-case pairwise similarity between class prototypes controls the softmax confidence floor for every possible image embedding; if that bound fails for embeddings far from all prototypes, the paper's formal explanation of why full orthogonality causes overconfidence loses its support.

Editorial extensions

If this is right

  • Test-time prompt tuning can become nearly as well calibrated as zero-shot CLIP (average ECE 5.4 vs 5.1) without giving up accuracy, removing a key obstacle to deployment in safety-critical settings.
  • Future prompt-tuning regularizers that penalize pairwise prototype similarity should cap or otherwise soften repulsion for high-similarity pairs, since the quadratic form itself is what inflates confidence.
  • SoC degrades much less than O-TPT when a second gradient step is taken (ECE worsens 23% vs 39%), making multi-step test-time adaptation more practical.
  • Selective classification accuracy improves by roughly 5–10 points at matched confidence thresholds, which matters for systems that gate decisions on confidence.
  • Post-hoc calibrators such as logit normalization add less to SoC than to O-TPT, indicating the model is already closer to calibrated end-to-end.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same 'cap the repulsion, preserve semantic proximity' principle transfers beyond test-time prompt tuning to few-shot prompt tuning or CLIP fine-tuning, where pairwise prototype penalties also appear.
  • My inference: a cheap diagnostic for future methods is to monitor per-pair ECE restricted to class pairs with high zero-shot cosine similarity; the paper's Fig. 2 shows a strong positive correlation (r≈0.69) for O-TPT and near-zero for SoC on EuroSAT.
  • My inference: the theoretical confidence-floor bound is not shown for image embeddings far from every prototype; a unit vector orthogonal to all prototypes gives pmax = 1/K, which falls below the claimed bound whenever μ < 1, so the proof needs correction or the bound needs a weaker form — the empirical benefits could still hold without the theorem as stated.
  • My inference: setting δ as a percentile of the empirical similarity distribution makes the method dataset-dependent; adapting δ per task or learning it alongside the prompts is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Semantic Orthogonal Calibration (SoC), a Huber-based regularizer for test-time prompt tuning in vision-language models, replacing the full orthogonality penalty used in O-TPT. The authors claim that full orthogonality aggressively repels semantically related prototypes and systematically inflates confidence, and they support this with a theoretical analysis (Proposition 1 and Corollary 1) linking the worst-case pairwise prototype similarity µ to a softmax confidence floor. They also provide a first-order gradient analysis of the one-step prompt update. Empirically, they evaluate SoC against TPT, C-TPT, and O-TPT on 11 fine-grained/classification datasets and ImageNet distribution-shift benchmarks with ViT-L/14 and ViT-B/16 backbones, reporting consistent ECE improvements and competitive accuracy, plus ablations on prompts, CoOp initialization, selective accuracy, and SaLS compatibility.

Significance. If the empirical findings hold, SoC is a simple and effective modification that improves calibration across diverse benchmarks while preserving discriminative performance. The experimental body is extensive: multiple backbones, 15 datasets, reliability diagrams, ablations on λ, prompt initialization, and two-step updates. The theoretical analysis, however, is a central advertised contribution and is currently not sound as stated. The paper would be valuable if the theory is repaired or downscaled, since the empirical case alone may justify the method. The lack of released code is a limitation but not disqualifying.

major comments (3)
  1. [Section 4.1, Proposition 1, Appendix A] Proposition 1 claims a universal lower bound on softmax confidence for any unit vector v. The proof only treats v aligned with the top prototype t_i*, calling this the 'worst' case, but this is not the worst case. For example, with K=2, t1=e1, t2=e2, and v=e3 (d>=3), all logits are zero, so pmax(v)=1/2. The claimed RHS is 1/(1+exp(-α)), which is greater than 1/2 for any α>0 and tends to 1 for CLIP-scale α. Thus Proposition 1 is false as stated, and the theoretical bridge from prototype coherence µ to a confidence floor is not established. This invalidates the paper's stated mechanism for why O-TPT is overconfident.
  2. [Section 4.2, Corollary 1, Appendix B] The first-order analysis (Eqs. 2–4) isolates the regularizer gradient and drops the L_TPT term, which is part of the actual optimization objective used by SoC, O-TPT, and TPT. The derived comparisons µ'_O-TPT < µ'_Huber therefore describe only a projected dynamics, not the real one-step update. Furthermore, Corollary 1 concludes that O-TPT 'yields a strictly higher confidence' from the fact that a lower bound (under a now-false Proposition 1) increases. Even if Proposition 1 were correct, this shows only that the confidence floor increases, not that the model's actual predictive distribution becomes more confident. The theoretical claim that full orthogonality inflates confidence more aggressively than SoC is therefore not supported.
  3. [Appendix B, exact similarity shifts] The 'exact' derivation in Appendix B is still an O(η^2)-truncated first-order approximation and still excludes the gradient of L_TPT. The concluding sentence that 'the stronger second-order structure of O-TPT translates directly into more confident predictions' is not justified by the equations, which analyze only the regularizer's contribution to the similarity shift. This does not disprove the empirical calibration differences, but it means the paper's theoretical explanation of those differences is incomplete.
minor comments (5)
  1. [Section 4.2, Eq. (3) and Eq. (4)] Typos in the gradient expressions: in Eq. (3), the second term should involve t_i, not t_j; likewise in Eq. (4) the second term should be t_i^ op(gδ(sij)t_i). The final results are unaffected, but the derivation as written is confusing.
  2. [Section 4.2, Huber gradient] The text writes '∇Huber tj ≈ 2gδ(sij)ti' with an extra factor 2, while Eq. (4) uses only gδ(sij). This inconsistency should be fixed.
  3. [Section 4.1, footnote 1] The assumption that all pairwise cosine similarities are non-negative is not justified. CLIP text embeddings are unit-norm but can have negative cosines; if negative similarities can occur, the domain of δ and the statement µ∈[0,1] need clarification.
  4. [Section 4.1, proof of Proposition 1] The phrase 'i.e., worst' when setting v aligned with t_i* is not supported. The proof should either justify why this is indeed the worst case over all unit v or be corrected to a different statement.
  5. [General] Corollary 1 is phrased as 'Confidence increases under full orthogonality,' but the result concerns the confidence lower bound, not the actual confidence. The wording should be adjusted to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SoC's empirical calibration gains are measured against external baselines; the theoretical bound is mathematically flawed but not constructed from the method's own outputs.

full rationale

The paper's central claim—that SoC improves calibration over O-TPT—is supported by external benchmark evaluations (Tables 1–3) and does not reduce to the regularizer's definition. The theoretical analysis in Section 4.1 is a standalone mathematical attempt to bound softmax confidence in terms of worst-case prototype similarity µ; it is not a fitted parameter renamed as a prediction. Proposition 1's proof is indeed invalid for arbitrary unit vectors (e.g., a vector orthogonal to all prototypes yields pmax = 1/K, which can lie below the claimed bound), but that is a mathematical correctness flaw, not circular reasoning: the bound is not defined as the quantity it is meant to explain. Corollary 1 follows logically from the (flawed) bound and the first-order gradient comparison in Section 4.2, neither of which is fitted to the ECE results. Hyperparameters δ and λ are tuned in Appendix D, but they are not presented as predicted outcomes. Self-citations such as SaLS [33] are used as baselines or comparisons, not as load-bearing justification for SoC's central claim. Consequently, no step in the derivation chain is equivalent to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about CLIP embeddings and the single-step TPT protocol, plus an ad hoc theoretical proposition that is false. The free parameters δ and λ are tuned on the benchmarks, which weakens the generality of the empirical claims.

free parameters (3)
  • δ (Huber margin) = 20th percentile of normalized cosine similarities (dataset-dependent)
    Chosen per dataset as the 20th percentile of normalized S, not derived from theory.
  • λ (regularization weight) = 30 (or 14 for distribution shift), scaled by |L_SoC|/|L_TPT|
    Set by hand; the appendix states these values were chosen, and the scaling is designed to maintain a constant optimal value across datasets.
  • Similarity normalization choice = S~ = (S - Smin)/(Smax - Smin)
    Min-max normalization of cosine similarities is a design choice; ablation shows it outperforms two alternatives on average ECE.
assumptions (5)
  • domain assumption CLIP class embeddings are unit-norm and pairwise cosine similarities are non-negative.
    The analysis assumes s_ij ≥ 0 and ||t_i|| = 1 (Section 3.1, Section 3.3 note). This holds for CLIP in practice but is not universally guaranteed for other VLMs.
  • ad hoc to paper The dominant-pair approximation preserves the qualitative behavior of the gradient update.
    Used in Section 4.2 to simplify the analysis; the appendix gives exact forms but the qualitative claim that O-TPT always reduces µ more is asserted, not proven for all similarity matrices.
  • domain assumption A single gradient-step, first-order approximation with O(η²) neglected is valid for the behavior of TPT.
    TPT uses one step, but the analysis also ignores the gradient of the TPT entropy loss, which also moves prototypes; the paper does not quantify the effect.
  • ad hoc to paper Proposition 1's claimed universal bound on softmax confidence is true.
    The proposition is the theoretical lynchpin but is false as stated; the proof only covers v = t_i* and a counterexample exists (v orthogonal to all prototypes).
  • domain assumption Lowering the softmax confidence floor improves calibration (lower ECE).
    The paper uses the confidence floor as a proxy for overconfidence; empirically this is plausible but not formally established between the bound and ECE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning." pith.science (2026). https://pith.science/paper/PBLBF5OR

@misc{pith2026260108617,
  author       = {Pith},
  title        = {Pith review of: SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBLBF5OR}},
  note         = {Machine review of arXiv:2601.08617}
}
read the original abstract

With the increasing adoption of vision-language models (VLMs) in critical decision-making systems such as healthcare or autonomous driving, the calibration of their uncertainty estimates becomes paramount. Yet, this dimension has been largely underexplored in the VLM test-time prompt-tuning (TPT) literature, which has predominantly focused on improving their discriminative performance. Recent state-of-the-art advocates for enforcing full orthogonality over pairs of text prompt embeddings to enhance separability, and therefore calibration. Nevertheless, as we theoretically show in this work, the inherent gradients from fully orthogonal constraints will strongly push semantically related classes away, ultimately making the model overconfident. Based on our findings, we propose Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smooth prototype separation while preserving semantic proximity, thereby improving calibration compared to prior orthogonality-based approaches. Across a comprehensive empirical validation, we demonstrate that SoC consistently improves calibration performance, while also maintaining competitive discriminative capabilities.

Figures

Figures reproduced from arXiv: 2601.08617 by the authors.

Figure 1
Figure 1. Motivation for SoC. With O-TPT, ambiguity inherent to the class semantics is lost due to the aggressive orthogonal￾ity constraint, leading to artificially high confidence, even when predictions are incorrect. Let us take this image as an example, whose correct class is “annual crop land”, and whose clos￾est semantic class across all categories is “permanent crop land”. The zero-shot CLIP prediction incorrectly class… view at source ↗
Figure 2
Figure 2. ECE per class pair as a function of the zero-shot co￾sine similarity. We compute the ECE for the wrong predictions across each class pair (i.e., the model predicted class i when the label was class j) and analyze the relation with the zero-shot sim￾ilarity between both classes on EuroSAT. For classes with high initial semantic similarity, O-TPT is overconfident, caused by the underlying drawbacks of enforcing orthog… view at source ↗
Figure 3
Figure 3. Reliability diagrams of O-TPT vs SoC. Plots showing the calibration error across the Flowers102, EuroSAT and FGVC Aircraft datasets for O-TPT (top row) and SoC (bottom row). Robustness under natural distributional drifts. Tab. 2 presents the results across the four different variants of Im￾ageNet, which present a distributional drift. In particular, we can observe that while performing on par with O-TPT in terms of … view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Calibration sensitivity of O-TPT vs. SoC for various initial text prompts. ECE on the DTD (left) and Aircraft (right) datasets for 18 different prompts from CLIP [39]. Calibration across prompt variability. Prompt initializa￾tion remains a critical factor in CLIP-based…
Figure 4
Figure 4. Figure 4: ECE and accuracy for one and two gradient steps. Standard one-step and two-step gradient updates for C-TPT, O￾TPT, and SoC averaged across 11 datasets. Hatched bars indicate the result when applying two gradient updates over text prompts. Robustness to backbone. To ass…
Figure 6
Figure 6. Figure 6: Selective accuracy across different thresholds. For each threshold, we select only the samples whose confidence (i.e., maximum of softmax) exceeds that value, and compute the accu￾racy on this subset. gies. To assess the compatibility of our method with post￾hoc calibr…
Figure 7
Figure 7. Figure 7: ECE with and without SaLS. ECE for C-TPT, O-TPT, and SoC with and without applying SaLS for further calibration. 6. Conclusion In this work, we have shown that enforcing full orthogo￾nality (i.e., O-TPT) in test-time prompt tuning, while intu￾itively appealing, systema…
Figure 8
Figure 8. Figure 8: Ablation on the value of λ. Accuracy and calibration error for multiple values of the regularization term λ of SoC for the Flowers, Aircraft, UCF101, and Food101 datasets. 5 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Reliability plots for O-TPT. Calibration error across all 11 datasets for O-TPT. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Reliability plots for SoC. Calibration error across all 11 datasets for SoC. 7 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 2 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

  2. [2]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. InEuropean Conference on Computer Vision, 2014. 5, 3

    2014

  3. [3]

    Calibrating deep neural networks by pairwise constraints

    Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13709–13718, 2022. 2

    2022

  4. [4]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. InIEEE Confer- ence on Computer Vision and Pattern Recognition, 2014. 5, 3

    2014

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 5, 3

    2009

  6. [6]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning - Volume 70, page 1321–1330. JMLR.org, 2017. 2

    2017

  7. [7]

    Visual-language prompt tuning with knowledge-guided context optimization

    Changsheng Xu Hantao Yao, Rui Zhang. Visual-language prompt tuning with knowledge-guided context optimization. InThe IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2023. 2

    2023

  8. [8]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. 5, 3

    2019

Show all 57 references
  1. [9]

    Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan

    Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. InInternational Conference on Learning Repre- sentations (ICLR), 2020. 2

    2020

  2. [10]

    The many faces of robustness: A critical analysis of out-of-distribution generalization.ICCV, 2021

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization.ICCV...

    2021

  3. [11]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 15262–15271,

  4. [12]

    Peter J. Huber. Robust estimation of a location parameter. Annals of Mathematical Statistics, 35(1):73–101, 1964. 4

    1964

  5. [13]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational Con- ference on Machine Learning, 2021. 2

    2021

  6. [14]

    Simple but effective: CLIP embed- dings for embodied AI

    Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: CLIP embed- dings for embodied AI. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14829–14838, 2022. 2

    2022

  7. [15]

    Biomedcoop: Learning to prompt for biomedi- cal vision-language models

    Taha Koleilat, Hojat Asgariandehkordi, Hassan Rivaz, and Yiming Xiao. Biomedcoop: Learning to prompt for biomedi- cal vision-language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14766– 14776, 2025. 2

    2025

  8. [16]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013. 5, 3

    2013

  9. [17]

    One-shot learning of object categories.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006

    Fei-Fei Li, Rob Fergus, and Pietro Perona. One-shot learning of object categories.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006. 5, 3

    2006

  10. [18]

    Selvaraju, Akhilesh Deepak Got- mare, Shafiq Joty, Caiming Xiong, and Steven Hoi

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Got- mare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align be- fore fuse: Vision and language representation learning with momentum distillation. InNeurIPS, 2021. 2

    2021

  11. [19]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InIn- ternational Conference on Machine Learning, 2022. 2

    2022

  12. [20]

    CLIP-reID: exploiting vision-language model for image re-identification without concrete text labels

    Siyuan Li, Li Sun, and Qingli Li. CLIP-reID: exploiting vision-language model for image re-identification without concrete text labels. InProceedings of the AAAI conference on artificial intelligence, pages 1405–1413, 2023. 2 9

    2023

  13. [21]

    The devil is in the margin: Margin-based label smooth- ing for network calibration

    Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smooth- ing for network calibration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 80–88, 2022. 2

    2022

  14. [22]

    Class adaptive network calibration

    Bingyuan Liu, J ´erˆome Rony, Adrian Galdran, Jose Dolz, and Ismail Ben Ayed. Class adaptive network calibration. InConference on Computer Vision and Pattern Recognition (CVPR), pages 16070–16079, 2023. 2

    2023

  15. [23]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2

    2023

  16. [24]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. InInternational Conference on Learning Representations (ICLR), 2019. 5

    2019

  17. [25]

    Contrast-aware calibration for fine-tuned CLIP: Leveraging image-text alignment.arXiv preprint arXiv:2501.19060, 2025

    Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Yu-Feng Li, and Lan-Zhe Guo. Contrast-aware calibration for fine-tuned CLIP: Leveraging image-text alignment.arXiv preprint arXiv:2501.19060, 2025. 3

    2025 arXiv

  18. [26]

    S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical re- port, 2013. 5, 3

    2013

  19. [27]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Gold- stein Tom, Anandkumar Anima, and Xiao Chaowei. Test- time prompt tuning for zero-shot generalization in vision- language models. InNeurIPS, 2022. 2, 3, 5

    2022

  20. [28]

    Revisiting the calibration of modern neu- ral networks

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neu- ral networks. InAdvances in Neural Information Processing Systems, pages 15682–15694. Curran Associates, Inc., 2021. 5

    2021

  21. [29]

    Bayesadapter: enhanced uncertainty estimation in CLIP few-shot adaptation.International Journal of Computer Vision (IJCV), 2025

    Pablo Morales- ´Alvarez, Stergios Christodoulidis, Maria Vakalopoulou, Pablo Piantanida, and Jose Dolz. Bayesadapter: enhanced uncertainty estimation in CLIP few-shot adaptation.International Journal of Computer Vision (IJCV), 2025. 3

    2025

  22. [30]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. InComputer Vision – ECCV 2022, pages 514–532. Springer International Publishing, 2022. 2

    2022

  23. [31]

    Calibrating deep neural networks using focal loss

    Jishnu Mukhoti et al. Calibrating deep neural networks using focal loss. InAdvances in Neural Information Processing Systems (NeurIPS), 2020. 2

    2020

  24. [32]

    When does label smoothing help? NeurIPS, 32, 2019

    Rafael M ¨uller et al. When does label smoothing help? NeurIPS, 32, 2019. 2

    2019

  25. [33]

    Robust calibration of large vision- language adapters

    Balamurali Murugesan, Julio Silva-Rodriguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large vision- language adapters. InEuropean Conference on Computer Vision (ECCV), 2024. 2, 3, 6, 8, 4, 5

    2024

  26. [34]

    Nilsback and A

    M-E. Nilsback and A. Zisserman. Automated flower classi- fication over a large number of classes. InIndian Conference on Computer Vision, Graphics and Image Processing, 2008. 5, 3

    2008

  27. [35]

    Towards calibrated robust fine- tuning of vision-language models.Advances in Neural Infor- mation Processing Systems, 37:12677–12707, 2024

    Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine- tuning of vision-language models.Advances in Neural Infor- mation Processing Systems, 37:12677–12707, 2024. 3

    2024

  28. [36]

    VLP: Vision language planning for autonomous driv- ing

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. VLP: Vision language planning for autonomous driv- ing. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14760–14769,

  29. [37]

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V . Jawahar. Cats and dogs. InIEEE Conference on Computer Vision and Pattern Recognition, 2012. 5, 3

    2012

  30. [38]

    Regularizing neural networks by pe- nalizing confident output distributions

    Gabriel Pereyra et al. Regularizing neural networks by pe- nalizing confident output distributions. InInternational Con- ference on Learning Representations (ICLR), 2017. 2

    2017

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InInternational Conf...

    2021

  32. [40]

    Do ImageNet classifiers generalize to Ima- geNet? InProceedings of the 36th International Conference on Machine Learning, pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to Ima- geNet? InProceedings of the 36th International Conference on Machine Learning, pages 5389–5400. PMLR, 2019. 5, 3

    2019

  33. [41]

    O-TPT: Orthogonality constraints for calibrating test-time prompt tuning in vision-language models

    Ashshak Sharifdeen, Muhammad Akhtar Munir, Sanoojan Baliah, Salman Khan, and Muhammad Haris Khan. O-TPT: Orthogonality constraints for calibrating test-time prompt tuning in vision-language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages ...

    2025

  34. [42]

    Test- time prompt tuning for zero-shot generalization in vision- language models.Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models.Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022. 1

    2022

  35. [43]

    Few-shot, now for real: Medi- cal VLMs adaptation without balanced sets or validation

    Julio Silva-Rodr ´ıguez, Fereshteh Shakeri, Houda Bahig, Jose Dolz, and Ismail Ben Ayed. Few-shot, now for real: Medi- cal VLMs adaptation without balanced sets or validation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 237–...

  36. [44]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012

    Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.ArXiv, abs/1212.0402, 2012. 5, 3

    2012 arXiv

  37. [45]

    An empirical study into what matters for calibrating vision-language models

    Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating vision-language models. InInternational Con- ference on Machine Learning, pages 48791–48808. PMLR,

  38. [46]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. InAdvances in Neural Information Processing Systems, pages 10506–10518, 2019. 5, 3

    2019

  39. [47]

    Open-vocabulary calibration for fine-tuned CLIP

    Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned CLIP. InInternational Conference on Machine Learning, pages 51734–51754. PMLR, 2024. 3

    2024

  40. [48]

    Sun database: Large-scale scene 10 recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene 10 recognition from abbey to zoo. InIEEE Conference on Com- puter Vision and Pattern Recognition, 2010. 5, 3

    2010

  41. [49]

    Demysti- fying CLIP data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Rus- sell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demysti- fying CLIP data. InThe Twelfth International Conference on Learning Representations, 2024. 1

    2024

  42. [50]

    Beyond probability partitions: Calibrating neural networks with semantic aware grouping.Advances in Neural Information Processing Systems, 36:58448–58460,

    Jia-Qi Yang et al. Beyond probability partitions: Calibrating neural networks with semantic aware grouping.Advances in Neural Information Processing Systems, 36:58448–58460,

  43. [51]

    Hasegawa-Johnson, Yingzhen Li, and Chang D

    Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Mark A. Hasegawa-Johnson, Yingzhen Li, and Chang D. Yoo. C- TPT: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. InThe Twelfth Interna- tional Conference on Learning Representations, 20...

    2024

  44. [52]

    Cutmix: Regular- ization strategy to train strong classifiers with localizable fea- tures

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular- ization strategy to train strong classifiers with localizable fea- tures. InProceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 6023–60...

    2019

  45. [53]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. InInternational Conference on Learning Representa- tions (ICLR), 2018. 2

    2018

  46. [54]

    Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning

    Jize Zhang et al. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. InIn- ternational conference on machine learning (ICML), 2020. 2

    2020

  47. [55]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Conditional prompt learning for vision-language models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  48. [56]

    Learning to prompt for vision-language models.Inter- national Journal of Computer Vision (IJCV), 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.Inter- national Journal of Computer Vision (IJCV), 2022. 2, 7, 3 11 SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning Supplementary Material A. Proof of Pr...

    2022

  49. [2022]

    Curran Associates Inc. 2

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.