Pith. sign in

REVIEW 5 major objections 6 minor 68 references

Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CalShift claims a pair of regularizers added to CLIP's contrastive loss can fix both covariate shift and overconfident predictions in few-shot vision-language classification.

desk verdict Plausible regularizer combo for CLIP few-shot tuning, but the paper never tests the combined loss it advertises, and the zero-shot row is impossible. read the letter →

arxiv 2502.07847 v2 pith:KYDQQPHH submitted 2025-02-11 cs.CV cs.LG

Behraj Khan , Rizwan Qureshi , Nouman Muhammad Durrani , Tahir Syed This is my paper · ORCID
classification cs.CVcs.LG
keywords few-shotclassificationvision-languagemodelsCLIPcovariateshiftconfidencecalibrationexpectederrorpromptlearningFisherinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CalShift claims that a single regularized loss can fix two problems that plague CLIP-based few-shot classification at once: covariate shift between pre-training and target data, and overconfident wrong predictions. The loss adds a Fisher information penalty to flatten the loss landscape and resist distribution shift, plus a Confidence Misalignment Penalty to push probability mass back toward the true class, inside CoOp prompt learning. Across 19 vision and domain-shift benchmarks, the paper reports that the Fisher penalty improves accuracy by up to 7.2% over CoOp and the confidence penalty reduces Expected Calibration Error by up to 12.5%, with the headline covariate-shift gains being 3.5% accuracy and 5.82% ECE. If these effects hold jointly, CalShift would offer a simple drop-in regularizer for more reliable low-shot vision-language models.

What carries the argument

The machinery is a two-term regularizer added to the contrastive loss: the Fisher information $I(\theta) = -\mathbb{E}[\partial^2 \log P(X; \theta)/\partial \theta^2]$, the negative Hessian of the log-likelihood, which the paper treats as a curvature penalty that stabilizes the model under distribution shift; and the Confidence Misalignment Penalty $\text{CMP} = P(x,y)\sum_{y' \neq y: P(x,y') > P(x,y)} P(x,y')$, which sums the excess softmax probability assigned to wrong classes that outrank the true class. The combined loss $L_{\text{CalShift}} = L_c + \lambda_1 I(\theta) + \lambda_2 \text{CMP}$ is the object that is supposed to carry both corrections at once.

What would settle it

Run the joint loss $L_{\text{CalShift}} = L_c + \lambda_1 I(\theta) + \lambda_2 \text{CMP}$ with both $\lambda_1 > 0$ and $\lambda_2 > 0$ on the same benchmarks and compare against each penalty alone; the paper never reports this configuration, so if the joint model fails to improve over the isolated penalties on both accuracy and ECE, the central claim of simultaneous correction collapses. A second check is to compute the actual Hessian of the CLIP text encoder's log-likelihood and measure whether minimizing it actually reduces a direct covariate-shift metric such as KL divergence between source and target feature distributions, which the paper's Proposition A.1 only approximates locally.

Watch

Extended reading notes

Core claim

The paper's central claim is that covariate shift and confidence misalignment are coupled failure modes, and both can be corrected in the prompt-learning setting by minimizing $L_{\text{CalShift}} = L_c + \lambda_1 I(\theta) + \lambda_2 \text{CMP}$. The Fisher information term, defined as the negative Hessian of the log-likelihood, is argued to penalize sharp curvature and tighten a PAC-Bayes generalization bound, while the CMP term redistributes softmax likelihood from overconfident incorrect classes to the true class. Empirically, CoOp+FIM beats CoOp on accuracy across all shot counts and most datasets, and CoOp+CMP reduces ECE on nearly all of them; the paper interprets this as evidence that the two penalties are complementary and that their combination improves both robustness and calibration.

Load-bearing premise

The load-bearing premise is that the Fisher information $I(\theta)$ can be computed for the CLIP text encoder and that minimizing it genuinely corrects covariate shift, yet the paper gives no algorithm for computing the Hessian and its appendix proof only establishes a local approximation.

Editorial extensions

If this is right

  • On ImageNet, adding the Fisher penalty to CoOp raises average accuracy from 79.8% to 84.5% across 0 to 16 shots, with the largest relative gain of 7.2% at 1 shot.
  • Adding the CMP penalty lowers average ECE on ImageNet from 2.90% to 2.62%, a 9.66% relative reduction, and the improvement grows as shot count increases.
  • On eight covariate-shift benchmarks (PACS, Office-Home, VLCS, DomainNet, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R), CoOp+FIM improves accuracy on every dataset, up to 3.5%, and CoOp+CMP lowers ECE on every dataset, up to 5.82%.
  • The Fisher term acts as a prior-preserving regularizer since it also improves zero-shot accuracy over CoOp, suggesting it protects pre-trained knowledge rather than only helping few-shot fitting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the paper evaluates FIM and CMP only in isolation, the reported 5.82% ECE and 3.5% accuracy numbers are not evidence that the full $L_{\text{CalShift}}$ loss with both penalties active reproduces them; a joint experiment is the direct next test.
  • Editorial inference: The theoretical link between $I(\theta)$ and covariate shift rests on a local quadratic approximation (Proposition A.1); an empirical check would be to compute both $I(\theta)$ and a direct shift measure such as KL divergence between source and target feature distributions and see whether they rank shifted benchmarks the same way.
  • Editorial inference: If the combined regularizer works, it should transfer to other prompt-tuning or adapter methods beyond CoOp, such as class-specific prompts or multi-modal prompt tuning, since neither penalty is specific to CoOp's unified-context design.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CalShift, a unified loss for few-shot classification with CLIP-based prompt learning. The loss is L_CalShift = Lc + λ1I(θ) + λ2CMP, where I(θ) is a Fisher information penalty intended to correct covariate shift and CMP is a confidence misalignment penalty intended to reduce overconfidence. The authors claim this unified objective improves accuracy by up to 3.5% on covariate-shift datasets and reduces Expected Calibration Error by up to 5.82%. Experiments are reported on ImageNet few-shot settings, eleven vision datasets, and eight covariate-shift datasets, always comparing CoOp with either CoOp+FIM or CoOp+CMP against a vanilla CoOp baseline.

Significance. If the unified regularizer actually delivered the claimed simultaneous gains in accuracy and calibration, it would be a practically useful addition to prompt learning for vision-language models. The paper's motivation is reasonable, and the experimental scope is broad: 19 datasets and multiple shot settings are considered, and the ablations separately study each penalty. However, the paper's central claim is not tested: no experiment ever runs the full CalShift objective with both penalties active. Moreover, the zero-shot rows in Table 1 are logically impossible for a training-time regularizer, the hyperparameters are tuned on datasets that also appear in the main tables, and the theoretical propositions are not valid as stated. The strengths are the breadth of the evaluation and the clarity of the single-penalty ablations, but these do not compensate for the absence of evidence for the method as proposed.

major comments (5)
  1. [§4.2, Eq. (3), Tables 1–3] The headline claim, stated in the abstract and §1, is that the unified loss L_CalShift = Lc + λ1I(θ) + λ2CMP simultaneously corrects covariate shift and overconfidence. No experiment evaluates this objective. The accuracy rows of Tables 1–3 compare 'CoOp + FIM' to CoOp (i.e., λ2=0), the ECE rows compare 'CoOp + CMP' to CoOp (i.e., λ1=0), and the ablations in Appendix B are explicit that either λ1=0 or λ2=0. The assertion in §3 that 'neither penalty degrades the other's effect' is therefore untested; the reported 5.82% ECE reduction and 3.5% accuracy gain belong to two separate single-penalty methods, not to CalShift. A combined run with both penalties active on the same benchmarks is required to support the paper's title claim.
  2. [Table 1, zero-shot rows] In the 0-shot column of Table 1, CoOp+FIM reports 84.9% versus 79.5% for CoOp, and CoOp+CMP reports ECE 3.06 versus 3.36. In a genuine zero-shot setting no training examples are available to update prompts or to compute either penalty, so these entries cannot differ from the CoOp baseline. Their difference indicates either a protocol inconsistency (e.g., the penalty computed on some held-out data, or the row not actually zero-shot) or a reporting error. Because the same table is used to claim that FIM 'acts as a prior-preserving regularizer' in zero-shot scenarios, this issue directly undermines the experimental evidence.
  3. [Appendix B, Tables 6–7; §4.1] The hyperparameters λ1 and λ2 are tuned on the same datasets that appear in the main results: Tables 6 and 7 report λ1 and λ2 selection on Flowers102, Food101, UCF101, and DTD, and these four datasets are also included in the main evaluation tables (Tables 2, 4, and 5). The paper does not describe a held-out validation split. The reported improvements are therefore at risk of being artifacts of test-set hyperparameter selection; an independent validation split or a sensitivity analysis that does not reuse evaluation data is needed.
  4. [Appendix A, Propositions A.1 and A.2] The theoretical support is not established as stated. Proposition A.1 claims equality between Fisher information and KL divergence, but the proof only derives a local quadratic approximation DKL ≈ 1/2 (θq−θ)^T I(θ)(θq−θ); no equality is shown, and the assumptions under which the approximation holds are not stated. Proposition A.2's definition of CMP changes between Eq. (2) and the proof (in Eq. (2), y is the true class and y′ the incorrect class, while in the proof y′ is the predicted class), and the limits P(x,y′)→1 and P(x,y′)→0 do not establish the claimed bounds 0≤CMP≤1. In addition, no algorithm is given for computing I(θ) for the CLIP text encoder; Eq. (1) is a definition, not an estimation procedure. Since these propositions are used to justify the loss in Eq. (3), the theoretical argument needs to be corrected or removed.
  5. [§4.2, Table 2] Table 2 contains accuracy degradations that are not acknowledged: CoOp+FIM decreases accuracy on StanfordCars (59.2→54.3, ∆=8.2↓) and EuroSAT (53.8→49.2, ∆=8.6↓). The text says 'improvement across most datasets' but elsewhere claims 'CalShift demonstrates robust performance improvement' and lists only positive cases. The average 3.2% gain is driven by large gains on some datasets; the negative results should be discussed and the robustness claim tempered.
minor comments (6)
  1. [§3, Eq. (2)] The notation P(x,y) and P(x,y′) is never formally defined; since CLIP operates on image-text pairs, the meaning of x and y in a softmax probability over classes should be stated explicitly.
  2. [§1 and §5] The introduction and conclusion describe CalShift as 'training-free adaptation', but the method is integrated into CoOp, which learns prompts from training examples; this is contradictory and should be clarified.
  3. [Table 2] The blocks in Table 2 are arranged with ECE first and ACC second, while the caption says the upper half is accuracy and the lower half is ECE; the order should be corrected.
  4. [Throughout] The manuscript contains numerous typos and grammatical errors, including 'addresse', 'covriate', 'Calshift', 'confidene', 'emprical', 'ststistics', and 'remains always'; these should be corrected in a careful editing pass.
  5. [§3, Proposition 3.1] Proposition 3.1 is not a proposition in the mathematical sense: it restates the intended effect of the two penalties and is followed by no proof; it should be reworded as a design statement rather than a formal claim.
  6. [§4.1, Implementation details] The paper does not report the number of random seeds, the variance across runs, or the exact few-shot training protocol used for CoOp, which makes the numerical differences in the tables difficult to assess.

Circularity Check

3 steps flagged · score 6.0 of 10

CalShift's headline gains come from single-penalty runs (FIM-only and CMP-only), the unified loss of Eq. 3 is never evaluated, and the Fisher-information premise is carried by the authors' own prior work with hyperparameters selected on the evaluation datasets.

  1. fitted input called prediction [Appendix B, Tables 6 and 7; main results in Tables 2 and 4]
    "Table 6 given in appendix B shows accuracy performance on four datasets, including Flowers 102, Food 101, UCF 101, and DTD, for different values of lambda1 within the range (0.0 to 1.0), while lambda2 remains fixed at 0. ... The results in the table indicate that CalShift achieves the highest performance at lambda1 = 0.4 ... In appendix Table 7 shows the calibration results of CalShift when tuning lambda2 within range (0.0 to 1.0) while lambda1 value fixed to 0. It is show in Table that with lambda2 = 0.4 CalShift achieve better calibration results across four dataset."

    The regularization strengths lambda1 = 0.4 and lambda2 = 0.4 are selected by running on Flowers102, Food101, UCF101, and DTD, and those same four datasets appear in the main evaluation tables (e.g., Table 2 and the ablation Tables 4/5). The reported CalShift gains on these datasets are therefore not independent predictions; they are the result of fitting the method's free parameters to the evaluation data. No held-out validation split is described, so the 'improvement' on these datasets is partly a fitted input presented as a measured result.

  2. self citation load bearing [Section 3, 'Fisher information in covariate shift'; Proposition A.1 in Appendix A]
    "Recently, the FI matrix was used to compute and by consequence, help correct, divergence in data distributions under a batched-streaming setting [27]. We are possibly the first to borrow the idea into the VLM post-training regime. ... Proposition A.1 The Fisher information I(theta) is equal to the Kullback-Leibler (KL) divergence DKL between the source distribution p(X) and target distribution q(X), providing a measure of covariate shift."

    Reference [27] is the authors' own prior work (Khan, Mirza, and Syed, ICLR 2024 Tiny Papers). It is the only cited basis for the load-bearing premise that minimizing the Fisher information I(theta) corrects covariate shift. The paper's own Proposition A.1 does not supply independent support: its proof ends at the local approximation DKL(q(X) || p(X;theta)) approximately 1/2 (theta_q - theta)^T I(theta) (theta_q - theta), which is not the asserted equality. Thus the central mechanism for the FIM component reduces to an assumption imported from the authors' own earlier paper rather than an externally established fact.

1 more flagged steps
  1. other [Section 4.2 and Tables 1-3; Equation (3); Abstract]
    "LCalShift(x, y; theta) = Lc + lambda1 I(theta) + lambda2 CMP (Eq. 3). ... The upper part shows CalShift accuracy results on covariate shift vision datasets with and without FIM penalty. ... The lower part shows CalShift ECE results on covariate shift vision datasets with and without CMP penalty. ... Table 1 shows that FIM integration significantly enhances generalization, particularly in low-shot regimes, while CMP integration improves calibration in these settings. These results establish CalShift as a versatile regularizer."

    The proposed method is defined as the sum of both penalties (Eq. 3), and the abstract claims CalShift 'achiev[es] up to a 5.82% reduction in Expected Calibration Error' and 'improv[es] accuracy by 3.5%'. But every reported result isolates one penalty: the accuracy rows use CoOp+FIM (lambda2 = 0) and the ECE rows use CoOp+CMP (lambda1 = 0). The 5.82% ECE reduction is the CMP-only result in Table 3 (Office-Home), and the 3.5% accuracy gain is the FIM-only result in Table 3 (Office-Home/VLCS). No run evaluates Eq. 3 with both lambda1 and lambda2 nonzero, so the unified CalShift loss is never tested.

full rationale

The strongest circularity concerns are concentrated in the experimental presentation rather than in a single equation. First, the hyperparameters lambda1 and lambda2 are tuned on the same datasets that later appear in the main evaluation tables, so some of the reported gains are fitted values presented as predictions. Second, the load-bearing justification for the Fisher-information penalty is the authors' own prior work [27], and the paper's Proposition A.1 only proves a local quadratic approximation, not the equality claimed; the covariate-shift-correcting role of I(theta) therefore rests on a self-citation rather than on independent mathematical support. Third, and most importantly, the paper's central claim is for the unified loss LCalShift = Lc + lambda1 I(theta) + lambda2 CMP, yet all experiments disable one of the two penalties, with accuracy reported for CoOp+FIM and ECE reported for CoOp+CMP. The headline numbers are thus not measurements of CalShift itself but of its two separately evaluated components, and the assertion that 'neither penalty degrades the other's effect' is never tested. These issues make the central contribution partially circular: the unified result reduces by construction to the sum of two untested-in-combination penalties, one of which is justified by the authors' own prior work. Because the paper does compare against external baselines and benchmarks, the circularity is not total, but it is substantial enough to warrant a score of 6.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central method's two building blocks come from the authors' own prior papers [26] and [27]. The theoretical justifications for both blocks are asserted with faulty proofs, and the two hyperparameters were selected on datasets that later appear in the evaluation, so the reported gains are partly fitted. No new entities are introduced.

free parameters (2)
  • lambda_1 (Fisher penalty weight) = 0.4
    Selected by tuning on Flowers102, Food101, UCF101, and DTD (Appendix B, Table 6); these datasets are also included in the main evaluation table (Table 2), so the tuning set overlaps the test set.
  • lambda_2 (CMP weight) = 0.4
    Selected by tuning on Flowers102, Food101, UCF101, and DTD for ECE (Appendix B, Table 7), again overlapping the evaluation set.
assumptions (5)
  • ad hoc to paper Fisher information I(θ) is equal to the KL divergence between source and target distributions (Proposition A.1).
    The stated equality is not true in general; the proof only gives a local quadratic approximation D_KL ≈ 0.5 (θ_q − θ)^T I(θ)(θ_q − θ). The paper relies on this to claim covariate shift correction.
  • ad hoc to paper A PAC-Bayes-style bound of the form E_gen ≤ E_emp + I(θ) + log(1/δ)/(2n) holds for the CLIP model.
    No such standard bound ties generalization error directly to Fisher information; the formula is asserted in Section 3 without reference and has incorrect scaling.
  • ad hoc to paper CMP is a valid calibration penalty with 0 ≤ CMP ≤ 1 and CMP approaches 1 when the model is overconfident (Proposition A.2).
    The main-text definition (Eq. 2) and the appendix definition conflict; under the appendix formula CMP can be 0 when a wrong class is highly confident, contradicting the claim.
  • domain assumption Penalizing I(θ) corrects covariate shift in the VLM prompt-learning setting.
    Carried over from the authors' prior work [27] on batched-streaming settings; no derivation shows this applies to CoOp prompt learning.
  • domain assumption The conditional distribution P(y|x) is unchanged under covariate shift, while only P(x) shifts.
    Stated in Section 1; standard but unverified for the CLIP feature space.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models." pith.science (2026). https://pith.science/paper/KYDQQPHH

@misc{pith2026250207847,
  author       = {Pith},
  title        = {Pith review of: Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYDQQPHH}},
  note         = {Machine review of arXiv:2502.07847}
}
read the original abstract

Since the establishment of vision-language foundation models as the new mainstay in low-shot vision classification tasks, the question of domain generalization arising from insufficient target data is assuming more importance. This scarcity challenge induces sampling bias and amplifies model sensitivity to variations and shifts in data distributions. While fine-tuning on multiple domains could mitigate such domain generalization issues, it is resource-intensive and demands diverse data sources. In this work, we systematically analyze two critical challenges: (1) covariate shift between the pre-training distribution and the underspecified target distribution, and (2) confidence misalignment, where predictions on novel data are overconfident. To address both challenges simultaneously, we introduce \textbf{Confidence-Calibrated Covariate Shift Correction (CalShift)} -- a unified approach that combines a Fisher information penalty to mitigate covariate shift and a Confidence Misalignment Penalty (CMP) to reduce overconfidence in misclassified examples. Experimental evaluations across various vision and covariate shift benchmarks demonstrate that CalShift significantly improves model calibration, achieving up to a 5.82\% reduction in Expected Calibration Error (ECE). Furthermore, CalShift enhances robustness, improving accuracy by 3.5\% on challenging datasets impacted by covariate shifts. Our results highlight CalShift as a promising strategy for building robust and reliable low-shot vision-language systems for real-world applications.

Figures

Figures reproduced from arXiv: 2502.07847 by the authors.

Figure 1
Figure 1. Workflow of the proposed CalShift framework: The sub-figure (a) illustrates the confidence misalignment problem caused by [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 34 canonical work pages

  1. [26]

    Technical report on label- informed logit redistribution for better domain generaliza- tion in low-shot classification with foundation models.arXiv preprint arXiv:2501.17595, 2025

    Behraj Khan and Tahir Syed. Technical report on label- informed logit redistribution for better domain generaliza- tion in low-shot classification with foundation models.arXiv preprint arXiv:2501.17595, 2025. 1, 2, 3, 4

    arXiv 2025

  2. [27]

    Causal covari- ate shift correction using fisher information penalty

    Behraj Khan, Behroz Mirza, and Tahir Syed. Causal covari- ate shift correction using fisher information penalty. In The Second Tiny Papers Track at ICLR 2024. 1, 4

    work page 2024

  3. [1]

    Pattern recognition letters, 31(14): 2214–2224, 2010

    Adaptive filtering and hypothesis testing: Application to can- cerous cells detection. Pattern recognition letters, 31(14): 2214–2224, 2010. 4

    work page 2010

  4. [2]

    Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization

    Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hus- sein, Muhammad Uzair Khattak, Muhammad Muzammal Naseer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time prompting with distribution align- ment for zero-shot generalization. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1

    work page 2024

  5. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  6. [4]

    Focused anchors loss: Cost- sensitive learning of discriminative features for imbalanced classification

    Bahram K Baloch, Sateesh Kumar, Sanjay Haresh, Abeerah Rehman, and Tahir Syed. Focused anchors loss: Cost- sensitive learning of discriminative features for imbalanced classification. In Asian Conference on Machine Learning , pages 822–835. PMLR, 2019. 3

    work page 2019

  7. [5]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 5

    2014

  8. [6]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 5

    2014

Show all 68 references
  1. [7]

    Domain adaptation and sample bias correction theory and algorithm for regression

    Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103–126, 2014. 1

    2014

  2. [8]

    Monotonicity of entropy and fisher information: a quick proof via maximal correlation

    Thomas A Courtade. Monotonicity of entropy and fisher information: a quick proof via maximal correlation. arXiv preprint arXiv:1610.04174, 2016. 4

    2016 arXiv

  3. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  4. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 5

    2010 arXiv

  5. [11]

    A video saliency detection model in com- pressed domain

    Yuming Fang, Weisi Lin, Zhenzhong Chen, Chia-Ming Tsai, and Chia-Wen Lin. A video saliency detection model in com- pressed domain. IEEE transactions on circuits and systems for video technology, 24(1):27–38, 2013. 5

    2013

  6. [12]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004. 5

    2004

  7. [13]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050–1059. PMLR, 2016. 1

    2016

  8. [14]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2): 581–595, 2024. 3

    2024

  9. [15]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR,

  10. [16]

    Model selection and the princi- ple of minimum description length

    Mark H Hansen and Bin Yu. Model selection and the princi- ple of minimum description length. Journal of the american statistical association, 96(454):746–774, 2001. 5

    2001

  11. [17]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  12. [18]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5

    2019

  13. [19]

    Generalization bounds: Perspec- tives from information theory and pac-bayes

    Fredrik Hellstr ¨om, Giuseppe Durisi, Benjamin Guedj, Maxim Raginsky, et al. Generalization bounds: Perspec- tives from information theory and pac-bayes. Foundations and Trends® in Machine Learning, 18(1):1–223, 2025. 4

    2025

  14. [20]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. Proceedings of the International Conference on Learn- ing Representations, 2019. 3

    2019

  15. [21]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF internation...

    2021

  16. [22]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 15262–15271, 2021. 5

    2021

  17. [23]

    Vaccine: Perturbation-aware alignment for large language mod- els against harmful fine-tuning attack

    Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language mod- els against harmful fine-tuning attack. arXiv preprint arXiv:2402.01109, 2024. 1, 3

    2024 arXiv

  18. [24]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  19. [25]

    Calibrated lan- guage models must hallucinate

    Adam Tauman Kalai and Santosh S Vempala. Calibrated lan- guage models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing , pages 160–171, 2024. 5

    2024

  20. [28]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023. 3

    2023

  21. [29]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 5

    2013

  22. [30]

    Verified un- certainty calibration

    Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified un- certainty calibration. Advances in Neural Information Pro- cessing Systems, 32, 2019. 1

    2019

  23. [31]

    Theory of point esti- mation

    Erich L Lehmann and George Casella. Theory of point esti- mation. Springer Science & Business Media, 2006. 2, 4

    2006

  24. [32]

    Deeper, broader and artier domain generaliza- tion

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generaliza- tion. In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017. 5

    2017

  25. [33]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 5

    2013 arXiv

  26. [34]

    Lecture notes on advanced statistical theory

    Ryan Martin. Lecture notes on advanced statistical theory. Supplement to the lectures for Stat, 511, 2016. 4

    2016

  27. [35]

    Study on the impact of partition-induced dataset shift on k-fold cross-validation

    Jose Garc ´ıa Moreno-Torres, Jos´e A S´aez, and Francisco Her- rera. Study on the impact of partition-induced dataset shift on k-fold cross-validation. IEEE transactions on neural net- works and learning systems, 23(8):1304–1312, 2012. 4

    2012

  28. [36]

    When does label smoothing help? Advances in neural in- formation processing systems, 32, 2019

    Rafael M ¨uller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural in- formation processing systems, 32, 2019. 3

    2019

  29. [37]

    Robust calibration of large vision- language adapters

    Balamurali Murugesan, Julio Silva-Rodr ´ıguez, Ismail Ben Ayed, and Jose Dolz. Robust calibration of large vision- language adapters. In European Conference on Computer Vision, pages 147–165. Springer, 2025. 1, 3, 4

    2025

  30. [38]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 5

    2008

  31. [39]

    Blackvip: Black-box visual prompting for robust transfer learning

    Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2...

    2023

  32. [40]

    Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshmi- narayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information proces...

    2019

  33. [41]

    Be confident in what you know: Bayesian parameter efficient fine-tuning of vision foundation models

    Deep Shankar Pandey, Spandan Pyakurel, and Qi Yu. Be confident in what you know: Bayesian parameter efficient fine-tuning of vision foundation models. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys- tems, 2024. 3, 4

    2024

  34. [42]

    Prevalence of neural collapse during the terminal phase of deep learning training

    Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652–24663, 2020. 4

    2020

  35. [43]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5

    2012

  36. [44]

    Regularizing neural networks by penalizing confident output distributions

    Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017. 3

    2017 arXiv

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400. PMLR, 2019. 5

    2019

  39. [47]

    Focal loss for dense ob- ject detection

    T-YLPG Ross and GKHP Doll ´ar. Focal loss for dense ob- ject detection. In proceedings of the IEEE conference on computer vision and pattern recognition, pages 2980–2988,

  40. [48]

    Improving predictive inference un- der covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. Improving predictive inference un- der covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244,

  41. [49]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 ,

    arXiv

  42. [50]

    Input- dependent estimation of generalization error under covariate shift

    Masashi Sugiyama and Klaus-Robert M ¨uller. Input- dependent estimation of generalization error under covariate shift. 2005. 1

    2005

  43. [51]

    Covariate shift adaptation by importance weighted cross validation

    Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert M¨uller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8 (5), 2007. 4

    2007

  44. [52]

    On mixup train- ing: Improved calibration and predictive uncertainty for deep neural networks

    Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup train- ing: Improved calibration and predictive uncertainty for deep neural networks. Advances in neural information processing systems, 32, 2019. 3

    2019

  45. [53]

    Toward a holis- tic evaluation of robustness in clip models

    Weijie Tu, Weijian Deng, and Tom Gedeon. Toward a holis- tic evaluation of robustness in clip models. arXiv preprint arXiv:2410.01534, 2024. 3, 4

    2024

  46. [54]

    Sparkr: Scal- ing r programs with spark

    Shivaram Venkataraman, Zongheng Yang, Davies Liu, Eric Liang, Hossein Falaki, Xiangrui Meng, Reynold Xin, Ali Ghodsi, Michael Franklin, Ion Stoica, et al. Sparkr: Scal- ing r programs with spark. In Proceedings of the 2016 Inter- national Conference on Management of Data , pag...

    2016

  47. [55]

    Calibration in deep learning: A survey of the state-of-the-art

    Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222, 2023. 3

    2023 arXiv

  48. [56]

    Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power.Advances in Neural Information Pro- cessing Systems, 32, 2019. 5

    2019

  49. [57]

    Under- standing and mitigating miscalibration in prompt tuning for vision-language models

    Shuoyuan Wang, Yixuan Li, and Hongxin Wei. Under- standing and mitigating miscalibration in prompt tuning for vision-language models. arXiv preprint arXiv:2410.02681,

    arXiv

  50. [58]

    Open-vocabulary calibration for fine-tuned clip

    Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open-vocabulary calibration for fine-tuned clip. In Forty-first International Conference on Machine Learning, 2024. 1, 2, 3

    2024

  51. [59]

    Pac-bayes information bottleneck

    Zifeng Wang, Shao-Lun Huang, Ercan E Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. Pac-bayes information bottleneck. arXiv preprint arXiv:2109.14509, 2021. 4

    2021 arXiv

  52. [60]

    Mitigating overconfidence in large language models: A behavioral lens on confidence estimation and calibration

    Bingbing Wen, Chenjun Xu, HAN Bin, Robert Wolfe, Lucy Lu Wang, and Bill Howe. Mitigating overconfidence in large language models: A behavioral lens on confidence estimation and calibration. In NeurIPS 2024 Workshop on Behavioral Machine Learning. 1

    2024

  53. [61]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vi- ...

    2022

  54. [62]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 5

    2010

  55. [63]

    Any-shift prompt- ing for generalization over distributions

    Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees GM Snoek. Any-shift prompt- ing for generalization over distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13849–13860, 2024. 1

    2024

  56. [64]

    When and how mixup improves calibration

    Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In In- ternational Conference on Machine Learning, pages 26135– 26160. PMLR, 2022. 3

    2022

  57. [65]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling. arXiv preprint arXiv:2111.03930, 2021. 3

    2021 arXiv

  58. [66]

    Large lan- guage models as commonsense knowledge for large-scale task planning

    Zirui Zhao, Wee Sun Lee, and David Hsu. Large lan- guage models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024. 5

    2024

  59. [67]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  60. [2022]

    3, 5 A. Theoretical Background Proposition A.1 The Fisher information I(θ)is equal to the Kullback-Leibler (KL) divergence DKL between the source distribution p(X) and target distribution q(X), pro- viding a measure of covariate shift. Proof A.1 Let p(X; θ) is probability dens...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.