Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard few-shot CLIP benchmarks overstate inductive generalization because CLIP has already seen most test classes; unlearning those classes makes thirteen baselines drop by 55% on average.

desk verdict A real problem and a clever pipeline, but the unlearning validity needs stronger evidence before the benchmark is trusted. read the letter →

arxiv 2507.20834 v1 pith:MUKX6WU6 submitted 2025-07-28 cs.CV

Alexey Kravets , Da Chen , Vinay P. Namboodiri This is my paper
classification cs.CV
keywords few-shotclassificationCLIPinductiveevaluationmachineunlearningprompttuningvision-languagemodelsbenchmarkcontaminationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard few-shot CLIP benchmarks are not measuring what they claim. Because CLIP was pretrained on a huge, undisclosed corpus, the classes in popular evaluation datasets have almost certainly been seen during pretraining, so high few-shot scores may come from the model remembering those classes rather than learning them from the few support examples. The authors build an inductive benchmark by unlearning the target dataset's classes from CLIP before running few-shot methods, and report that thirteen existing methods lose about 55% of their accuracy on average in this setting, with CoOp falling from 71.4% to 15.3%. They validate the unlearning by comparing against CLIP models trained from scratch on ImageNet with and without a subset of classes, finding similar behavior, and they propose a new method, SEPRES, that retains much more of its performance after unlearning. If correct, the paper implies that many published few-shot CLIP results overstate generalization to truly unseen classes.

What carries the argument

The machinery has two parts. The first is the unlearned CLIP checkpoint: Selective Synaptic Dampening (SSD) computes Fisher information over a forget set (the target benchmark classes) and a retain set (other datasets), then shrinks parameters that are important for the forget set but not for the retain set. This produces a model that scores near zero on the target classes while preserving accuracy on held-out validation datasets, and it is this checkpoint that turns any standard dataset into an inductive benchmark. The second part is the proposed SEPRES method (Self-Enhanced Prompt Tuning with Residual Textual Features), which combines SEP's token-fusion prompts with learnable residual terms added to the final textual classifier weights, so the model can store knowledge that is not constrained by CLIP's frozen internal representations. An auxiliary piece is Total Knowledge Loss, a weighted average of accuracy degradation on validation datasets, used to control how much general knowledge is lost during unlearning.

What would settle it

Train two CLIP models from scratch on the same corpus, one with a set of fine-grained classes included and one with those classes excluded; run SSD unlearning on the first and compare few-shot accuracies across many datasets and shots. If the unlearned model's per-dataset results do not track the never-seen model within a small margin, or if membership-inference attacks still identify the unlearned classes as present, the inductive-benchmark claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the standard evaluation protocol for few-shot CLIP classification is partially transductive rather than inductive: CLIP's pretraining corpus very likely contains the classes of popular benchmarks, so the model carries class-specific knowledge into the support set before any few-shot adaptation happens. After applying Selective Synaptic Dampening (SSD) to remove knowledge of the target dataset's classes, all thirteen baseline methods show a large collapse in accuracy, with the average drop across datasets and shots around 55%. This collapse is presented as evidence that most of the apparent few-shot learning was actually retrieval of already-encoded class information. The paper further claims that unlearning approximates never having seen the classes: in oracle experiments with CLIP trained from scratch on ImageNet, unlearning a coarse subset (dogs, birds, vehicles) yields few-shot results close to training from scratch without that subset. On the same inductive benchmarks, the proposed SEPRES method, which adds learnable residual parameters to the final textual embeddings of a prompt-tuning method, consistently outperforms the thirteen baselines.

Load-bearing premise

The load-bearing premise is that unlearning the target classes from CLIP yields a model that behaves like a CLIP never trained on those classes, an equivalence the paper checks only on three coarse ImageNet subsets.

Editorial extensions

If this is right

  • Published few-shot CLIP numbers should be read as partially transductive scores; a truly inductive evaluation would require a backbone that has not seen the target classes.
  • Method rankings can flip between the two settings, so a method that wins on standard benchmarks is not necessarily the best choice for genuinely novel classes.
  • Any dataset and any model with a reliable unlearning method can be converted into an inductive benchmark, making the pipeline a reusable evaluation tool.
  • Performance degrades further as unlearning removes more general knowledge, quantifying how much each method depends on pretrained memory rather than few-shot learning.
  • SEPRES's smaller drop after unlearning indicates that adding learnable residual textual features is a viable direction for building few-shot methods that work on unseen classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same unlearning protocol could audit other foundation models, such as large language models, whose pretraining corpora are also undisclosed; the paper names this direction but does not test it.
  • The oracle validation covers only three coarse ImageNet subsets, so the equivalence between unlearning and never seeing the classes is an extrapolation; extending it to many fine-grained classes would directly test whether the 55% drop is universal.
  • One could attempt to prove the leakage directly by running membership-inference or zero-shot retrieval probes on the actual benchmark classes before unlearning, rather than inferring leakage from the accuracy collapse after unlearning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that standard few-shot CLIP benchmarks are partially transductive because CLIP's pretraining corpus almost certainly contained the benchmark datasets, so few-shot methods can exploit previously seen class knowledge. To measure true inductive generalization, the authors propose an evaluation pipeline that first applies selective synaptic dampening (SSD) to unlearn the target dataset classes from CLIP, then evaluates few-shot methods on the unlearned model. They report that 13 recent baselines drop by 55% on average in this inductive setting (e.g., CoOp drops from 71.4% to 15.3%), and they propose SEPRES, a prompt-tuning method with residual textual features, which performs substantially better in both inductive and transductive settings. The unlearning pipeline is validated through oracle experiments in which CLIP is trained from scratch on ImageNet and compared between excluding a subset from training and unlearning that subset after training.

Significance. The concern the paper raises is important and widely applicable: if CLIP has indeed seen the benchmark classes, many published few-shot CLIP results are inflated. The proposed pipeline is a practical, general recipe for constructing inductive benchmarks for any model with a reliable unlearning method, and the experimental scale (5,880 experiments across 14 methods, 7 datasets, 5 shot counts, and 4 knowledge-loss levels) is a clear strength. SEPRES is a useful new baseline that appears robust across settings. However, the load-bearing assumption that SSD unlearning faithfully reproduces a model that never saw the target classes rests on a narrow oracle study (three coarse ImageNet subsets on a scratch-trained CLIP) and on an unverified premise that CLIP actually saw the benchmark classes. As it stands, the headline result is plausible but not fully established.

major comments (4)
  1. [Sec. 5.3, Tables 2, 7, 8] The oracle validation covers only three coarse ImageNet subsets (dogs, birds, vehicles) and uses a CLIP model trained from scratch on ImageNet, not the real CLIP and not the seven fine-grained benchmark datasets used in the main evaluation. The absolute accuracies in Tables 2, 7, and 8 are very low for most methods (often below 10%), so the comparison between 'excluded' and 'unlearned' conditions operates near a performance floor and has low power to detect systematic differences. This is insufficient to establish that unlearning the actual benchmark classes from the real CLIP produces a model equivalent to one that never saw those classes, especially for fine-grained datasets such as StanfordCars and FGVCAircraft, where class knowledge is more entangled.
  2. [Sec. 3.3 and Supplementary Sec. V] The paper's central premise that CLIP has seen the benchmark classes is based on the assertion that 'it is almost certain' (Sec. 3.3) and a later assumption that CLIP 'has seen similar images' of those datasets (Supplementary Sec. V). No empirical evidence for membership is provided; the authors cite membership inference attacks [31] in the supplementary discussion but do not run them on the real CLIP model. If this premise is false, the partially-transductive framing collapses, and if it is true, it needs to be demonstrated more rigorously, particularly because the unlearning procedure operates on actual dataset images without verifying their presence in pretraining.
  3. [Sec. 4.1, Eqs. (2)-(6)] The Fisher Information Matrix used by SSD is computed from cross-entropy loss over classification logits (Eq. 2), but CLIP is trained with a contrastive objective that aligns images and text. The paper does not argue why classification-logit-based importance scores identify parameters that specifically encode class identity in CLIP's joint embedding space; dampening such parameters may instead degrade generally useful visual-textual alignment. The observed large drops in few-shot performance could therefore reflect general model damage rather than removal of partially-transductive information. The authors should justify the use of a classification-based FIM for a contrastively trained model, or compare against a contrastive-loss-based unlearning objective.
  4. [Sec. 5.2, Table 1] The headline '55% average drop' and the SEPRES advantage are reported as means across datasets, shots, and seeds, without variance or significance tests. Table 1 shows large between-dataset variation (e.g., CoOp's 1-shot accuracy ranges from near 0% on StanfordCars to roughly 24% on OxfordFlowers in the unlearned setting), and many methods are near zero on several datasets, indicating floor effects. The claim that 'most methods are unable to generalize' would be much stronger with per-dataset breakdowns demonstrating consistent drops and with confidence intervals or paired significance tests across the three seeds.
minor comments (6)
  1. [Sec. 5.1] The CLIP backbone variant (e.g., ViT-B/32) is not stated in the main text; it should be specified in Section 5.1 so that the results are reproducible.
  2. [Sec. 4.2, Eq. (7)] In Eq. (7), 'KnowledgeLost i' should be written 'KnowledgeLost_i' for consistency with the subscript notation used elsewhere.
  3. [Tables 9-13] The column header 'CLIPLora' is used in the supplementary tables while the main text uses 'CLIP-LoRA' and 'LoRA for CLIP'; the notation should be unified.
  4. [Fig. 4(b) caption] The caption of Fig. 4(b) refers to 'retain datasets used while unlearning a dataset and total knowledge lost', but the table columns are labeled 'Forget Ds', 'Retain Ds', and 'Total Knowledge Lost'; the meaning of each column should be clarified.
  5. [Sec. 2] The sentence 'Later methods [14, 18, 19, 19, 39] extended to multimodal prompting' contains a duplicate reference [19]; remove the duplication.
  6. [Sec. 5.4] The paper states '5880 experiments' without explaining the counting formula; add a sentence describing the total as 7 datasets × 5 shot counts × 3 seeds × 14 methods × 4 unlearning levels (or the intended formula).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the inductive-benchmark claim rests on an external unlearning method and an oracle comparison, not on a fitted input or a load-bearing self-citation.

full rationale

The paper's central derivation is a controlled measurement: it unlearns target classes from CLIP using SSD [8] (Eqs. 2-6) and then compares 14 few-shot methods on the unlearned versus original CLIP (Fig. 2, Tab. 1). The reported -55% drop is an empirical comparison, not a fitted parameter renamed as a prediction, and no equation in the paper defines the drop in terms of the methods' own outputs. The oracle validation (Sec. 5.3, Tabs. 2-3, 7-8) compares SSD-unlearned CLIP against CLIP trained from scratch without the subset; this is an external empirical check rather than a self-referential definition. The authors' self-citations [23, 24] appear only as alternative unlearning baselines in the supplementary comparison (Supp. Tab. 6) and are not load-bearing for the main claim; the central unlearning method, SSD, is external prior work. SEPRES (Eqs. 14-16) is an independently evaluated method with ablations (Supp. Tab. 5), and its performance is measured rather than derived from the benchmark construction. The main weakness of the paper—the unverified assumption that the real CLIP actually saw the benchmark classes and that SSD removes class knowledge rather than general adaptability—is a validity threat, not a circularity. The derivation chain is self-contained, and the manuscript's own limitation statements (e.g., 'Even if we do not know the exact data CLIP was trained on, we can assume...') acknowledge the assumption without converting it into a circular proof.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the empirical assumption that CLIP has seen the benchmark classes (Sec. 3.3) and that SSD unlearning approximates training-from-scratch-without-those-classes (Sec. 5.3). Several hyperparameters are fitted or tuned: SSD's α and λ (the latter varied to hit 25/50/90% knowledge-loss targets), and SEPRES's α=0.1, plus unstated ω_t and ω_v. No invented entities are introduced.

free parameters (4)
  • SSD damping threshold α = not reported
    Controls which parameters are dampened in Eq. (5); default chosen to yield minimum knowledge loss (Sec. 4.1).
  • SSD damping strength λ = varied to achieve 25%, 50%, 90% knowledge loss
    Controls the extent of unlearning in Eq. (6); the 'default' is the minimal knowledge loss setting.
  • SEPRES residual scaling α = 0.1
    Selected by ablation in Supplementary Fig. 7; balances pretrained and new knowledge.
  • SEPRES knowledge-guided loss weights ω_t, ω_v = not reported
    Appear in Eq. (15)-(16) but values are not given in the main text.
assumptions (3)
  • domain assumption CLIP's pretraining data includes the evaluation benchmark classes (or images highly correlated with them).
    Sec. 3.3: 'it is almost certain that DC_train ∩ DF_s ≠ ∅'. This makes the standard benchmark 'partially transductive'.
  • domain assumption SSD unlearning on the real CLIP model approximates training a CLIP from scratch without those classes.
    Sec. 5.3 validates only on an ImageNet-trained proxy with three coarse subsets; assumed to transfer to the true CLIP and fine-grained datasets.
  • domain assumption Retain sets composed of six other datasets are sufficient to preserve general CLIP knowledge.
    Sec. 4.1; validated on five held-out datasets with TKL ≤ 2.5% (Fig. 4b).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting." pith.science (2026). https://pith.science/paper/MUKX6WU6

@misc{pith2026250720834,
  author       = {Pith},
  title        = {Pith review of: Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MUKX6WU6}},
  note         = {Machine review of arXiv:2507.20834}
}
read the original abstract

CLIP is a foundational model with transferable classification performance in the few-shot setting. Several methods have shown improved performance of CLIP using few-shot examples. However, so far, all these techniques have been benchmarked using standard few-shot datasets. We argue that this mode of evaluation does not provide a true indication of the inductive generalization ability using few-shot examples. As most datasets have been seen by the CLIP model, the resultant setting can be termed as partially transductive. To solve this, we propose a pipeline that uses an unlearning technique to obtain true inductive baselines. In this new inductive setting, the methods show a significant drop in performance (-55% on average among 13 baselines with multiple datasets). We validate the unlearning technique using oracle baselines. An improved few-shot classification technique is proposed that consistently obtains state-of-the-art performance over 13 other recent baseline methods on a comprehensive analysis with 5880 experiments - varying the datasets, differing number of few-shot examples, unlearning setting, and with different seeds. Thus, we identify the issue with the evaluation of CLIP-based few-shot classification, provide a solution using unlearning, propose new benchmarks, and provide an improved method.

Figures

Figures reproduced from arXiv: 2507.20834 by the authors.

Figure 1
Figure 1. Comparisons between the previous evaluation pipeline [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Unlearned vs Not Unlearned performance for all few [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of different few-shot learning methods aggregated across 7 datasets and different shots for CLIP without unlearning vs. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Shows few-shot accuracy of different methods aggregated across 7 datasets and all shots for various levels of CLIP’s lost [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visual features distributions across different settings with CLIP trained from scratch. Left visualisation: excluding the subset [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of unlearned CLIP and CLIP trained ex [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Ablation on α parameter of SEPRES. 9.1. Weight of Residual Features Ablation We conduct an ablation study for the residual feature im￾portance in SEPRES, shown in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visual features distributions across different settings with CLIP trained from scratch. Left visualisation: Excluding the subset [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A closed-form, multi-layer prototype residual corrects medical VLM logits from few support samples and improves OOD balanced accuracy over zero-shot and most trained adapters.

Reference graph

Works this paper leans on

53 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [31]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy (SP), 2017. 13

    work page 2017

  2. [1]

    Analysis of representations for domain adaptation

    Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems. MIT Press, 2006. 4

    work page 2006

  3. [2]

    Food-101 – mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. Computer Vision – ECCV 2014 , pages 446–461,

    work page 2014

  4. [3]

    Self-supervised learning for few-shot im- age classification

    Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. Self-supervised learning for few-shot im- age classification. In ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1745–1749. IEEE, 2021. 3

    work page 2021

  5. [4]

    A closer look at few-shot classi- fication

    Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classi- fication. arXiv preprint arXiv:1904.04232, 2019. 3

    arXiv 1904

  6. [5]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 5

    work page 2014

  7. [6]

    Salun: Empowering machine un- learning via gradient-based weight saliency in both image classification and generation

    Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine un- learning via gradient-based weight saliency in both image classification and generation. In ICLR 2024, 2023. 3, 4, 13

    work page 2024

  8. [7]

    Learning gen- erative visual models from few training examples: An in- cremental bayesian approach tested on 101 object cate- gories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gen- erative visual models from few training examples: An in- cremental bayesian approach tested on 101 object cate- gories. Computer Vision and Image Understanding , 106: 59–70, 2007. 4, 5

    work page 2007

Show all 53 references
  1. [8]

    Fast machine unlearning without retraining through selective synaptic dampening

    Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12043–12051, 2024. 3, 4, 7

    2024

  2. [9]

    Gen- eralization bounds for few-shot transfer learning with pre- trained classifiers

    Tomer Galanti, Andr’as Gyorgy, and Marcus Hutter. Gen- eralization bounds for few-shot transfer learning with pre- trained classifiers. In First Workshop of Pre-training: Per- spectives, Pitfalls, and Paths Forward at ICML 2022, 2022. 3

    2022

  3. [10]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision , 2023. 1, 2, 6

    2023

  4. [11]

    Amne- siac machine learning, 2020

    Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amne- siac machine learning, 2020. 4

    2020

  5. [12]

    A kernel method for the two- sample-problem

    Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Sch¨olkopf, and Alex Smola. A kernel method for the two- sample-problem. In Advances in Neural Information Pro- cessing Systems. MIT Press, 2006. 4

    2006

  6. [13]

    Visual-language prompt tuning with knowledge-guided context optimization

    Changsheng Xu Hantao Yao, Rui Zhang. Visual-language prompt tuning with knowledge-guided context optimization. In The IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2023. 1, 2, 6

    2023

  7. [14]

    Tcp: Textual-based class-aware prompt tuning for visual-language model

    Changsheng Xu Hantao Yao, Rui Zhang. Tcp: Textual-based class-aware prompt tuning for visual-language model. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 6

    2024

  8. [15]

    Fine-grained visual-textual representation learning

    Xiangteng He and Yuxin Peng. Fine-grained visual-textual representation learning. IEEE Transactions on Circuits and Systems for Video Technology, 30:520–531, 2020. 3, 5

    2020

  9. [16]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5

    2019

  10. [17]

    Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference

    Shell Xu Hu, Da Li, Jan St ¨uhmer, Minyoung Kim, and Tim- othy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pa...

    2022

  11. [18]

    Maple: Multi-modal prompt learning

    Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2, 6

    2023

  12. [19]

    Self-regulating prompts: Foundational model adaptation without forgetting

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzam- mal Naseer, Salman Khan, Ming-Hsuan Yang, and Fa- had Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pa...

    2023

  13. [20]

    Novel dataset for fine-grained image categorization

    Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, 2011. 5

    2011

  14. [21]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops, page 554–561, 2013. 3, 5

    2013

  15. [22]

    Clip adaptation by intra-modal overlap reduction

    Alexey Kravets and Vinay Namboodiri. Clip adaptation by intra-modal overlap reduction. InThe British Machine Vision Conference (BMVC), 2024. 2

    2024

  16. [23]

    Kravets and V

    A. Kravets and V . Namboodiri. Zero-shot class unlearning in clip with synthetic samples. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2025. 2, 4, 13

    2025

  17. [24]

    Namboodiri

    Alexey Kravets and Vinay P. Namboodiri. Zero-shot CLIP class forgetting via text-image space adaptation. Transac- tions on Machine Learning Research, 2025. 3, 4, 13

    2025

  18. [25]

    Fine-grained visual classi- fication of aircraft, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft, 2013. 5

    2013

  19. [26]

    McInnes, J

    L. McInnes, J. Healy, and J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Re- duction. ArXiv e-prints, 2018. 8

    2018

  20. [27]

    Nilsback and Andrew Zisserman

    M. Nilsback and Andrew Zisserman. Automated flower clas- sification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Process- ing, 2008. 5 9

    2008

  21. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  22. [29]

    Transtrum, James P

    Rahul Ramesh, Jialin Mao, Itay Griniasty, Rubing Yang, Han Kheng Teoh, Mark K. Transtrum, James P. Sethna, and Pratik Chaudhari. A picture of the space of typical learnable tasks. In Proceedings of the 40th International Conference on Machine Learning. JMLR.org, 2023. 4

    2023

  23. [30]

    Consistency-guided prompt learning for vision-language models

    Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In ICLR, 2024. 1, 2, 6

    2024

  24. [32]

    Plantdoc: A dataset for visual plant disease detection

    Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. Plantdoc: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, page 249–253, 2020. 5

    2020

  25. [33]

    Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012. 4, 5

    2012

  26. [34]

    Thudi, G

    A. Thudi, G. Deza, V . Chandrasekaran, and N. Papernot. Un- rolling sgd: Understanding factors influencing machine un- learning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319, Los Alamitos, CA, USA, 2022. IEEE Computer Society. 3

    2022

  27. [35]

    Sus-x: Training-free name-only transfer of vision-language models

    Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free name-only transfer of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2725–2736,

  28. [36]

    Matching networks for one shot learning.Ad- vances in neural information processing systems , 29, 2016

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning.Ad- vances in neural information processing systems , 29, 2016. 3

    2016

  29. [37]

    Machine unlearning of features and la- bels

    Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and la- bels. In Proc. of the 30th Network and Distributed System Security (NDSS), 2023. 3

    2023

  30. [38]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition,

    2010

  31. [39]

    Sep: Self-enhanced prompt tuning for visual-language model,

    Hantao Yao, Rui Zhang, Lu Yu, and Changsheng Xu. Sep: Self-enhanced prompt tuning for visual-language model,

  32. [40]

    T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang. Task residual for tuning vision-language models. In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 10899–10909, Los Alamitos, CA, USA, 2023. 1, 2, 6

    2023

  33. [41]

    Low-rank few-shot adaptation of vision-language models

    Maxime Zanella and Ismail Ben Ayed. Low-rank few-shot adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1593–1603, 2024. 2, 6

    2024

  34. [42]

    Tip- adapter: Training-free adaption of clip for few-shot classi- fication

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classi- fication. In Computer Vision – ECCV 2022: 17th European Conference, page 493–510, 2022. 2

    2022

  35. [43]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130:2337 – 2348,

  36. [44]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16795–16804, 2022. 6

    2022

  37. [45]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. Interna- tional Conference on Computer Vision, 2023. 1, 2, 6 10 Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting Supplementary Material Tab...

    2023

  38. [46]

    Ease of Benchmarking and Compute Re- quirements Our main experiments can be performed on a single NVIDIA GeForce RTX 3090 GPU with 24GB of memory. Tab. 4 displays the processing time required to compute the gradients for unlearning CLIP across various datasets. Hence, the proc...

  39. [47]

    Our work builds upon the Self- Enhanced Prompt Tuning (SEP) method, which refines vi- sual prompts using a Token Fusion Module (TFM)

    Mathematical Formulation of SEPRES This section formalizes the mathematical framework under- lying our proposed SEPRES method, introduced in Section 4.3 of the main paper. Our work builds upon the Self- Enhanced Prompt Tuning (SEP) method, which refines vi- sual prompts using ...

  40. [48]

    Ablation on α parameter of SEPRES

    SEPRES Ablations Figure 7. Ablation on α parameter of SEPRES. 9.1. Weight of Residual Features Ablation We conduct an ablation study for the residual feature im- portance in SEPRES, shown in Fig. 7. Without residual features (α = 0.) the accuracy is worse in both settings but ...

  41. [49]

    4.1 of the main paper

    Unlearning Analysis In this section, we provide additional unlearning analy- sis extending Sec. 4.1 of the main paper. Specifically, we present additional unlearning techniques evaluated in our experiments. We explored two zero-shot unlearning methods, one utilizing Lipschitz ...

    1907

  42. [50]

    Discussion Unlearning is a broad concept that cannot be easily guaran- teed, and there are multiple metrics available to evaluate it. SSD unlearning was evaluated using accuracy and Member- ship Inference Attacks (MIA) [31], which assesses whether a particular data point was p...

  43. [51]

    UMAP Visual Features Distribution for all Subsets In this section, we provide UMAP visual features visu- alizations on additional subsets that we discussed in Sec. 5.3. Fig. 8 shows these results for ”birds”, ”vehicles” and ”dogs” subsets from ImageNet. No subset represents CL...

  44. [52]

    5.3 evaluated under two distinct settings - excluding a subset and unlearning a subset

    CLIP Trained from Scratch Full Results In this section we present the full results for our oracle method discussed in Sec. 5.3 evaluated under two distinct settings - excluding a subset and unlearning a subset. Tab. 7 reports the performance of various few-shot learning meth- ...

  45. [53]

    Default runs unlearned CLIP

    Original CLIP Full Results 14.1. Default runs unlearned CLIP. In this section, we present full results across different meth- ods, number of shots and datasets after unlearning CLIP. Aggregated results are discussed in Sec. 5.2 of the main paper. SEPRES consistently outperform...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.