Pith. sign in

REVIEW 3 major objections 9 minor 75 references

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

T0 review · 3 major / 9 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CLIP-PING shows that lightweight vision-language models gain substantially when their usual image-text contrastive loss is supplemented with nearest-neighbor and cross nearest-neighbor contrastive targets derived from frozen features of…

desk verdict A cheap, well-tested NN/XNN supervision trick for small CLIP models, but the headline ImageNet gains are confounded by an ImageNet-pretrained teacher. read the letter →

arxiv 2412.03871 v2 pith:JAIBWS5L submitted 2024-12-05 cs.CV cs.AIcs.ETcs.IRcs.MM

Chu Myaet Thwal , Ye Lin Tun , Minh N. H. Nguyen , Eui-Nam Huh , Choong Seon Hong This is my paper · ORCID
classification cs.CVcs.AIcs.ETcs.IRcs.MM
keywords contrastivelanguage-imagepre-traininglightweightvision-languagemodelsnearest-neighborsupervisioncross-modalretrievalzero-shotclassificationfrozenfeaturebanksefficienttrainingknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLIP-PING claims that a lightweight vision-language model trained with the standard image-text contrastive loss can be substantially improved by adding two extra contrastive losses that pull each sample toward its nearest neighbors in frozen feature banks produced by off-the-shelf unimodal encoders. The paper argues that this "Proximus Intrinsic Neighbors Guidance" gives the student richer semantic supervision than the single paired image-text objective, without the architectural constraints or repeated teacher forward passes of explicit distillation. With a small ViT-XS image encoder and MobileBERT text encoder trained on about 3 million COCO+CC3M pairs, the method reports 22.7% zero-shot ImageNet-1K top-1 accuracy and 49.7%/35.4% Flickr30K I2T/T2I Recall@1, beating standard CLIP by 5.5, 10.7, and 5.7 points respectively at nearly the same memory and per-epoch time. A variant that keeps the teachers active during training, A-CLIP-PING, gives further gains at the same cost as CLIP-KD distillation.

What carries the argument

The mechanism is the Proximus Intrinsic Neighbors (PING) objective built on two frozen feature banks: image features $\tilde{z}^I_k = F^*_I(I_k)$ and text features $\tilde{z}^T_k = F^*_T(T_k)$ are extracted once from off-the-shelf encoders and stored, and two FIFO support sets $Q_I$, $Q_T$ (default size 32768) serve for retrieval. Nearest-neighbor supervision applies InfoNCE-style losses between a student embedding and $\mathrm{NN}(\tilde{z})$, the closest bank feature in the same modality; cross nearest-neighbor supervision instead uses the partner of the closest feature in the other modality, $\mathrm{XNN}(\tilde{z}^I_k) = \tilde{z}^I_i$ where $\tilde{z}^T_i = \mathrm{NN}(\tilde{z}^T_k)$. The full loss is $L = (1-\lambda) L_{\text{CLIP}} + \lambda [(1-\alpha)L_{\text{NN}}^{\text{PING}} + \alpha L_{\text{XNN}}^{\text{PING}}]$ with $\alpha=0.25$, $\lambda=0.6$ chosen by ablation. A linear adapter maps frozen features to the student's projection dimension when the dimensions differ.

What would settle it

Train CLIP-PING with the same NN/XNN losses but replace the image teacher with a self-supervised vision model that never saw ImageNet labels (or a randomly initialized encoder), and compare ImageNet-1K zero-shot accuracy. If the 5.5-point gain persists, the effect is alignment-driven; if it largely disappears on ImageNet but stays on COCO retrieval, the teacher's ImageNet label knowledge was the driver.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that frozen unimodal encoders can act as "intrinsic" teachers without any explicit distillation loss: their precomputed features, stored in auxiliary banks, define nearest-neighbor (NN) and cross nearest-neighbor (XNN) targets that the student learns to align to. Intra-modal NN supervision pulls an image or text embedding toward the feature-bank neighbor the teacher considers most similar within that modality, while inter-modal XNN supervision cross-references the modalities — the image whose paired text is the text neighbor of the query — to encourage indirect alignment of semantically related pairs. Adding these losses with weight λ to the standard CLIP objective consistently improves zero-shot classification, cross-modal retrieval, and linear-transfer accuracy across three lightweight image encoders (a small ViT, a compact ConvNeXt, and a hybrid MobileNet-v4 variant), with reported gains growing on the 3M-pair dataset. The authors interpret the result as evidence that the single image-text pair signal under-supervises small models, and that cheap neighbor-based contrastive supervision from frozen teacher features fills that gap.

Load-bearing premise

The load-bearing premise is that the ImageNet-pretrained teacher's features do not secretly carry ImageNet class knowledge into the student through the neighbor losses — in other words, the gains come from better multimodal alignment, not from the teacher already knowing the test classes.

Editorial extensions

If this is right

  • Across all three lightweight architectures tested, CLIP-PING beats CLIP and CLIP-KD on COCO and COCO+CC3M pretraining for cross-modal retrieval and zero-shot classification.
  • On the 3M-pair setting with ViT-XS, ImageNet-1K zero-shot top-1 rises to 22.7%, and ImageNet distribution-shift sets (IN-V2, IN-R, IN-O, IN-S) all improve over CLIP and CLIP-KD.
  • The method adds only a few MiB of memory and negligible per-epoch time compared with plain CLIP, because teacher features are precomputed once rather than recomputed each batch.
  • A-CLIP-PING, which runs the teachers actively, yields additional gains (1.4, 2.9, and 1.8 points over CLIP-PING for ViT-XS on ImageNet, Flickr30K I2T, and T2I), confirming that stronger teacher guidance is the source of the signal.
  • Ablations show both NN and XNN supervision matter, the optimal support size is 32768, top-1 neighbor selection beats larger k, and a projection dimension of 256 balances cost and accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the NN/XNN losses never require the teacher's logits, the recipe transfers to any frozen unimodal encoder, not just CLIP-style ones; a natural test is whether self-supervised visual teachers (no class labels) give the same ImageNet boost, which would separate alignment gains from label leakage.
  • The same two-stage frozen-bank mechanism could be applied to other paired modalities, such as audio-text or video-audio, where a lightweight student needs cross-modal alignment without running large teachers online.
  • The teacher here is ResNet-v2-50 pretrained on ImageNet-21k/1k, so part of the reported ImageNet-family improvement may come from the teacher's class knowledge flowing through neighbor targets; evaluating on a non-ImageNet benchmark not seen by the teacher would quantify this.
  • The support-set sweet spot (gains up to 32768, then decline) suggests neighbor diversity helps only up to the point where retrieved neighbors stay semantically relevant, implying the method's benefit depends on dataset redundancy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. This paper proposes CLIP-PING, a training procedure for lightweight vision-language models that augments the standard CLIP contrastive loss with two auxiliary contrastive terms: intra-modal nearest-neighbor (NN) supervision and inter-modal cross nearest-neighbor (XNN) supervision, using features from frozen off-the-shelf unimodal encoders (a BiT ResNet-v2-50 image teacher and BERT-Base text teacher). The final objective is L = (1-λ)L_CLIP + λ[(1-α)L_NN + α L_XNN] (Eqs. 14-15). The authors evaluate on COCO (600K pairs) and COCO+CC3M (3M pairs) with three lightweight image encoders (ViT-XS, ConvNeXt-Pico, MNv4-Hybrid-M) paired with MobileBERT-TINY. They report consistent gains over CLIP and CLIP-KD on cross-modal retrieval (Flickr30K, COCO, CC3M), zero-shot classification (ImageNet1K and four ImageNet robustness sets plus four other datasets), and linear evaluation on 12 datasets. An 'active' variant (A-CLIP-PING) that recomputes teacher features during training yields further gains.

Significance. The proposed mechanism is simple, plausible, and, if the reported gains reflect the mechanism's generic alignment properties, would be a useful addition to the toolkit for training lightweight vision-language models in low-resource settings. The paper is methodologically strong in its breadth: three student architectures, two training sets, multiple ablations (loss weights, support set size, top-k, projection dimension, supervision source, teacher swap), and explicit wall-clock/memory measurements. The method itself is not internally circular: the PING losses add new contrastive terms rather than reducing to a single fitted constant. The central weakness is that the headline ImageNet-family results are obtained with an ImageNet-pretrained teacher, and the most important hyperparameters were selected on the same benchmarks that are later reported as results. These two issues prevent the paper from currently supporting its strongest claims.

major comments (3)
  1. [Sec. IV-A-d, Tables V-VI] The default image teacher is timm/resnetv2_50x1_bit.goog_in21k_ft_in1k, a BiT ResNet-v2-50 pretrained on ImageNet-21k and fine-tuned on ImageNet-1k, and the only teacher swap in Table XIa and Tables XXII-XXIII is to ViT-B/16, which is also ImageNet-pretrained. Because the PING losses in Eqs. (4)-(13) directly shape the student's visual feature geometry toward teacher-derived nearest-neighbor structure, the large gains reported on ImageNet1K and the ImageNet robustness sets in Tables V and VI (5.5% on IN-1K, 5.1% on IN-V2, 5.6% on IN-R, 5.6% on IN-O, 3.9% on IN-S) may substantially reflect transfer of the teacher's ImageNet label knowledge rather than generic cross-modal alignment. Please add a control experiment with an image teacher that has not been trained on ImageNet labels (e.g., a self-supervised DINO teacher or the same BiT model without the ImageNet-1k fine-tune) and report the ImageNet-family zero-shot results with that teacher; if the gains shrink materially, the claims in the abstract should be revised to focus on the non-ImageNet benchmarks.
  2. [Sec. IV-C and VIII-C (Tables IX, X, XX)] The hyperparameters α, λ, support-set size |Q|, top-k, and projection dimension d are all selected by optimizing Flickr30K retrieval and/or the average zero-shot accuracy across the same four datasets (STL, C10, C100, SA-40) that are later reported as the main results (Tables IXa, IXb, Xa, Xb, and XX; see also the default settings in Sec. IV-A-e). For example, α=0.25 and λ=0.6 are chosen because they achieve the best values in Table IX, and |Q|=32768 is chosen because it is best in Table Xa. This is a test-set selection problem: the headline numbers in Tables III-VII are not an unbiased estimate of the method's generalization ability. Please tune on a held-out validation set (e.g., a split of COCO captions or a separate validation split of CC3M) and report the resulting settings, or present the main results for a range of hyperparameter values to show that the conclusions are not sensitive to the specific choices.
  3. [Table XIII] The comparison with DeCLIP is not on equal footing: DeCLIP is trained at batch size 512 while CLIP-PING uses 1024, and the DeCLIP learning rates are modified because of reported instability. Since DeCLIP's contrastive objective depends on in-batch negatives, this asymmetric setup systematically disadvantages DeCLIP. Given that DeCLIP is the most closely related prior work on neighbor-supervision for CLIP training, please retrain DeCLIP at batch size 1024 with its original hyperparameters, or state explicitly that the reported DeCLIP results are obtained under a different training budget and are not directly comparable.
minor comments (9)
  1. [Sec. IV-A-d] The teacher is described as 'ResNet-v2-50', but the timm checkpoint name indicates a BiT model pretrained on ImageNet-21k with fine-tuning on ImageNet-1k; please make this explicit in the text.
  2. [Table II] The column headers in Table II are incomplete; 'Feat.dim' and 'Size(GB)' are not clearly labeled as columns, and the ViT-B/16 dimension of 21952 requires explanation.
  3. [Algorithm 1] Line 9 uses '˜zI_k ← B∗_I (Ik)', which treats the feature bank as a function; please use clearer notation such as 'retrieve the stored feature for Ik from B∗_I'.
  4. [Sec. IV-C-f] The text refers to 'selecting one random neighbor from the top-k NN', but the method section defines NN(·) only as the single nearest neighbor; please clarify the random selection procedure.
  5. [Table XIa] The ViT-B/16 ablation table omits the plain CLIP baseline; please include it for completeness.
  6. [Sec. IV-B-b] The statement that CLIP-PING is 'as efficient as the original CLIP [21]' cites CLIP-CID rather than the original CLIP paper; the citation should be [2].
  7. [Fig. 5 caption] The caption misspells the text encoder as 'MoblileBERT TINY'; the correct name is MobileBERT-TINY.
  8. [Sec. III-B-a] Reference [1] is a medical-imaging contrastive-learning paper, but it is cited together with [44] for InfoNCE; the InfoNCE citation should point to [44] only.
  9. [Abstract / Sec. I] The claim of gains 'without extra computational costs' should be qualified: the method requires one-time feature extraction and disk storage for the feature banks (Table II, up to 22.1 GB for the 3M dataset), and A-CLIP-PING requires active teacher computation.

Circularity Check

1 steps flagged · score 4.0 of 10

The PING loss itself is not circular, but the COCO-scale headline numbers are partly self-selected because the default hyperparameters (alpha, lambda, |Q|, top-k, projection dimension) are chosen by maximizing the same Flickr30K and zero-shot metrics that are later reported as the main results. The ImageNet-family teacher-pretraining confound is an external-validity risk, not a circular reduction.

  1. fitted input called prediction [Sec. IV-A.e (default hyperparameters), Sec. IV-C.d and Tables IX-X, XX-XXI (ablations), Tables III and V (headline COCO-scale results)]
    "By default, we set the supervision loss weight values as α = 0.25 and λ = 0.6. Our support set is implemented as a first-in-first-out (FIFO) queue, with a queue size of |Q| = 32768."

    The paper selects α, λ, |Q|, top-k, and projection dimension by maximizing Flickr30K R@1 and the average zero-shot accuracy over four datasets using the COCO 600K model (Tables IX, X, XX, XXI). The main COCO-scale tables then report exactly the selected configuration: CLIP-PING with ViT-XS on COCO gives 28.1 I2T@1, 20.2 T2I@1 on Flickr30K and 41.2 ZS AVG (Tables III and V), identical to the best ablation rows. Those numbers are therefore the argmax of the hyperparameter search objective, not independent out-of-sample evidence for those particular rows. The COCO+CC3M (3M) headline results are not directly used in this search and remain substantially independent.

full rationale

The derivation chain of CLIP-PING is self-contained: the PING objective (Eqs. 4-15) is an additional contrastive loss on frozen teacher-derived NN/XNN targets, and the reported gains are empirical results of training the student, not consequences of the loss definition. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via citation. The teacher's ImageNet-21k/1k pretraining is a legitimate external input; the possibility that it inflates ImageNet-family zero-shot numbers is a confound for the generalization claim, but no equation equates the student's ImageNet accuracy to the teacher's, so it is not circularity under the stated rules. The one partially circular element is hyperparameter selection: α, λ, |Q|, top-k, and projection dimension are tuned on the exact Flickr30K and zero-shot metrics that are then reported as main COCO-scale results, making those particular rows self-selected. The COCO+CC3M (3M) results, including the headline 22.7% ImageNet1K and 49.7/35.4 Flickr30K numbers, were not used for this search and provide independent support for the method's central claim. The score of 4 reflects this partial selection bias while acknowledging that the core claim retains substantial independent content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. It depends on several domain assumptions about teacher feature quality and neighbor semantics, and it relies on five hyperparameters that are tuned on the same evaluation benchmarks later used for the headline results. The choice of ImageNet-pretrained ResNet as the image teacher is an additional modeling assumption that may affect ImageNet-family evaluations.

free parameters (5)
  • alpha (loss weight between NN and XNN supervision) = 0.25
    Tuned via Table IXa using Flickr30K retrieval and average zero-shot accuracy, which are also the final evaluation benchmarks.
  • lambda (loss weight between CLIP loss and PING loss) = 0.6
    Tuned via Table IXb using the same Flickr30K and zero-shot benchmarks reported as final results.
  • Support set size |Q| = 32768
    Selected via Table Xa; performance degrades for larger and smaller values on the evaluation benchmarks.
  • Top-k neighbors = 1
    Selected via Table Xb; larger k degrades Flickr30K and zero-shot results on the evaluation benchmarks.
  • Projection dimension d = 256
    Selected via Table XX across three model pairs using Flickr30K and zero-shot accuracy on the evaluation benchmarks.
assumptions (4)
  • domain assumption Nearest neighbors in teacher feature space are semantically similar enough to serve as useful contrastive targets.
    The entire PING loss relies on NN and XNN retrieval in frozen ResNet/BERT feature space; if neighbors are not semantically related, the extra supervision would add noise or hurt performance. This is assumed rather than measured.
  • domain assumption The off-the-shelf unimodal teacher encoders provide features that are useful for the downstream zero-shot and retrieval tasks.
    ResNet-v2-50 pretrained on ImageNet and BERT-Base pretrained on text are used as guides; the paper does not study what happens with weaker or self-supervised teachers, so the benefit is tied to these specific pretrained encoders.
  • domain assumption The FIFO support set is a representative subset of the dataset and does not introduce self-retrieval artifacts.
    The support set size is ablated, but the possibility that the nearest neighbor is the sample's own frozen feature or an identical feature from another caption of the same COCO image is not examined.
  • standard math Standard InfoNCE / CLIP contrastive objective is a valid base loss for learning cross-modal alignment.
    The method builds on CLIP's symmetric InfoNCE loss, which is taken as given from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance." pith.science (2026). https://pith.science/paper/JAIBWS5L

@misc{pith2026241203871,
  author       = {Pith},
  title        = {Pith review of: CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAIBWS5L}},
  note         = {Machine review of arXiv:2412.03871}
}
read the original abstract

Beyond the success of Contrastive Language-Image Pre-training (CLIP), recent trends mark a shift toward exploring the applicability of lightweight vision-language models for resource-constrained scenarios. These models often deliver suboptimal performance when relying solely on a single image-text contrastive learning objective, spotlighting the need for more effective training mechanisms that guarantee robust cross-modal feature alignment. In this work, we propose CLIP-PING: Contrastive Language-Image Pre-training with Proximus Intrinsic Neighbors Guidance, a novel yet simple and efficient training paradigm designed to boost the performance of lightweight vision-language models with minimal computational overhead and lower data demands. CLIP-PING bootstraps unimodal features extracted from arbitrary pre-trained encoders to obtain intrinsic guidance of proximus neighbor samples, i.e., nearest-neighbor (NN) and cross nearest-neighbor (XNN). We find that extra contrastive supervision from these neighbors substantially boosts cross-modal alignment, enabling lightweight models to learn more generic features with rich semantic diversity. Extensive experiments reveal that CLIP-PING notably surpasses its peers in zero-shot generalization and cross-modal retrieval tasks. Specifically, a 5.5% gain on zero-shot ImageNet1K classification with 10.7% (I2T) and 5.7% (T2I) on Flickr30K retrieval, compared to the original CLIP when using ViT-XS image encoder trained on 3 million (image, text) pairs. Moreover, CLIP-PING showcases a strong transferability under the linear evaluation protocol across several downstream tasks.

Figures

Figures reproduced from arXiv: 2412.03871 by the authors.

Figure 1
Figure 1. Comparison on zero-shot classification and retrieval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of nearest-neighbor (NN) and cross nearest [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the CLIP-PING pipeline. Unimodal feature extraction is performed prior to the multi-modal training, with [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example for nearest-neighbor (NN) and cross nearest [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Curves for ViT-XS [12] + MoblileBERTTINY [47] [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 31 canonical work pages

  1. [1]

    Contrastive learning of medical visual representations from paired images and text,

    Y . Zhang, H. Jiang, Y . Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” in Machine Learning for Healthcare Conference . PMLR, 2022, pp. 2–25

    2022

  2. [2]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  3. [3]

    Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,

    H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei, “Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,” Advances in Neural Information Pro- cessing Systems, vol. 35, pp. 32 897–32 912, 2022

    work page 2022

  4. [4]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International conference on machine learning . PMLR, 2021, pp. 4904–4916

    2021

  5. [5]

    Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data,

    D. Qi, L. Su, J. Song, E. Cui, T. Bharti, and A. Sacheti, “Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data,” arXiv preprint arXiv:2001.07966 , 2020

    arXiv 2001

  6. [6]

    Reproducible scaling laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2818–2829. TABLE XIX: Comparison on linear evaluation performance. MNv4-Hybrid...

    work page 2023

  7. [7]

    Scaling language- image pre-training via masking,

    Y . Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He, “Scaling language- image pre-training via masking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 23 390–23 400

    2023

  8. [8]

    An inverse scaling law for clip training,

    X. Li, Z. Wang, and C. Xie, “An inverse scaling law for clip training,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

Show all 75 references
  1. [9]

    Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,

    Y . Li, F. Liang, L. Zhao, Y . Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, “Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openrevie...

    2022

  2. [10]

    Too large; data reduction for vision-language pre-training,

    A. J. Wang, K. Q. Lin, D. J. Zhang, S. W. Lei, and M. Z. Shou, “Too large; data reduction for vision-language pre-training,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3147–3157

    2023

  3. [11]

    Slip: Self-supervision meets language-image pre-training,

    N. Mu, A. Kirillov, D. Wagner, and S. Xie, “Slip: Self-supervision meets language-image pre-training,” in European conference on computer vision. Springer, 2022, pp. 529–544

    2022

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning...

    2021

  5. [13]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer,...

    2014

  6. [14]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. ...

    2018

  7. [15]

    Image as a foreign language: Beit pretraining for vision and vision-language tasks,

    W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som et al. , “Image as a foreign language: Beit pretraining for vision and vision-language tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  8. [16]

    The crucial role of data collection in research: Techniques, challenges, and best practices,

    I. Karunarathna, P. Gunasena, T. Hapuarachchi, and S. Gunathilake, “The crucial role of data collection in research: Techniques, challenges, and best practices,” Uva Clinical Research , pp. 1–24, 2024. 13

    2024

  9. [17]

    Tinyclip: Clip distillation via affinity mimicking and weight inheritance,

    K. Wu, H. Peng, Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. S. Chen, X. Wang et al., “Tinyclip: Clip distillation via affinity mimicking and weight inheritance,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 21 970–21 980

    2023

  10. [18]

    Clip-kd: An empirical study of clip model distillation,

    C. Yang, Z. An, L. Huang, J. Bi, X. Yu, H. Yang, B. Diao, and Y . Xu, “Clip-kd: An empirical study of clip model distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 952–15 962

    2024

  11. [19]

    Mobileclip: Fast image-text models through multi-modal reinforced training,

    P. K. A. Vasu, H. Pouransari, F. Faghri, R. Vemulapalli, and O. Tuzel, “Mobileclip: Fast image-text models through multi-modal reinforced training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15 963–15 974

    2024

  12. [20]

    Comkd-clip: Comprehen- sive knowledge distillation for contrastive language-image pre-traning model,

    Y . Chen, X. Qiao, Z. Sun, and X. Li, “Comkd-clip: Comprehen- sive knowledge distillation for contrastive language-image pre-traning model,” arXiv preprint arXiv:2408.04145 , 2024

    2024 arXiv

  13. [21]

    Clip-cid: Efficient clip distillation via cluster-instance discrimination,

    K. Yang, T. Gu, X. An, H. Jiang, X. Dai, Z. Feng, W. Cai, and J. Deng, “Clip-cid: Efficient clip distillation via cluster-instance discrimination,” arXiv preprint arXiv:2408.09441 , 2024

    2024 arXiv

  14. [22]

    Module-wise adaptive distillation for multimodality foun- dation models,

    C. Liang, J. Yu, M.-H. Yang, M. Brown, Y . Cui, T. Zhao, B. Gong, and T. Zhou, “Module-wise adaptive distillation for multimodality foun- dation models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  15. [23]

    Self-supervised co-training for video representation learning,

    T. Han, W. Xie, and A. Zisserman, “Self-supervised co-training for video representation learning,” Advances in neural information processing systems, vol. 33, pp. 5679–5690, 2020

    2020

  16. [24]

    Improving generalization via scalable neighborhood component analysis,

    Z. Wu, A. A. Efros, and S. X. Yu, “Improving generalization via scalable neighborhood component analysis,” in Proceedings of the european conference on computer vision (ECCV) , 2018, pp. 685–701

    2018

  17. [25]

    With a little help from my friends: Nearest-neighbor contrastive learning of visual representations,

    D. Dwibedi, Y . Aytar, J. Tompson, P. Sermanet, and A. Zisserman, “With a little help from my friends: Nearest-neighbor contrastive learning of visual representations,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9588–9597

    2021

  18. [26]

    Promoting semantic connectivity: Dual nearest neighbors contrastive learning for unsupervised domain generalization,

    Y . Liu, Y . Wang, Y . Chen, W. Dai, C. Li, J. Zou, and H. Xiong, “Promoting semantic connectivity: Dual nearest neighbors contrastive learning for unsupervised domain generalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023,...

    2023

  19. [27]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  20. [28]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,

    P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014

    2014

  21. [29]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  22. [30]

    A survey of vision-language pre-trained models,

    Y . Du, Z. Liu, J. Li, and W. X. Zhao, “A survey of vision-language pre-trained models,” arXiv preprint arXiv:2202.10936 , 2022

    2022 arXiv

  23. [31]

    Vlp: A survey on vision-language pre-training,

    F.-L. Chen, D.-Z. Zhang, M.-L. Han, X.-Y . Chen, J. Shi, S. Xu, and B. Xu, “Vlp: A survey on vision-language pre-training,” Machine Intelligence Research, vol. 20, no. 1, pp. 38–56, 2023

    2023

  24. [32]

    Self- supervised learning of visual features through embedding images into text topic spaces,

    L. Gomez, Y . Patel, M. Rusinol, D. Karatzas, and C. Jawahar, “Self- supervised learning of visual features through embedding images into text topic spaces,” in Proceedings of the ieee conference on computer vision and pattern recognition , 2017, pp. 4230–4239

    2017

  25. [33]

    Beyond instance-level image retrieval: Lever- aging captions to learn a global visual representation for semantic retrieval,

    A. Gordo and D. Larlus, “Beyond instance-level image retrieval: Lever- aging captions to learn a global visual representation for semantic retrieval,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 6589–6598

    2017

  26. [34]

    Learning visual n-grams from web data,

    A. Li, A. Jabri, A. Joulin, and L. Van Der Maaten, “Learning visual n-grams from web data,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 4183–4192

    2017

  27. [35]

    Virtex: Learning visual representations from textual annotations,

    K. Desai and J. Johnson, “Virtex: Learning visual representations from textual annotations,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 162–11 173

    2021

  28. [36]

    Learning visual representa- tions with caption annotations,

    M. B. Sariyildiz, J. Perez, and D. Larlus, “Learning visual representa- tions with caption annotations,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16. Springer, 2020, pp. 153–170

    2020

  29. [37]

    Combined scaling for zero-shot transfer learning,

    H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, J. Yu, Y .-T. Chen, M.-T. Luong, Y . Wuet al., “Combined scaling for zero-shot transfer learning,” Neurocomputing, vol. 555, p. 126658, 2023

    2023

  30. [38]

    SimVLM: Simple visual language model pretraining with weak supervision,

    Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y . Tsvetkov, and Y . Cao, “SimVLM: Simple visual language model pretraining with weak supervision,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=GUrhfTuf 3

    2022

  31. [39]

    Florence: A new foundation model for computer vision,

    L. Yuan, D. Chen, Y .-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li et al. , “Florence: A new foundation model for computer vision,” arXiv preprint arXiv:2111.11432 , 2021

    2021 arXiv

  32. [40]

    Lit: Zero-shot transfer with locked-image text tuning,

    X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, “Lit: Zero-shot transfer with locked-image text tuning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 18 123–18 133

    2022

  33. [41]

    Compressing visual-linguistic model via knowledge distillation,

    Z. Fang, J. Wang, X. Hu, L. Wang, Y . Yang, and Z. Liu, “Compressing visual-linguistic model via knowledge distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1428–1438

    2021

  34. [42]

    Distilling large vision-language model with out-of-distribution generalizability,

    X. Li, Y . Fang, M. Liu, Z. Ling, Z. Tu, and H. Su, “Distilling large vision-language model with out-of-distribution generalizability,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2492–2503

    2023

  35. [43]

    Multimodal adaptive distilla- tion for leveraging unimodal encoders for vision-language tasks,

    Z. Wang, N. Codella, Y .-C. Chen, L. Zhou, X. Dai, B. Xiao, J. Yang, H. You, K.-W. Chang, S.-f. Chang et al., “Multimodal adaptive distilla- tion for leveraging unimodal encoders for vision-language tasks,” arXiv preprint arXiv:2204.10496, 2022

    2022 arXiv

  36. [44]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  37. [45]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019

    2019

  38. [46]

    Pytorch image models,

    R. Wightman, “Pytorch image models,” https://github.com/rwightman/ pytorch-image-models, 2019

    2019

  39. [47]

    MobileBERT: a compact task-agnostic BERT for resource-limited devices,

    Z. Sun, H. Yu, X. Song, R. Liu, Y . Yang, and D. Zhou, “MobileBERT: a compact task-agnostic BERT for resource-limited devices,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds...

    2020

  40. [48]

    A convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 11 976–11 986

    2022

  41. [49]

    Mobilenetv4-universal models for the mobile ecosystem,

    D. Qin, C. Leichner, M. Delakis, M. Fornoni, S. Luo, F. Yang, W. Wang, C. Banbury, C. Ye, B. Akin et al. , “Mobilenetv4-universal models for the mobile ecosystem,” arXiv preprint arXiv:2404.10518 , 2024

    2024 arXiv

  42. [50]

    Big transfer (bit): General visual representation learning,

    A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, “Big transfer (bit): General visual representation learning,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16 . Springer, 2020, p...

    2020

  43. [51]

    Identity mappings in deep residual networks,

    K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer, 2016, pp. 630–645

    2016

  44. [52]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2

    2019

  45. [53]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  46. [54]

    Algorithm 799: revolve: an implementa- tion of checkpointing for the reverse or adjoint mode of computational differentiation,

    A. Griewank and A. Walther, “Algorithm 799: revolve: an implementa- tion of checkpointing for the reverse or adjoint mode of computational differentiation,” ACM Transactions on Mathematical Software (TOMS) , vol. 26, no. 1, pp. 19–45, 2000

    2000

  47. [55]

    Training deep nets with sublinear memory cost,

    T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” arXiv preprint arXiv:1604.06174 , 2016

    2016 arXiv

  48. [56]

    Mixed precision training,

    P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in International Conference on Learning Representations , 2018. [Online]. Available: https: //openreview.net/forum?i...

    2018

  49. [57]

    An analysis of single-layer networks in unsupervised feature learning,

    A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2011, pp. 215–223

    2011

  50. [58]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” Toronto, ON, Canada, Tech. Rep., 2009

    2009

  51. [59]

    Human action recognition by learning bases of action attributes and 14 parts,

    B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. Guibas, and L. Fei-Fei, “Human action recognition by learning bases of action attributes and 14 parts,” in 2011 International conference on computer vision . IEEE, 2011, pp. 1331–1338

    2011

  52. [60]

    Do imagenet clas- sifiers generalize to imagenet?

    B. Recht, R. Roelofs, L. Schmidt, and V . Shankar, “Do imagenet clas- sifiers generalize to imagenet?” in International conference on machine learning. PMLR, 2019, pp. 5389–5400

    2019

  53. [61]

    The many faces of robustness: A critical analysis of out-of-distribution generalization,

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo et al. , “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in Proceedings of the IEEE/CVF international conference on computer visio...

    2021

  54. [62]

    Natural adversarial examples,

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 15 262–15 271

    2021

  55. [63]

    Learning robust global representations by penalizing local predictive power,

    H. Wang, S. Ge, Z. Lipton, and E. P. Xing, “Learning robust global representations by penalizing local predictive power,” Advances in neural information processing systems , vol. 32, 2019

    2019

  56. [64]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in 2012 IEEE conference on computer vision and pattern recog- nition. IEEE, 2012, pp. 3498–3505

    2012

  57. [65]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,

    L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in 2004 conference on computer vision and pattern recognition workshop . IEEE, 2004, pp. 178–178

    2004

  58. [66]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian conference on computer vision, graphics & image processing . IEEE, 2008, pp. 722–729

    2008

  59. [67]

    Fine- grained visual classification of aircraft,

    S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine- grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  60. [68]

    Food-101–mining discriminative components with random forests,

    L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” in Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13 . Springer, 2014, pp. 446–461

    2014

  61. [69]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 3606–3613

    2014

  62. [70]

    Sun database: Large-scale scene recognition from abbey to zoo,

    J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010, pp. 3485–3492

    2010

  63. [71]

    Collecting a large-scale dataset of fine-grained cars,

    J. Krause, J. Deng, M. Stark, and L. Fei-Fei, “Collecting a large-scale dataset of fine-grained cars,” 2013

    2013

  64. [72]

    3d object representations for fine-grained categorization,

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Proceedings of the IEEE interna- tional conference on computer vision workshops , 2013, pp. 554–561

    2013

  65. [73]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  66. [74]

    Knowledge distillation: A good teacher is patient and consistent,

    L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov, “Knowledge distillation: A good teacher is patient and consistent,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 925–10 934

    2022

  67. [75]

    The efficiency misnomer,

    M. Dehghani, Y . Tay, A. Arnab, L. Beyer, and A. Vaswani, “The efficiency misnomer,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/ forum?id=iulEMLYh1uR

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.