Pith. sign in

REVIEW 4 major objections 4 minor 68 references

Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A teacher-student distillation with YOLO region features and a frozen text encoder can specialize CLIP for retrieval on about 67,500 image-text pairs while retaining about 94% of its zero-shot classification accuracy.

desk verdict Incremental recipe, plausible direction, but the retrieval numbers are unproven until train/test overlap is ruled out. read the letter →

arxiv 2505.21549 v4 pith:NMGCZGJ3 submitted 2025-05-25 cs.CV cs.CL

Daniel Csizmadia , Andrei Codreanu , Victor Sim , Vighnesh Prabhu , Michael Lu , Kevin Zhu , Sean O'Brien , Vasu Sharma This is my paper · ORCID
classification cs.CVcs.CL
keywords DCLIPknowledgedistillationimage-textretrievalCLIPfine-tuningregion-levelfeaturescross-modalattentionzero-shotclassificationcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DCLIP claims that a CLIP model can be specialized for image-text retrieval with a small curated dataset—about 67,500 image-caption pairs from MSCOCO, Flickr30k, and Conceptual Captions—without losing its general-purpose character. The method fine-tunes CLIP's image encoder under supervision from a 'meta-teacher' that uses YOLO-detected image regions and bidirectional cross-modal attention to produce enriched embeddings, while the text encoder stays frozen. The resulting student model improves retrieval metrics, especially text-to-image Recall@1, and retains about 94% of the original CLIP's zero-shot classification accuracy on ImageNet for ViT-B backbones. This matters because it suggests task specialization and generalization are not a hard trade-off when the distillation signal is region-aware and the frozen text encoder anchors the model to CLIP's semantic space.

What carries the argument

The load-bearing mechanism is a meta teacher–student distillation loop. A 'meta-teacher' runs YOLOv8x over each image, weights the detected region crops by a penalty combining detection confidence, bounding-box area, and cosine similarity to the paired caption, encodes those regions with CLIP, and fuses the region embeddings with text embeddings through two fine-tuned multi-head attention layers—text attending to regions and regions attending to text. The resulting global image embedding is formed by temperature-scaled attention pooling over attended patches. A student CLIP image encoder is then fine-tuned to mimic these teacher embeddings through image-side and text-side cosine distillation, while an InfoNCE contrastive loss keeps it from collapsing to the teacher and preserves CLIP's discriminative structure. At inference the student sees ordinary images and text only; it never needs bounding boxes or the cross-modal module.

What would settle it

Compare the image IDs in the 67,500-sample training mix (42,000 MSCOCO 2017 images, 10,000 Flickr30K images, 15,000 Conceptual Captions entries) against the Karpathy test images (5,000 MSCOCO images and 1,000 Flickr30K images); if any IDs appear in both, retrain or re-evaluate on a strictly disjoint split and see whether the reported Recall@1 gains survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that a student CLIP model, taught by a cross-modal teacher that fuses YOLO-extracted image regions with text via bidirectional attention, can outperform the base CLIP model on MSCOCO and Flickr30K retrieval benchmarks while keeping most of CLIP's zero-shot classification ability. The student is trained with a hybrid objective combining InfoNCE contrastive learning with cosine distillation losses, and only the image encoder is adapted; the text encoder is frozen, which anchors the learned embeddings to CLIP's original language space. Reported results include consistent text-to-image Recall@1 gains across ViT-B/32, ViT-B/16, and ViT-L/14 backbones, with ViT-B/16 retaining about 94% of ImageNet zero-shot Top-1 accuracy. The authors also report that the optimal teacher is not the fully converged one: earlier checkpoints (epochs 2–3) give a better balance between retrieval gains and zero-shot preservation.

Load-bearing premise

The central numbers depend on the sampled training images from MSCOCO 2017 and Flickr30K being disjoint from the Karpathy test images used for retrieval evaluation, and the paper does not describe how that overlap was prevented.

Editorial extensions

If this is right

  • Text-to-image retrieval improves by 15–35 percentage points in Recall@1 across CLIP backbones, with consistent gains in image-to-text retrieval and MAP, using only about 67,500 training pairs for ViT-B.
  • The asymmetric student–teacher design keeps region processing on the teacher side, so the deployed student retains CLIP's inference-time architecture and efficiency.
  • Because the text encoder is frozen, the student's learned image embeddings are anchored to CLIP's semantic space, which is why zero-shot classification degrades only mildly.
  • Longer teacher training does not monotonically help: the paper finds a 'sweet spot' around epochs 2–3 where retrieval gains and zero-shot retention are best balanced.
  • The framework transfers to larger backbones like ViT-L/14 when extra preservation losses, multi-cluster embedding aggregation, and rotary position embeddings are added, though zero-shot retention there is lower (about 91%).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the absence of train/test leakage is confirmed, the main practical consequence is that cheap, small-scale retrieval specialization of frozen-text foundation models is feasible on consumer hardware, opening a low-resource path to domain-specific retrieval.
  • The 'teacher need not converge' result is probably a general distillation phenomenon: for other student–teacher setups, intermediate teacher checkpoints may provide a better balance than final checkpoints; this is directly testable in non-vision settings.
  • Because only the image side is adapted, the method is lopsided by construction; extending the same asymmetry to other paired modalities, such as audio–text or video–text, could be tried as a direct generalization.
  • The reliance on YOLO proposals creates an obvious failure mode: datasets whose salient objects YOLO was not trained to detect would receive weak region supervision, so the method may be less portable than its 67,500-sample cost suggests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DCLIP, a teacher-student distillation method that specializes a frozen CLIP model for image-text retrieval. A YOLO-based teacher with bidirectional cross-modal attention produces region-informed image embeddings that supervise a student CLIP image encoder; the text encoder is kept frozen and the student is trained with a hybrid contrastive-plus-cosine loss. The authors report consistent Recall@K and MAP gains over CLIP baselines on the MSCOCO and Flickr30K Karpathy splits across ViT-B/32, ViT-B/16, and ViT-L/14, while retaining roughly 94% of CLIP's ImageNet zero-shot top-1 accuracy, using only about 67.5k training pairs. The paper also provides ablations, training-efficiency notes, and an anonymous code link.

Significance. If the reported results hold, DCLIP would be a useful lightweight recipe for specializing CLIP to retrieval with small data, low compute, and no region processing at inference. The paper deserves credit for the consistent direction of the results across three backbones, for the explicit ablation of the teacher components, and for releasing code. The framework is simple enough to be reproduced and extended, and the Pareto-style framing of the retrieval-versus-zero-shot trade-off is a sensible practical contribution. However, the central empirical claim currently rests on an unverified data split, and one of the three stated loss terms is identically zero under the paper's own frozen-text-encoder design. These issues must be resolved before the quantitative claims can be accepted.

major comments (4)
  1. [§4 and §5.1] The paper never states that the Karpathy test images were excluded from the training mixture. Section 4 describes training on about 50,000 MSCOCO 2017 images and 10,000 Flickr30K images, while Section 5.1 says evaluation uses the Karpathy test splits of 5,000 MSCOCO images and 1,000 Flickr30K images. Because the Karpathy MSCOCO test images are a subset of the COCO images and Flickr30K has only about 31,000 images total, sampling 10,000 Flickr30K images without explicitly removing the 1,000 test images makes overlap likely. If any test image or caption appears in training, the text-to-image Recall@1 gains in Table 2 (e.g., MSCOCO 0.31 to 0.40 and Flickr30K 0.59 to 0.69 for ViT-B/32) are inflated by memorization. Please state the exact image IDs used for training and the exclusion rule relative to the Karpathy splits, or release the exact training split.
  2. [§3.2 and Appendix C] The text cosine distillation term L_T_cos is identically zero by the paper's own definitions. Equation (7) defines L_T_cos = 1 - sim(z_T_s, z_T_t), and Section 3.2 states that the student text embeddings come from CLIP's default frozen text encoder while Appendix C explicitly says z_T_t is 'identical to z_T_s'. Since both are the same frozen CLIP text encoder outputs, L_T_cos = 0 for every training example, so the stated student loss L_student = L_contrast + L_T_cos + L_I_cos reduces to only two effective terms. Please remove this term from the loss formulation, or, if a text-distillation effect is intended, specify a trainable or distinct text encoder that makes the term meaningful.
  3. [§4, Appendix A, Appendix B] The reported training-set sizes are mutually inconsistent, and the inconsistencies affect every quantitative claim in the paper. Section 4 says the ViT-B mixture is about 50,000 MSCOCO + 10,000 Flickr30K + 15,000 Conceptual Captions entries and calls this roughly 67,500 examples, but the stated sum is 75,000; Appendix B instead says 42,000 MSCOCO + 10,000 Flickr30K + 15,000 Conceptual Captions, which sums to 67,000. For ViT-L, Section 4 says 125,000 examples (80,000 MSCOCO + 30,000 Flickr30K + 15,000 Conceptual Captions), while Appendix B says 120,000 examples (85,000 MSCOCO + 20,000 Flickr30K + 15,000 Conceptual Captions). Please reconcile these numbers and state the exact counts used for teacher and student training.
  4. [Appendix A] Appendix A states that the number of cross-modal transformer layers, attention heads, and hidden dimension for the teacher module were 'illustratively set ... and should be confirmed from implementation.' This is a missing specification for the central architectural component, and it makes the method impossible to reproduce from the paper text alone. Please provide the exact teacher hyperparameters in the main text or appendix, not only in code, and remove the 'should be confirmed' language.
minor comments (4)
  1. [§5.2] The text claims 'substantial text→image R@1 gains of 15–35 pp' across backbones, but Table 2 shows the largest absolute gain is 12 percentage points (ViT-B/16 on Flickr30K T→I R@1: 0.62 to 0.74). The stated range is consistent with relative gains of roughly 15–35%, so please reword the claim to say 'relative gains' or provide the correct pp values.
  2. [Table 4] Table 4 has missing spacing in several entries (e.g., 'Full DCLIP0.44 ... 0.820.59' and 'No YOLO ... 0.750.64'). Also, the 'No CMA' row reports a Flickr30K I→T R@1 of 0.89, which is higher than the full model's 0.88; this counterintuitive result is not discussed.
  3. [Appendix E] The text says 'Table 9 shows when we increase the distillation dataset from 67.5 K to 200 K image-text pairs', but Table 9 appears to report results on 1k subsets rather than a 220k training mixture. The cross-reference appears to be wrong, and the table supporting the 220k-dataset experiment seems to be missing.
  4. [§5.2 and Table 2] No seeds, error bars, or run-to-run variance are reported, and the paper uses the word 'significant' in a non-statistical sense. Given that some image-to-text margins are only 1–2 percentage points, please state whether the reported numbers are stable across multiple training runs or provide standard deviations.

Circularity Check

1 steps flagged · score 2.0 of 10

No central circularity: retrieval gains are external benchmark comparisons. One loss term is definitionally vacuous because the text-cosine distillation objective is identically zero when teacher and student text embeddings are the same frozen CLIP output.

  1. self definitional [Section 3.2 (Student Loss); Appendix C (Loss Function Formulations)]
    "To distill knowledge from the teacher, we apply cosine distillation losses that directly align student embeddings with their teacher counterparts: LT cos = 1−sim(z T s ,z T t ). ... Here, zT s are student text embeddings (from frozen CLIP text encoder), zT t are teacher text embeddings (identical to zT s )."

    By the paper's own architecture, the student text encoder is the frozen default CLIP text encoder, and the teacher text embedding is also a default CLIP text embedding (Appendix C: 'identical to zT s'). Hence sim(zT_s, zT_t) = 1 and L_T_cos = 1 − 1 = 0 identically for every input, independent of learned weights or data. The paper reports L_T_cos as part of the hybrid student loss and describes cosine distillation objectives guiding the student, but this particular term cannot transmit any information. It is a definitional no-op rather than a fitted prediction, and it does not touch the main retrieval or zero-shot comparisons, which are measured against external CLIP baselines.

full rationale

DCLIP's central claim — that distilling a YOLO/cross-attention teacher into a CLIP student improves Recall@K/MAP on MSCOCO and Flickr30K Karpathy splits while retaining ~94% ImageNet zero-shot accuracy — is an empirical comparison against external baselines (CLIP, RegionCLIP, FineCLIP, TinyCLIP). The training signal is not derived from the evaluation labels by construction, and no self-citation chain is load-bearing. The only definitional artifact I can exhibit is the text-cosine distillation term: since both the student and teacher text embeddings come from the same frozen CLIP text encoder, L_T_cos = 1 - sim(z,z) is identically zero. This is a minor structural redundancy in the reported loss, not a circular derivation of the results. The possible overlap between the 67.5k training sample and the Karpathy test images is a data-leakage / experimental-validity concern, not a circularity of the derivation chain; it would need external evidence and is not counted in this score.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical fine-tuning paper, so its ledger is dominated by hand-set hyperparameters and domain assumptions about CLIP, YOLO, and the benchmarks. The lack of reported values for region weighting, temperature, loss coefficients, and cross-attention dimensions is the main gap.

free parameters (6)
  • InfoNCE temperature tau
    Temperature in teacher and student contrastive losses (Appendix C Eq. 1) set by hand and not reported; directly controls alignment sharpness.
  • Region weighting coefficients
    Teacher weights each YOLO region using a penalty based on classification confidence, bounding box area, and cosine similarity to the text (Section 3.1); the coefficients are not specified.
  • Number of YOLO regions per image
    Region proposal count (top-k crops) is not stated; affects teacher embedding quality and aggregation.
  • Cross-modal attention architecture = NL=2, NH=8, DH=512 (illustrative, unconfirmed)
    Appendix A says these were illustratively set and should be confirmed from the implementation.
  • ViT-L aggregation clusters and ROPE = 3 clusters; RoPE embeddings
    Selected after observing imbalance in retrieval and zero-shot performance, a post-hoc design choice (Section 4).
  • Distillation loss coefficients
    Student loss is written as an unweighted sum; no coefficients are given, though the figures suggest lambda weights exist.
assumptions (4)
  • domain assumption Frozen CLIP text encoder provides a stable semantic anchor so fine-tuning only the image encoder preserves zero-shot classification.
    Central design assumption in Section 3.2; tables show it holds approximately, with notable degradation for ViT-L.
  • domain assumption YOLOv8x region proposals capture the objects and relations needed for the target retrieval tasks.
    Teacher relies on YOLO crops (Section 3.1); the limitations section admits this may fail on out-of-domain images.
  • domain assumption InfoNCE contrastive loss prevents representation collapse during distillation and maintains discriminative structure.
    Invoked in Section 3.2 and Appendix C; depends on prior contrastive learning results.
  • domain assumption Karpathy test split and ImageNet/CIFAR zero-shot sets are valid held-out evaluations.
    Standard benchmarks, but the paper does not state that Karpathy test images were excluded from the 50k MSCOCO training sample, so this assumption is not fully verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation." pith.science (2026). https://pith.science/paper/NMGCZGJ3

@misc{pith2026250521549,
  author       = {Pith},
  title        = {Pith review of: Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMGCZGJ3}},
  note         = {Machine review of arXiv:2505.21549}
}
read the original abstract

We present Distill CLIP (DCLIP), a fine-tuned variant of the CLIP model that enhances multimodal image-text retrieval while preserving the original model's strong zero-shot classification capabilities. CLIP models are typically constrained by fixed image resolutions and limited context, which can hinder their effectiveness in retrieval tasks that require fine-grained cross-modal understanding. DCLIP addresses these challenges through a meta teacher-student distillation framework, where a cross-modal transformer teacher is fine-tuned to produce enriched embeddings via bidirectional cross-attention between YOLO-extracted image regions and corresponding textual spans. These semantically and spatially aligned global representations guide the training of a lightweight student model using a hybrid loss that combines contrastive learning and cosine similarity objectives. Despite being trained on only ~67,500 samples curated from MSCOCO, Flickr30k, and Conceptual Captions-just a fraction of CLIP's original dataset-DCLIP significantly improves image-text retrieval metrics (Recall@K, MAP), while retaining approximately 94% of CLIP's zero-shot classification performance. These results demonstrate that DCLIP effectively mitigates the trade-off between task specialization and generalization, offering a resource-efficient, domain-adaptive, and detail-sensitive solution for advanced vision-language tasks. Code available at https://anonymous.4open.science/r/DCLIP-B772/README.md.

Figures

Figures reproduced from arXiv: 2505.21549 by the authors.

Figure 1
Figure 1. Overview of the DCLIP teacher architecture. Given an image input, YOLO first proposes [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the DCLIP student architecture. The student never sees the YOLO bounding [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Zero-shot classification performance over training epochs. (b) Text-to-image retrieval [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Pareto curve comparing zero-shot classification vs. retrieval performance. DCLIP traces an efficient Pareto frontier under lightweight supervision. ViT-L/14, due to its larger capacity, tends to overfit rapidly to embedding distributions during distillation. That being…
Figure 5
Figure 5. Figure 5: DCLIP Teacher Architecture Overview. YOLO extracts region-level features [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: DCLIP Student Architecture Overview. The student never sees the YOLO [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Zero-shot confusion matrices for DCLIP ViT-B/16 on CIFAR-10 and CIFAR-100 under [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 52 canonical work pages

  1. [1]

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., and Krueger, G. (2021). Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763. PMLR

    work page 2021

  2. [2]

    Li, J., Li, D., Xiong, C., and Hoi, S. (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational Conference on Machine Learning, pages 12888–12900. PMLR

    work page 2022

  3. [3]

    Gao, P., Geng, S., Zhang, R., Meng, T., Fang, Y ., Kingsbury, Y ., and Han, J. (2021). CLIP-Adapter: Better vision-language models with feature adapters.arXiv preprint arXiv:2110.04544

    arXiv 2021

  4. [4]

    Zhu, Y ., Qin, J., Feng, G., Liu, J., and Xu, C. (2024). Selective Vision-Language Subspace Projection for Few-shot CLIP.ResearchGate Publication, 382526585

    work page 2024

  5. [5]

    Cui, C., Ragab, M., Jiao, Y ., Ramanathan, V ., and Saraf, A. (2023). Universal Projection: Making Modality-Agnostic Structural Knowledge Accessible to Any Model.arXiv preprint arXiv:2312.03700

    arXiv 2023

  6. [6]

    Ni, B., Lin, X., Wang, X., Chen, S., Zou, X., Jin, S., Zhou, X., Fang, H., Xu, Z., and Jiao, X. (2022). Fine-tuned CLIP models are efficient video learners.arXiv preprint arXiv:2212.03640

    arXiv 2022

  7. [7]

    Wang, Z., Cai, Z., Wang, Y ., Yang, Z., and Xiong, C. (2023). Efficient Multi-Task Learning via Decoupled Modular Architecture. InProceedings of the 2023 ACM International Conference on Multimedia Retrieval, pages 79–88

    work page 2023

  8. [8]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Min- derer, M., Heigold, G., Gelly, S., and Uszkoreit, J. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations

    work page 2021

Show all 68 references
  1. [9]

    He, K., Chen, X., Xie, S., Li, Y ., Dollár, P., and Girshick, R. (2022). Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009

    2022

  2. [10]

    Sharma, L.D., Manchanda, A., and Awasthi, P. (2021). Evaluation of effectiveness of patch based image classification technique using high resolution WorldView-2 image.ResearchGate Publication, 357298329

    2021

  3. [11]

    Hendrycks, D., and Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. InInternational Conference on Learning Representations

    2019

  4. [12]

    Liu, C., Li, Z., Song, S., Zhou, L., and Zhang, Y . (2023). Masked Image Modeling Advances 3D Medical Image Analysis. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5548–5558

    2023

  5. [13]

    Chambon, P., Lample, G., Shleifer, S., Agrawa, M., Makhzani, A., Kramar, M., V orontsov, E., and Miotto, R. (2023). Adapting Large Vision-Language Models to Medical Image Understanding.arXiv preprint arXiv:2304.07193

    2023 arXiv

  6. [14]

    Federico, F., Deza, A., Kreiman, G., and Vaziri, A. (2024). Optimal visual representations for generative text-to-image models: Lessons from a competition.arXiv preprint arXiv:2404.09707

    2024 arXiv

  7. [16]

    Bao, H., Dong, L., and Wei, F. (2022). BEiT: BERT Pre-Training of Image Transformers. InInternational Conference on Learning Representations. 10

    2022

  8. [17]

    Xie, Z., Zhang, Z., Cao, Y ., Lin, Y ., Bao, J., Yao, Z., Dai, Q., and Hu, H. (2022). SimMIM: A Simple Framework for Masked Image Modeling. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663

    2022

  9. [18]

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerg- ing Properties in Self-Supervised Vision Transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660

    2021

  10. [19]

    He, K., Fan, H., Wu, Y ., Xie, S., and Girshick, R. (2020). Momentum Contrast for Unsupervised Visual Representation Learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738

    2020

  11. [20]

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. (2020). Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. InAdvances in Neural Information Processing Systems

    2020

  12. [21]

    Li, J., Li, D., Xiong, C., and Hoi, S. (2021). ALBEF: Align Before Fuse: Vision and Language Representa- tion Learning with Momentum Distillation. InAdvances in Neural Information Processing Systems

    2021

  13. [22]

    Jia, C., Yang, Y ., Xia, Y ., Chen, Y ., Parekh, Z., Pham, H., Le, Q., Sung, Y ., Li, Z., and Duerig, T. (2021). Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In International Conference on Machine Learning, pages 4904–4916. PMLR

    2021

  14. [23]

    Li, S., Hu, X., Yu, G., Tsvetkov, Y ., Awadallah, A.H., Preston, V ., and Yu, F. (2022). ELEV ATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models. InAdvances in Neural Information Processing Systems

    2022

  15. [24]

    Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B., Haque, I., Beery, S., Leskovec, J., Kundaje, A., Pierson, E., Levine, S., Finn, C., and Liang, P...

    2021

  16. [25]

    Wang, M., Deng, W., Yu, J., Li, Z., You, T., Zhang, J., Zhou, Y ., Yi, S., Zheng, S., Wang, C., and Li, X. (2020). Visual Domain Adaptation Challenge 2020.arXiv preprint arXiv:2008.11191

    2020 arXiv

  17. [26]

    van den Oord, A., Li, Y ., and Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding.arXiv preprint arXiv:1807.03748

    2018 arXiv

  18. [27]

    Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014). Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, Cham

    2014

  19. [28]

    Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014). From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the Association for Computational Linguistics, 2, pages 67–78

    2014

  20. [29]

    Sharma, P., Ding, N., Goodman, S., and Soricut, R. (2018). Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018

  21. [30]

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE

    2009

  22. [31]

    Jocher, G., Chaurasia, A., & Qiu, J. (2023). YOLO by Ultralytics. GitHub repository. https://github. com/ultralytics/ultralytics

    2023

  23. [32]

    Chen, Z., Liu, Z., Wu, C., Lin, Z., & Wang, L. (2024). Long-CLIP: A Parameter-Efficient Approach for Long Text-Image Understanding.arXiv preprint arXiv:2405.20204

    2024 arXiv

  24. [34]

    M., Asano, Y

    Najdenkoska, I., Derakhshani, M. M., Asano, Y . M., van Noord, N., Worring, M., & Snoek, C. G. M. (2024). TULIP: TOKEN-LENGTH UPGRADED CLIP.arXiv preprint arXiv:2401.07056

    2024 arXiv

  25. [35]

    I., Farajtabar, M., Li, A., Ghasemzadeh, H., & Saupe, D

    Mirzadeh, S. I., Farajtabar, M., Li, A., Ghasemzadeh, H., & Saupe, D. (2020). Improved knowledge distillation via teacher assistant. InProceedings of the AAAI Conference on Artificial Intelligence(V ol. 34, No. 04, pp. 5191-5198). 11

    2020

  26. [36]

    Beyer, L., Zhai, X., Roy, A., Markeev, D., Anil, R., & Kolesnikov, A. (2022). Knowledge distillation: A good teacher is patient and consistent. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(pp. 10913-10923)

    2022

  27. [37]

    Jing, D., He, X., Luo, Y ., Fei, N., Yang, G., Wei, W., Zhao, H., and Lu, Z. (2024). FineCLIP: Self- distilled Region-based CLIP for Better Fine-grained Understanding. InConference on Neural Information Processing Systems (NeurIPS)

    2024

  28. [38]

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, A., Girshick, R., He, K., and Dollár, P. (2023). Segment Anything. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  29. [39]

    Ren, S., He, K., Girshick, R., and Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. InAdvances in Neural Information Processing Systems (NIPS), pages 91–99

    2015

  30. [40]

    Zhao, B., Meng, L., He, X., and Zhao, D. (2022). Distilling Knowledge from Unimodal to Multimodal Models.arXiv preprint arXiv:2207.02441

    2022 arXiv

  31. [41]

    Fang, K., Li, J., Zhang, P., Wang, X., and Hu, D. (2021). Distilling Cross-Modal Knowledge for Vision- Language Pre-Training.arXiv preprint arXiv:2112.08793

    2021 arXiv

  32. [42]

    Li, G., Lin, Z., Wu, S., Zhang, H., and Yang, J. (2023). Feature Distillation for Vision-Language Models. arXiv preprint arXiv:2303.00580

    2023 arXiv

  33. [43]

    Tian, Y ., Pei, D., He, K., and Keutzer, K. (2020). Contrastive Representation Distillation. InInternational Conference on Learning Representations (ICLR)

    2020

  34. [44]

    Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V ., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. (2021). OpenCLIP.Zenodo. https://doi.org/10.5281/zenodo.5143773

    2021 doi

  35. [45]

    Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y ., and Gao, J. (2019). ERNIE-ViL: Knowledge Enhanced Vision-Language Pre-training for Visual Question Answering and Image Captioning. arXiv preprint arXiv:1910.02961

    2019 arXiv

  36. [46]

    Yuan, L., Chen, D., Chen, Y .L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., Liu, C., Liu, M., Liu, Z., Lu, Y ., Ma, Y ., Mao, J., Ren, J., Sun, M., Sun, P., Tulyakov, S., Wang, W., Wang, Z., Yao, Y ., Zhang, L., Zhang, Z., Zhou, L., and Hou, Q. (2021). F...

    2021 arXiv

  37. [47]

    Yu, J., Wang, Z., Vasudevan, V ., Yeung, L., Seyedhosseini, M., and Wu, Y . (2022). Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917

    2022 arXiv

  38. [49]

    Zhang, B., Zhang, P., Dong, X., Zang, Y ., and Wang, J. (2024). Long-CLIP: Unlocking the long-text capability of CLIP.arXiv preprint arXiv:2403.15378

    2024 arXiv

  39. [50]

    M., Asano, Y

    Najdenkoska, I., Derakhshani, M. M., Asano, Y . M., van Noord, N., Worring, M., and Snoek, C. G. M. (2024). TULIP: TOKEN-LENGTH UPGRADED CLIP.arXiv preprint arXiv:2410.10034

    2024 arXiv

  40. [51]

    Xu, J., Xu, S., Wu, C., Wang, X., and Lin, S. (2022). GroupViT: Semantic Segmentation Emerges from Text Supervision. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17800–17810

    2022

  41. [52]

    Lu, J., Batra, D., Parikh, D., and Lee, S. (2019). ViLBERT: Pretraining for Vision-and-Language Represen- tation Learning. InAdvances in Neural Information Processing Systems (NeurIPS), 32

    2019

  42. [53]

    Tan, H., and Bansal, M. (2019). LXMERT: Learning Cross-Modality Encoder Representations from Trans- formers. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL...

    2019

  43. [54]

    Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., Choi, Y ., and Gao, J. (2020). Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. InEuropean Conference on Computer Vision (ECCV), pages 121–137. Springer, Cham. 12

    2020

  44. [55]

    Chen, Y .C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y ., and Liu, J. (2020). UNITER: UNiversal Image-TExt Representation Learning. InEuropean Conference on Computer Vision (ECCV), pages 104–120. Springer, Cham

    2020

  45. [56]

    Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y ., and Gao, J. (2021). VinVL: Revisiting Visual Representations in Vision-Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5579–5588

    2021

  46. [57]

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. InEuropean Conference on Computer Vision (ECCV), pages 213–229. Springer, Cham

    2020

  47. [58]

    Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y ., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., Chang, K.W., and Gao, J. (2022). Grounded Language-Image Pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10...

    2022

  48. [59]

    Zhong, Y ., Li, L.H., Zhang, P., Yang, J., Li, C., Yuan, L., Zhang, L., and Gao, J. (2022). RegionCLIP: Region-based Language-Image Pretraining. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16793–16803

    2022

  49. [60]

    Krishna, R., Zhu, Y ., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y ., Li, L.J., Shamma, D.A., Bernstein, M.S., and Fei-Fei, L. (2017). Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.International Journal of C...

    2017

  50. [61]

    Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network.arXiv preprint arXiv:1503.02531

    2015 arXiv

  51. [62]

    Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gkioxari, G., and Bengio, Y . (2014). FitNets: Hints for Thin Deep Nets.arXiv preprint arXiv:1412.6550

    2014 arXiv

  52. [63]

    Zagoruyko, S., and Komodakis, N. (2016). Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer.arXiv preprint arXiv:1612.03928

    2016 arXiv

  53. [64]

    Park, W., Kim, D., Lu, Y ., and Cho, M. (2019). Relational Knowledge Distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3967–3976

    2019

  54. [65]

    Wu, K., Peng, H., Chen, Z., Liu, Z., and Wang, L. (2022). TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance.arXiv preprint arXiv:2207.09648

    2022 arXiv

  55. [66]

    Furlanello, T., Fenu, S., Starovoitov, V ., Sakellaridi, A., and Larlus, D. (2018). Born again neural networks. InInternational conference on machine learning, pages 1607–1616. PMLR

    2018

  56. [67]

    Yang, C., Wang, S., and Tao, D. (2022). Meta Knowledge Distillation. InInternational Conference on Learning Representations (ICLR)

    2022

  57. [68]

    S., Wang, X., and Wang, J

    Liu, Y ., Zhang, X. S., Wang, X., and Wang, J. (2022). MetaDistiller: Flower-Inspired Knowledge Distilla- tion. InProceedings of the 30th ACM International Conference on Multimedia, pages 3436–3445

    2022

  58. [69]

    Zhao, B., Cui, Q., Song, R., Qiu, Y ., and Liang, J. (2022). Revisiting knowledge distillation: A teacher- student optimization perspective. InAdvances in Neural Information Processing Systems, 35, pages 1009–1022

    2022

  59. [70]

    Karpathy, A., and Fei-Fei, L. (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3128–3137. The MSCOCO Karpathy split is derived from this work

    2015

  60. [71]

    simple cosine distillation preservation loss between the student’s image embeddings and the original CLIP ViT-L model’s image embeddings

    Su, J., Lu, Y ., Pan, S., Wen, B., and Liu, Y . (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. InCoRR, abs/2104.09864. 13 Appendix: DCLIP Further Details This appendix provides supplementary information to the main paper, including detailed hyperparamet...

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.