Pith. sign in

REVIEW 5 major objections 6 minor 75 references

Decoupled Global-Local Alignment for Improving Compositional Understanding

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DeGLA beats hard-negative baselines and keeps CLIP's old skills

desk verdict DeGLA shows a real trade-off improvement for CLIP fine-tuning, but the compositional gain is mostly order-type; worth a serious referee. read the letter →

arxiv 2504.16801 v3 pith:2TSQYNWW submitted 2025-04-23 cs.CV

Xiaoxing Hu , Kaicheng Yang , Jun Wang , Haoran Xu , Ziyong Feng , Yupei Wang This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelscompositionalunderstandingCLIPhardnegativesself-distillationcontrastivelearningzero-shotclassificationLLM-generatedcaptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contrastive vision-language models compare whole images with whole captions, which leaves them blind to compositional structure: they cannot reliably tell the dog chases the cat from the cat chases the dog. This paper claims that fine-tuning on hard negatives can fix that blindness without the usual catastrophic forgetting, if global alignment is decoupled from local alignment and anchored to the pretrained model. The proposed DeGLA framework keeps a frozen exponential-moving-average teacher copy of the CLIP encoders, adds image-grounded and text-grounded contrast losses over LLM-generated hard negative captions, and reports an average gain of 3.5% over CE-CLIP on VALSE, SugarCrepe, and ARO while raising zero-shot classification by 13.0% on average across eleven datasets. The central claim is that hard-negative fine-tuning and knowledge retention are not opposing goals when the loss structure keeps the student close to the pretrained representation space.

What carries the argument

The load-bearing object is the decoupling itself: a global alignment channel, the standard image-to-text and text-to-image InfoNCE losses with four hard negative captions per text, is stabilized by an EMA self-distillation loss, while a separate local alignment channel sharpens compositionality through the Image-Grounded Contrast (IGC) loss and the Text-Grounded Contrast (TGC) loss. IGC uses the image as anchor and contrasts the positive caption against K negatives; TGC uses the frozen EMA text embedding as a stable positive anchor within the text modality, preventing the text encoder from drifting into a narrow fine-tuning space. The data feeding both channels comes from an LLM-driven negative-caption pipeline with five rewrite rules, whole-sentence reshuffle, noun swap, adjective swap, adjective replacement, and noun replacement, scaled to five negatives per COCO caption for about 2.07 million negatives in total. The machinery lets the model learn compositionality from minimal textual edits while the EMA anchor preserves the pretrained representation geometry.

What would settle it

A random sample of 200 generated negative captions could be labeled by human annotators as hard negative, hard positive (same scene and meaning, only wording changed), or easy negative (semantics changed beyond one minimal edit); if hard positives or easy negatives make up a large share, the data premise of DeGLA is falsified. A cleaner experiment would rerun DeGLA with an automatic filter that keeps only captions whose frozen-CLIP similarity to the source falls inside a hard-negative band, then check whether the compositional gains shrink, stay, or grow.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the trade-off previous hard-negative methods accept, compose better but forget more, is not forced. DeGLA keeps the global CLIP contrastive objective but augments it with hard negatives and with a self-distillation term that penalizes the squared distance between learnable and frozen EMA teacher embeddings of images, texts, and negative texts. On the local side, an Image-Grounded Contrast loss pulls the image embedding toward the correct caption and away from the generated negatives, while a Text-Grounded Contrast loss uses the frozen EMA text embedding as the positive anchor so the text encoder does not overfit to the fine-tuning distribution. The paper reports that this decoupling yields an average 3.5% compositional gain over CE-CLIP on VALSE, SugarCrepe, and ARO, and a 13.0% average zero-shot classification improvement across eleven datasets, with general capability largely retained: DeGLA averages 58.7% zero-shot accuracy versus CLIP's 61.0% and CE-CLIP's 45.7%.

Load-bearing premise

The load-bearing premise is that the roughly 2.07 million LLM-generated captions are actually hard negatives, minimally altered sentences with genuinely different semantics, rather than easy negatives or hard positives; the paper hand-selects 50 seed examples per rewrite rule but does not evaluate the full generated set.

Editorial extensions

If this is right

  • The paper's ablations show that each component pays: the LLM negatives add 23.4 points on ARO over CLIP, IGC adds 4.0, TGC adds 1.4, and the self-distillation term recovers 1.8 points of zero-shot accuracy while costing only 0.1 on ARO.
  • If the reported averages hold, DeGLA becomes the reference point for compositional fine-tuning of CLIP-sized models, beating CE-CLIP on every SugarCrepe group and on both ARO order tasks.
  • The remaining gap is explicit in the tables: DeGLA's ARO relation and attribute scores trail CE-CLIP, which the paper attributes to its negative-generation diversity being broader rather than relation- and attribute-focused.
  • Because the EMA anchor is a single squared-distance term, the mechanism can be dropped into existing hard-negative fine-tuning recipes without changing the pretraining or evaluation protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same EMA-anchored global-local split could apply to other narrow fine-tuning regimes, such as debiasing, domain adaptation, or retrieval distillation, since nothing in the loss is specific to compositionality.
  • The data premise could be tightened with an automatic hard-negative filter, for example keeping generated captions whose frozen-CLIP similarity to the source falls inside a hard band; the paper validates only 50 hand-picked seed examples per rewrite rule.
  • A relation-focused negative subtype that swaps grammatical roles rather than replacing or swapping nouns and adjectives is the obvious complement, because the ARO relation scores show where DeGLA leaves points on the table.
  • The zero-shot table also shows that retention is partial: DeGLA at 58.7% average remains 2.3 points below the original CLIP's 61.0%, so the honest framing is much less forgetting rather than no forgetting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DeGLA, a fine-tuning framework for CLIP that aims to improve compositional understanding while mitigating catastrophic forgetting of general capabilities. The method has three components: (1) an LLM-driven pipeline (ChatGPT-4-Turbo for 200 seed examples per type, manual selection of 50, then Llama-3.1-8B-Instruct to generate about 2.07M hard negative captions from COCO 2014 train captions, covering five subtypes under intra-sentence reshuffling and minimal semantic substitution); (2) a global alignment loss (Eq. 4-5) that incorporates hard negative texts into the CLIP contrastive loss; (3) a self-distillation regularizer (Eq. 7) using an EMA teacher (image and text encoders) to preserve pretrained knowledge; and (4) two local alignment losses, Image-Grounded Contrast (IGC, Eq. 8) and Text-Grounded Contrast (TGC, Eq. 9), with the latter using the frozen EMA text embedding as anchor. The total loss is Eq. (10). Experiments are conducted on CLIP ViT-B/32, fine-tuned for 5 epochs on COCO, and evaluated on VALSE, SugarCrepe, ARO, zero-shot classification on 11 datasets, linear probe, and zero-shot retrieval on MSCOCO/Flickr30k. The headline claims are an average +3.5% over CE-CLIP on VALSE/SugarCrepe/ARO and +13.0% over CE-CLIP on zero-shot classification, leading to the stated 'more effective balance' between compositional and general capabilities.

Significance. If the reported results hold, the paper would make a useful contribution to the line of work on hard-negative fine-tuning of CLIP: the decoupled global-local formulation with an EMA self-distillation teacher is a plausible mechanism for retaining pretrained zero-shot capability while still gaining on compositionality benchmarks. The LLM-driven negative generation pipeline with five explicit subtypes is straightforward and applicable to other models, and the paper provides a clear ablative breakdown (Table 7) attributing the general-capability retention mainly to the distillation term. The paper also ships code and includes detailed prompt templates in the appendix. The significance is moderate: the gains are on established but narrow benchmarks, the method is incremental over CE-CLIP (which already used ranking/image-grounded hard negatives and intra-modal hard negatives), and the claim of an 'optimal balance' rests on averages that mask a regression on ARO relation/attribute subsets.

major comments (5)
  1. [Abstract, Figure 5, Section 4.4] The central claim that DeGLA achieves 'a more effective balance between compositional reasoning and general comprehension capabilities' is over-scoped. The +3.5% average over VALSE, SugarCrepe, and ARO hides the fact that DeGLA is below CE-CLIP on ARO Relation (81.6 vs 83.9) and ARO Attribute (74.3 vs 76.4) in Table 4, and its ARO lead comes almost entirely from the Order subsets. The paper itself acknowledges this in Section 4.2 ('it still trails CE-CLIP and Structure-CLIP in the domains of relations and attributes'), but Figure 5 and the abstract are still phrased as an unconditional improvement. For the balance claim to be supported, the authors should either restrict the claim to order-type compositionality plus zero-shot retention, or provide an explicit justification for treating the three benchmarks as commensurable dimensions of a single 'compositional reasoning' average.
  2. [Section 3.2 and Figure 10] The assumption that the Llama-3.1-8B generated captions are genuinely hard negatives is not validated. The paper asserts semantic divergence via prompt constraints and manual selection of 50 ChatGPT exemplars, but no automatic or human evaluation is reported for the 2.07M generated negatives. Inspection of Figure 10 shows that several generated items are not hard negatives: e.g., the 'subtype 2' example 'Two young children playing around a fire hydrant' → 'Fire hydrant around playing a young two children' is a non-grammatical word salad that the model will trivially reject, while 'A gray cat is sitting on a wooden bench' → 'A goldfish is sitting on a concrete road' changes scene composition completely and is an easy negative rather than a hard one. The paper should quantify the difficulty and validity of the generated negatives, for example by reporting CLIP similarity distributions between positive and generated captions before and after post-filtering, and by reporting human-annotation agreement on a subsample. Without this, the contribution of the data-generation pipeline relative to rule-based or unmasking-based baselines is not cleanly established.
  3. [Section 4.1, Table 8] The hyperparameters lambda_1, lambda_2, lambda_3 and EMA alpha are tuned without a clearly separated validation protocol, and no error bars or multiple-seed results are reported anywhere in the paper. The ablation of lambda_1/lambda_2 in Figure 6a shows a rather flat landscape with an apparent optimum at (0.1, 0.1), but the selection procedure over these values (and its potential leakage into the benchmark average) is not described. Given that the headline 3.5%/13.0% improvements are point estimates over a single run, the authors should report at least two or three seeds with means and standard deviations for the main tables, and describe how the validation set was chosen for hyperparameter selection (e.g., a held-out subset of ARO or a separate validation benchmark).
  4. [Table 7 and Section 4.4] The component ablation in Table 7 is reported only on ARO and zero-shot classification, and the row ordering makes the individual contribution of IGC ambiguous. The row 'CN + IGC' (84.8 ARO, 55.1 ZS) compared with 'CN' alone (80.8, 55.4) and 'CN + TGC' (82.2, 57.2) shows IGC and TGC each help ARO, but the subsequent 'CN + IGC + TGC' row (86.2, 56.9) shows a much larger ARO gain (+5.4 over CN+TGC) than the sum of individual gains, which is not discussed. More importantly, the table does not isolate the effect of the EMA self-distillation from the effect of the frozen teacher embeddings used inside TGC (Eq. 9); since TGC already uses the frozen EMA text embedding as its anchor, the 'SD' row that adds L_Distill may be partly redundant with the stabilizing effect already present in TGC. The ablation as presented therefore does not cleanly attribute general-capability retention to the distillation term.
  5. [Appendix A.4 and Section 4.1] The connection between the five generated subtypes and the training-time pairing K=4 is under-specified. The paper states that K=4 'due to the merging of two subtypes, as detailed in the appendix', but the appendix does not actually explain which subtypes were merged, how the four negatives per text were sampled across the five subtypes, whether the distribution over subtypes was balanced, or whether each subtype was paired with its corresponding positive text in a controlled way. This makes it difficult to reproduce the training data construction exactly.
minor comments (6)
  1. [Abstract and Section 3.3] There are inconsistencies in acronym usage: the Introduction and Figure 3 captions refer to 'ICC' and 'TCC', while the Method and experiments use 'IGC' and 'TGC'; the abstract and conclusion also give the variant spellings. The paper should standardize on one set of acronyms.
  2. [Throughout] Numerous typos appear in the text, table headers, and appendix. These include 'liner probe' (Tables 6 and Section 4.3), 'VASLE' in the appendix A.3, 'Composionablity' in Table 11 header, 'Detatils' in A.2, a duplicate reference (Oord et al. appears as [45] and [46]), 'Preformance' in Figure 5 axis, and the unmatched parenthesis in Eq. (8). A careful proofreading pass is needed.
  3. [Section 4.1] The implementation paragraph lists parameters 'beta_1 and beta_2' as 0.9 and 0.98, which is standard for AdamW, but the supplementary Table 8 does not list beta_1/beta_2; adding them to the table would make the configuration complete.
  4. [Tables 3 and 4] In Table 3 the 'Avg.' column for the Replace/Relation group appears to be computed over the three replacement types while the table layout suggests a different grouping; in Table 4, CLIP's row shows Relation 59.2 and Attribute 62.9, but the average 57.4 below it is not the average of the visible four columns (it appears to be the average of the four ARO sub-tasks, which would be 57.65). The definition of the reported average should be stated in the captions.
  5. [Section 3.2 and Figure 2] The paper refers to 'ChatGPT4-Turbo' and 'ChatGPT-4-Turbo' with inconsistent hyphenation and no version/date information; specifying the exact model snapshot and sampling parameters would help reproducibility, especially since generation prompts appear in the appendix but the decoding temperature and number of samples are not reported.
  6. [Appendix Figure 10] The first column of Figure 10 is labeled 'Positive caption' and the second 'Hard negative examples', but the fourth row shows the original-positive caption repeated as a hard negative ('A red bus going down the road behind a blue bus' appears both as the input and as one of the generated examples). This suggests the filtering may not have removed near-duplicate outputs; the authors should clarify whether such cases were actually included in the training set and how they were handled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DeGLA's gains are empirical measurements against external benchmarks, not derivations from fitted inputs or self-cited uniqueness claims.

full rationale

The paper's load-bearing claims are empirical: DeGLA is fine-tuned with a combination of global contrastive loss, IGC/TGC local losses, and an EMA-based distillation term, and its compositional and general abilities are then measured on fixed external benchmarks (VALSE, SugarCrepe, ARO, zero-shot classification, retrieval). No equation in the paper reduces the reported improvement to a fitted parameter by construction. The EMA teacher is a moving average of the student encoder, so the self-distillation constraint is a regularizer rather than a hidden re-introduction of the target result; whether it actually preserves general capability is verified by held-out zero-shot and linear-probe evaluations, not by definition. The LLM-generated hard negatives are an input data-generation choice; the paper's claim that they are high-quality is supported only by prompt constraints and a small manual selection, which is a data-quality limitation rather than a circularity. Likewise, the evaluation benchmarks overlapping with the COCO fine-tuning set is a potential distributional or contamination concern, but it is not a case of a prediction being identical to a training target by construction. The only apparent self-citations (RWKV-CLIP [14], ALIP [59], CLIP-CID [60], and related multimodal works) appear in related work or in the evaluation-protocol description, and none supplies a load-bearing premise or a uniqueness theorem. Therefore there is no circular step to report under the specified criteria, and the honest finding is a score of 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The EMA teacher is a standard algorithmic component, not a new entity.

free parameters (5)
  • lambda_1 = 0.1
    Weight for the Image-Grounded Contrast loss, selected by hyperparameter search (Section 4.1, Figure 6a).
  • lambda_2 = 0.1
    Weight for the Text-Grounded Contrast loss, selected by hyperparameter search (Section 4.1, Figure 6a).
  • lambda_3 = 0.005
    Weight for the self-distillation loss, selected by hyperparameter search (Section 4.1, Figure 6b).
  • EMA_decay_alpha = 0.9996
    Controls the update speed of the frozen teacher model; set by hand in Section 4.1.
  • K_negatives_per_text = 4
    Number of negative captions paired with each positive text during training; chosen due to merging of two subtypes, as noted in Section 3.3.
assumptions (4)
  • standard math L2-normalized embeddings and dot product define the contrastive similarity.
    Standard CLIP/InfoNCE assumption, used in Equations 1-4 and 7-9.
  • domain assumption The EMA teacher preserves the pretrained knowledge and provides a stable distillation target.
    Assumed in Section 3.3; the paper provides no analysis of why the EMA trajectory protects general features beyond the empirical zero-shot results.
  • domain assumption The LLM-generated captions are hard negatives with meaning distinct from the originals.
    Section 3.2 and appendix prompts rely on this; no quantitative quality metric or human evaluation is reported.
  • domain assumption Fine-tuning on MSCOCO transfers to the ARO, VALSE, and SugarCrepe benchmarks.
    These benchmarks are built from or similar to COCO; the paper follows NegCLIP and CE-CLIP in this protocol, but this limits generalization claims outside COCO-like data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoupled Global-Local Alignment for Improving Compositional Understanding." pith.science (2026). https://pith.science/paper/2TSQYNWW

@misc{pith2026250416801,
  author       = {Pith},
  title        = {Pith review of: Decoupled Global-Local Alignment for Improving Compositional Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TSQYNWW}},
  note         = {Machine review of arXiv:2504.16801}
}
read the original abstract

Contrastive Language-Image Pre-training (CLIP) has achieved success on multiple downstream tasks by aligning image and text modalities. However, the nature of global contrastive learning limits CLIP's ability to comprehend compositional concepts, such as relations and attributes. Although recent studies employ global hard negative samples to improve compositional understanding, these methods significantly compromise the model's inherent general capabilities by forcibly distancing textual negative samples from images in the embedding space. To overcome this limitation, we introduce a Decoupled Global-Local Alignment (DeGLA) framework that improves compositional understanding while substantially mitigating losses in general capabilities. To optimize the retention of the model's inherent capabilities, we incorporate a self-distillation mechanism within the global alignment process, aligning the learnable image-text encoder with a frozen teacher model derived from an exponential moving average. Under the constraint of self-distillation, it effectively mitigates the catastrophic forgetting of pretrained knowledge during fine-tuning. To improve compositional understanding, we first leverage the in-context learning capability of Large Language Models (LLMs) to construct about 2M high-quality negative captions across five types. Subsequently, we propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC) loss to enhance vision-language compositionally. Extensive experimental results demonstrate the effectiveness of the DeGLA framework. Compared to previous state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average performance improvement of 13.0% on zero-shot classification tasks across eleven datasets. Our code will be released at https://github.com/xiaoxing2001/DeGLA

Figures

Figures reproduced from arXiv: 2504.16801 by the authors.

Figure 1
Figure 1. (a) General performance comparison across 11 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our proposed LLM-driven neg [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The proposed training framework of DeGLA. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Zero-shot image-text retrieval performance com [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Ablation of the different loss weights and hard negative types. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The case study between CLIP and DeGLA. Green ✔ indicates true caption, while red ✗ indicates false caption. The bar chart represents the model’s prediction results. capabilities, as demonstrated by CE-CLIP, which, despite signifi￾cant gains in compositional reasoning c…
Figure 8
Figure 8. Figure 8: Details of the prompt utilized for generating high-quality examples and negative captions [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Rewritten examples generated by ChatGPT. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Hard negative examples generated by LLaMA3.1-8B-instruct. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 38 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. Springer, 446–461

    2014

  3. [3]

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Pi- otr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)

    arXiv 2015

  4. [4]

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. 2024. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In CVPR. 4113–4123

    work page 2024

  5. [5]

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and An- drea Vedaldi. 2014. Describing textures in the wild. In CVPR. 3606–3613

    2014

  6. [6]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. Ieee, 248–255

    2009

  7. [7]

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. 2023. Dense and aligned captions (dac) promote compositional reasoning in vl models. NIPS 36 (2023), 76137–76150

    work page 2023

  8. [8]

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. 2023. Teaching structured vision & language concepts to vision & language models. In CVPR. 2657–2668

    work page 2023

Show all 75 references
  1. [9]

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. 2023. Improving clip training with language rewrites. NIPS 36 (2023), 35544–35575

    2023

  2. [10]

    Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR. IEEE, 178–178

    2004

  3. [11]

    Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. 2023. HiCLIP: Contrastive language-image pretraining with hierarchy-aware attention. ICLR (2023)

    2023

  4. [12]

    Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. 2022. Cyclip: Cyclic contrastive language-image pretraining.NIPS 35 (2022), 6704–6719

    2022

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  6. [14]

    Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, and Jiankang Deng. 2024. RWKV-CLIP: a robust vision-language representa- tion learner. arXiv preprint arXiv:2406.06973 (2024)

    2024 arXiv

  7. [15]

    Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. 2025. Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs. arXiv preprint arXiv:2504.17432 (2025)

    2025

  8. [16]

    Tiancheng Gu, Kaicheng Yang, Chaoyi Zhang, Yin Xie, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, and Jiankang Deng. 2025. RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm. arXiv preprint arXiv:2502.12513 (2025)

    2025 arXiv

  9. [17]

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. 2021. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021)

    2021 arXiv

  10. [18]

    Geoffrey Hinton. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  11. [19]

    Lukas Hoyer, David Joseph Tan, Muhammad Ferjad Naeem, Luc Van Gool, and Federico Tombari. 2025. Semivl: Semi-supervised semantic segmentation with vision-language guidance. In ECCV. Springer, 257–275

    2025

  12. [20]

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2024. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. NIPS 36 (2024)

    2024

  13. [21]

    Libo Huang, Yan Zeng, Chuanguang Yang, Zhulin An, Boyu Diao, and Yongjun Xu. 2024. eTag: Class-Incremental Learning via Embedding Distillation and Task-Oriented Generation. In AAAI, Vol. 38. 12591–12599

    2024

  14. [22]

    Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. 2024. Structure- CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations. In AAAI, Vol. 38. 2417–2425

    2024

  15. [23]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

    2019 arXiv

  16. [24]

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s" up" with vision- language models? Investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310.19785 (2023)

    2023 arXiv

  17. [25]

    Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, and Ranjay Krishna. 2024. The hard positive truth about vision-language compositionality. In ECCV. Springer, 37–54

    2024

  18. [26]

    Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR. 3128–3137

    2015

  19. [27]

    Prannay Kaul, Weidi Xie, and Andrew Zisserman. 2023. Multi-modal classifiers for open-vocabulary object detection. In ICML. PMLR, 15946–15969

    2023

  20. [28]

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object repre- sentations for fine-grained categorization. In ICCV. 554–561

    2013

  21. [29]

    Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009)

    2009

  22. [30]

    Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. 2022. Uniclip: Unified framework for contrastive language-image pre-training. NIPS 35 (2022), 1008–1019

    2022

  23. [31]

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl

  24. [32]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. PMLR, 19730–19742

    2023

  25. [33]

    arXiv preprint arXiv:2201.03546 (2022)

    Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 (2022)

    2022 arXiv

  26. [34]

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He

  27. [35]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML. PMLR, 12888–12900

    2022

  28. [36]

    Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, and Zhigeng Pan. 2021. Online knowledge distillation for efficient pose estimation. In ICCV. 11740–11750

    2021

  29. [37]

    Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023. Vera: A general-purpose plausibility estimation model for commonsense statements. arXiv preprint arXiv:2305.03695 (2023)

    2023 arXiv

  30. [38]

    Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. 2023. Curriculum temperature for knowledge distillation. In AAAI, Vol. 37. 1504–1512

    2023

  31. [39]

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi

  32. [40]

    Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. 2022. Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting. arXiv preprint arXiv:2210.07179 (2022)

    2022 arXiv

  33. [41]

    Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi. 2023. Codet: Co- occurrence guided region-word alignment for open-vocabulary object detection. NIPS 36 (2023), 71078–71094

    2023

  34. [42]

    John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

  35. [43]

    Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. 2022. Slip: Self- supervision meets language-image pre-training. In ECCV. Springer, 529–544

    2022

  36. [44]

    Maria-Elena Nilsback and Andrew Zisserman. 2008. Automated flower classifica- tion over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing . IEEE, 722–729

    2008

  37. [45]

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. 2022. Simple open-vocabulary object detection. In ECCV. Springer, 728–755

    2022

  38. [46]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  39. [47]

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. 2021. VALSE: A task-independent benchmark for vision and language models centered on linguistic phenomena. arXiv preprint arXiv:2112.07566 (2021)

    2021 arXiv

  40. [48]

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In CVPR. IEEE, 3498–3505

    2012

  41. [49]

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. 2024. Syn- thesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Under- standing. In CVPR. 13279–13288

    2024

  42. [51]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, MM ’25, October 27–31, 2025, Dublin, Ireland Xiaoxing Hu et al. et al. 2021. Learning transferable visual models from natural lan...

    2025

  43. [52]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML. PMLR, 8748–8763

    2021

  44. [53]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR. 5238–5248

    2022

  45. [54]

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al

  46. [55]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hocken- maier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV. 2641–2649

    2015

  47. [56]

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. 2024. CLIPSelf: Vision Transformer Distills Itself for Open- Vocabulary Dense Prediction. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum...

    2024

  48. [57]

    Yao Wu, Mingwei Xing, Yachao Zhang, Yuan Xie, and Yanyun Qu. 2024. Clip2uda: Making frozen clip reward unsupervised domain adaptation in 3d semantic segmentation. In ACM MM. 8662–8671

    2024

  49. [58]

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba

  50. [59]

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. 2023. Alip: Adaptive language-image pre-training with synthetic caption. In ICCV. 2922–2931

    2023

  51. [60]

    arXiv preprint arXiv:2208.10442 (2022)

    Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208.10442 (2022)

    2022 arXiv

  52. [61]

    Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al. 2023. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In ICCV. 21970–21980

    2023

  53. [62]

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. 2023. Convolutions die hard: Open-vocabulary segmentation with single frozen convo- lutional clip. NIPS 36 (2023), 32215–32234

    2023

  54. [63]

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936 (2022)

    2022 arXiv

  55. [64]

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. 2022. Open-vocabulary detr with conditional matching. In ECCV. Springer, 106–122

    2022

  56. [66]

    Yan Zeng, Xinsong Zhang, and Hang Li. 2021. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276 (2021)

    2021 arXiv

  57. [67]

    Kaicheng Yang, Tiancheng Gu, Xiang An, Haiqiang Jiang, Xiangzi Dai, Ziyong Feng, Weidong Cai, and Jiankang Deng. 2024. Clip-cid: Efficient clip distillation via cluster-instance discrimination. AAAI (2024)

    2024

  58. [68]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiao- dan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 (2021)

    2021 arXiv

  59. [69]

    Lu Zhang, Ke Yan, and Shouhong Ding. 2024. AlignCLIP: Align Multi Domains of Texts Input for CLIP models with Object-IoU Loss. In ACM MM. 1092–1100

    2024

  60. [70]

    exactly one flower

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023). Decoupled Global-Local Alignment for Improving Compositional Understandi...

    2023 arXiv

  61. [74]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sig- moid loss for language image pre-training. In ICCV. 11975–11986

    2023

  62. [75]

    Le Zhang, Rabiul Awal, and Aishwarya Agrawal. 2024. Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Composi- tional Understanding. In CVPR. 13774–13784

    2024

  63. [2010]

    Sun database: Large-scale scene recognition from abbey to zoo. In CVPR. IEEE, 3485–3492

  64. [2013]

    arXiv preprint arXiv:1306.5151 (2013)

    Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)

    2013 arXiv

  65. [2020]

    arXiv preprint arXiv:2005.05909 (2020)

    Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909 (2020)

    2020 arXiv

  66. [2022]

    NIPS 35 (2022), 23716–23736

    Flamingo: a visual language model for few-shot learning. NIPS 35 (2022), 23716–23736

    2022

  67. [2023]

    Scaling language-image pre-training via masking. In CVPR. 23390–23400

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.