Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models

Ruibin Li; Tong Jia; Wei Zheng; Xin Huang; Ya Wang

REVIEW 4 major objections 4 minor 26 references

Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Text deviations become image negatives to sharpen VLM compositional reasoning

desk verdict A new idea for synthesizing visual hard negatives via text-embedding displacement, but the core assumption is unvalidated and the experiments lack key controls—worth review, not yet trustworthy. read the letter →

arxiv 2505.15576 v2 pith:4EGGB7AZ submitted 2025-05-21 cs.CV cs.LG

Xin Huang , Ruibin Li , Tong Jia , Wei Zheng , Ya Wang This is my paper · ORCID

classification cs.CVcs.LG

keywords compositionalreasoningvision-languagemodelshardnegativeminingcontrastivelearningvisualperturbationCLIPfine-tuningadaptivemarginlossVALSEbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that compositionality failures in vision-language models can be reduced by translating hard-negative captions into the visual embedding domain. The proposed method, AHNPL, takes the difference between a positive caption embedding and a hard-negative caption embedding, adds that difference to the image embedding, and uses the resulting vector as a hard negative image during fine-tuning. Two auxiliary losses then separate hard negatives within each modality and adaptively increase the margin for difficult positive and negative pairs. On the VALSE benchmark the method reaches 75.9% average accuracy, surpassing the previous hard-negative method CE-CLIP by 3.4 points, with gains also on ARO and SugarCrepe. If the gains hold, it offers a recipe for improving compositional reasoning without collecting new images.

What carries the argument

The load-bearing identity is the linear embedding transfer $e_{I_{neg}} = e_{I_{orig}} + (e_{T_{neg}} - e_{T_{orig}})$, which the paper calls visual perturbation with semantic shift. It converts a text-side hard negative into a synthetic image-side hard negative by reusing the CLIP text embedding difference as a displacement in image embedding space. The dynamic margin threshold $M_t^n = \frac{1}{|B|}\sum_{(I,T)\in B} (S^{t-1}(I,T) - S^{t-1}(I,T_n))$ is the adaptive component that changes contrastive pressure based on sample difficulty.

What would settle it

Compute cosine similarity between the generated negative image embedding and the hard negative text before and after training; if the perturbation does not shift the image embedding toward the negative text semantics, or if replacing the deviation embedding with a random unit vector reproduces the reported gains, the visual perturbation mechanism is not the source of the improvement.

Watch

Extended reading notes

Core claim

AHNPL's central claim is that the image encoder, not just the text encoder, needs hard negatives during fine-tuning, and that such image negatives can be synthesized directly in CLIP embedding space. The deviation embedding $\Delta e = e_{T_{neg}} - e_{T_{orig}}$ is added to the original image embedding to form $e_{I_{neg}} = e_{I_{orig}} + \Delta e$, giving the image encoder a training signal that mirrors the semantic shift already present in text. On top of this, a multimodal hard negative loss pushes generated visual negatives and textual negatives away from positives, and a dynamic margin loss adjusts the margin using the previous step's similarity gap, focusing training on the hardest pairs. The paper reports state-of-the-art results on VALSE and improvements over CE-CLIP on ARO and SugarCrepe, attributing the gains to better visual discrimination and difficulty-aware alignment.

Load-bearing premise

The argument depends on the assumption that adding the text-to-text semantic difference to an image embedding produces a legitimate hard-negative image embedding, rather than a meaningless point in embedding space.

Editorial extensions

If this is right

On VALSE, the method's 75.9% average surpasses CE-CLIP's 72.5%, with the largest margins on the quantifiers and clean subtasks.
On ARO, relation and attribute accuracy improve over CE-CLIP (83.8 vs 83.0 and 77.0 vs 76.4), showing better relationship and attribute understanding.
On SugarCrepe, object-level swap and add tasks rise by 3.5 and 4.8 points over CE-CLIP, indicating reduced language bias in favor of visual cues.
Ablations show that each component—hard negatives, multimodal hard negative loss, and dynamic margin loss—adds a positive increment, and the full combination gives the best results.
The method needs only existing captions to synthesize visual negatives, so it can fine-tune an image encoder without additional image collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Eq. (2) assumes linear arithmetic in CLIP space, the same trick could transfer to other joint embedding models only if their geometry is similarly linear; the paper does not test this.
The synthetic image negatives are never decoded or validated as real images, so a natural extension is to check whether they correspond to semantically altered images, for example by retrieval or generation.
The dynamic margin uses the previous training step's similarity gap; with small batches or high learning rates, this lag could cause oscillation. A sensitivity analysis would clarify the operating range.
The negative-generation step is limited to POS-based swaps and RoBERTa infilling; pairing AHNPL with LLM-generated negatives, as in SugarCrepe, could broaden hard-negative diversity and likely push the method further.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper has a genuinely new mechanism—synthesizing image-side hard negatives by adding a text-embedding difference to an image embedding—and the combined loss recipe is not in the cited literature. The gains on three benchmarks are consistent, and the ablations do show each loss contributes. But the central assumption (Eq. 2) is unvalidated, the ablation never isolates the visual perturbation, and there are no error bars or multi-seed runs. As it stands, I can't tell whether the improvement comes from the mechanism claimed or from the text-side losses and regularization.

What's new: existing methods use text-side negatives or cross-modal ranking; this paper transfers a text shift into image space and trains a visual negative loss. The dynamic margin that adjusts based on the model's own previous-step similarity is also a reasonable addition. The method is simple to implement and the paper is clearly written.

The soft spots: The stress-test note is on target. Eq. (2) assumes text-embedding differences are valid displacements for image embeddings in CLIP space. The paper doesn't show the resulting vectors decode to anything image-like, nor that they are closer to real images matching the negative text. Table 5 reports cosine distances that are guaranteed to move in the expected direction by construction, so it's not evidence. The ablation in Table 3 has a 'negatives' row that includes both visual and textual negatives, so removing the visual perturbation alone is never tested. Without that ablation, the contribution of the paper's claimed novelty is unmeasured.

Also missing: error bars, multiple seeds, and a clear statement about the MSCOCO training/evaluation overlap with SugarCrepe, which shares COCO images. The CREPE citation is wrong—the referenced paper is about question answering, not the compositional benchmark.

The paper is not incoherent; the mechanism might work. But the evidence is not there yet. I'd send it to a serious referee, but the author's response would need to add a validation study of the synthetic embeddings and an ablation that isolates the visual perturbation. For a reader, treat the VALSE improvement as promising but unproven.

Referee Report

4 major / 4 minor

Summary. The paper proposes AHNPL, a fine-tuning method for CLIP on compositional reasoning (CR) benchmarks. It generates textual hard negatives via POS-based noun swapping and RoBERTa-based masked infilling, then creates synthetic visual hard negatives by adding the text-embedding difference to the original image embedding (Eq. 2). Training adds two auxiliary losses to the standard contrastive loss: a multimodal hard negative loss (Eqs. 5-7) and a dynamic margin loss (Eqs. 8-11). The authors report state-of-the-art average accuracy on VALSE (75.9% vs. 72.5% for CE-CLIP), with gains also on ARO and SugarCrepe (Tables 1-2). Ablations (Table 3) and a small embedding-distance visualization (Table 5) are provided.

Significance. If the reported gains are robust, AHNPL offers a simple and reproducible recipe for improving compositional reasoning in CLIP-style models, with code release and evaluation on standard benchmarks being clear strengths. The central novelty--transferring text-derived semantic shifts into the image embedding space--is conceptually attractive but is also the main risk: the paper does not demonstrate that the synthetic vectors produced by Eq. (2) behave like genuine image hard negatives. The empirical claims are currently supported by single-run results without error bars, and the ablation table does not isolate the visual perturbation mechanism. These issues are addressable and do not, in my view, invalidate the approach, but they need to be resolved before the paper's central claim can be accepted.

major comments (4)

[Section 3.2, Eq. (2)] The load-bearing assumption is that e_Ineg = e_Iorig + (e_Tneg - e_Torig) yields an embedding that behaves like a real hard-negative image. The paper does not validate that this vector lies on or near the image-embedding manifold, nor that it is more similar to the negative text than to the positive text in a way matching an actual image. Table 5 only reports cosine distances among computed embeddings, and because e_Ineg is defined as a sum, those distances move as expected by construction. If Eq. (2) does not hold, L_visual_neg in Eq. (5) trains the image encoder to repel arbitrary directions that may not correspond to any real image, so the reported gains could come from regularization or the text-side losses rather than the claimed visual-perturbation mechanism. Please provide evidence such as decoding nearest images, measuring retrieval consistency with real hard-negative images, or showing that the synthetic embeddings are indistinguishable from genuine image negatives in downstream behavior.
[Table 3, ablation study] The ablation does not isolate the visual perturbation of Eq. (2). The row labeled "negatives" includes the combined L_neg of Eq. (7), which already contains L_visual_neg; there is no condition with only textual hard negatives and no visual perturbation, and no condition with the visual negative loss removed while keeping the textual hard negatives. As a result, the contribution of the visual perturbation to the final result is untested. Please add ablations that factor out Eq. (5) separately from Eq. (6), and also report the effect of replacing the synthetic visual negatives with actual images drawn as hard negatives.
[Tables 1-3, Section 4.3] All results appear to be from a single training run, with no error bars, no multiple-seed reporting, and no significance tests. The headline VALSE gain over CE-CLIP is 3.4 percentage points, and several subtask differences are much smaller than that; without variance information the claim of a new state of the art is not statistically supported. Please report mean and standard deviation over at least three seeds, and ideally paired significance tests for the main benchmark comparisons.
[Section 3.3, Eqs. (5)-(6)] The loss definitions in Eqs. (5) and (6) are written with an ambiguous fraction: the displayed "-log(1 / sum exp(...))" is algebraically equivalent to log(sum exp(...)), which is a soft-plus-like penalty that pushes similarities down, but the notation should be clarified. More importantly, the losses use text-text and image-image similarities (S(I, In), S(T, Tn)) that are not defined by Eq. (3), which only defines text-image similarity; please specify the exact similarity computation used for within-modality pairs and whether a temperature is applied.

minor comments (4)

[Table 1] The VALSE column headers are garbled in the typeset table (e.g., "A vgExistence", "Av g", "Foil-it! A vgquantifiers"), making it hard to read which sub-benchmark corresponds to which score. Please reformat the table with clear column separators.
[Figure 2] The "+++" symbol in the visual-perturbation panel is unexplained; it appears to denote vector addition in embedding space, but a one-line caption definition would help.
[Section 4.6, Table 5] Table 5 presents "cosine distances" but the numbers shown are sparse and it is not clear which pairs they correspond to, nor whether higher or lower values are intended as evidence. Please label each distance and state explicitly which comparison supports the claim.
[Section 3.1] The sentence "The first type involves swapping two nouns in a sentence to generate hard negatives" is followed by a sentence fragment describing the purpose; please merge these into a complete paragraph.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the headline benchmark gains rest on external held-out evaluations; only Table 5's mechanism illustration reduces to the algebraic definition of eIneg.

self definitional [Section 3.2, Eq. (2); Section 4.6, Table 5]
"Next, we directly incorporate the generated deviation embedding ∆e into the image embedding space by adding it to the original image’s embedding vector eIorig to create the embedding representation eIneg for the image negative sample: eIneg = eIorig + ∆e (2) ... The changes in cosine distances demonstrate that the visual perturbation strategy effectively pushes the original image features towards negative texts, improving feature distinguishability."

The visualization in Table 5 is offered as evidence that visual perturbation moves image features toward negative texts. But eIneg is defined as eIorig + (eTneg - eTorig), so the cosine shift of eIneg toward eTneg and away from eTorig follows directly from the definition itself. It is not an independent empirical check that the summed vector lies on the image-embedding manifold or behaves like a real hard-negative image. This makes the illustrative validation tautological rather than evidence of semantic validity. It is not the basis of the headline results: VALSE, ARO, and SugarCrepe are held-out benchmarks, and the training losses on MSCOCO do not encode their labels, so the central performance claim remains externally testable.

full rationale

The paper's central claim is evaluated on external benchmarks (ARO, VALSE, SugarCrepe) after fine-tuning on MSCOCO, and none of the training losses directly encode the evaluation labels. The visual perturbation eIneg = eIorig + (eTneg - eTorig) is an explicit construction, not a parameter fitted to the test sets. The dynamic margin threshold M_t_n and learnable margin a are adaptive training mechanisms computed from the model's own previous-step similarities; they are self-referential in an algorithmic sense but do not smuggle in the target result. The one genuinely definitional element is the Table 5 'demonstration' that the constructed eIneg moves toward the negative text, which follows by construction from Eq. (2) rather than providing independent validation of the linear-transfer assumption. This is a minor auxiliary circularity, not a load-bearing one; the headline improvements remain empirical and externally falsifiable. No self-citation chain or imported uniqueness theorem carries the argument.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method rests on one strong unexamined assumption about linear transfer of text differences to image embeddings, plus a self-referential margin threshold. The only fitted numeric parameter is the learned margin a; its final value and sensitivity are not reported. Loss weights are fixed at 1 and temperature is inherited from CLIP. No invented entities are introduced.

free parameters (1)

learnable margin a = not reported (initialized from standard normal, lower bound 0.2)
Used in the positive margin loss L+mar (Eq. 8). It is learned during fine-tuning, and the final value, gradient behavior, and sensitivity to its lower bound are not analyzed or reported.

assumptions (3)

domain assumption The CLIP embedding space is sufficiently linear that adding a text-embedding difference to an image embedding produces a valid hard negative image embedding.
Section 3.2, Eq. (2): e_Ineg = e_Iorig + (e_Tneg - e_Torig). No validation that the resulting vector corresponds to an image with the intended semantic change.
domain assumption The previous-step similarity gap S(I,T) - S(I,Tn) is a valid difficulty signal for setting the current margin.
Eq. (10): the adaptive threshold uses the model's own prior outputs. This assumes the self-referential measure tracks true semantic difficulty.
domain assumption Spacy POS parsing and RoBERTa mask-filling generate hard negatives that are semantically close to positives.
Section 3.1: no quality filtering, human evaluation, or quantitative check of the generated textual negatives is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models." pith.science (2026). https://pith.science/paper/4EGGB7AZ

@misc{pith2026250515576,
  author       = {Pith},
  title        = {Pith review of: Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EGGB7AZ}},
  note         = {Machine review of arXiv:2505.15576}
}

read the original abstract

Vision-Language Models (VLMs) are essential for multimodal tasks, especially compositional reasoning (CR) tasks, which require distinguishing fine-grained semantic differences between visual and textual embeddings. However, existing methods primarily fine-tune the model by generating text-based hard negative samples, neglecting the importance of image-based negative samples, which results in insufficient training of the visual encoder and ultimately impacts the overall performance of the model. Moreover, negative samples are typically treated uniformly, without considering their difficulty levels, and the alignment of positive samples is insufficient, which leads to challenges in aligning difficult sample pairs. To address these issues, we propose Adaptive Hard Negative Perturbation Learning (AHNPL). AHNPL translates text-based hard negatives into the visual domain to generate semantically disturbed image-based negatives for training the model, thereby enhancing its overall performance. AHNPL also introduces a contrastive learning approach using a multimodal hard negative loss to improve the model's discrimination of hard negatives within each modality and a dynamic margin loss that adjusts the contrastive margin according to sample difficulty to enhance the distinction of challenging sample pairs. Experiments on three public datasets demonstrate that our method effectively boosts VLMs' performance on complex CR tasks. The source code is available at https://github.com/nynu-BDAI/AHNPL.

Figures

Figures reproduced from arXiv: 2505.15576 by the authors.

**Figure 2.** The proposed Adaptive Hard Negative Perturbation Learning (AHNPL). (a) Performing visual perturbation by computing the se [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** (a) The typical contrastive learning framework. (b) The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

[1]

Distill- ing knowledge from text-to-image generative models im- proves visio-linguistic reasoning in clip

[Basu et al., 2024] Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi. Distill- ing knowledge from text-to-image generative models im- proves visio-linguistic reasoning in clip. InEMNLP, pages 6105–6113,

work page 2024
[6]

Cross-modal common representation learning by hybrid transfer network

[Huang et al., 2017] Xin Huang, Yuxin Peng, and Mingkuan Yuan. Cross-modal common representation learning by hybrid transfer network. In IJCAI, pages 1893–1900,

work page 2017
[7]

Structure-clip: Towards scene graph knowledge to en- hance multi-modal structured representations

[Huang et al., 2024] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge to en- hance multi-modal structured representations. In AAAI, volume 38, pages 2417–2425,

work page 2024
[8]

Adaptive prompt-based semantic embedding with inspire potential of implicit knowledge for cross-modal retrieval

[Huang et al., 2025] Xin Huang, Shilong Wang, Tong Jia, Zhihang Gou, and Jingjing Li. Adaptive prompt-based semantic embedding with inspire potential of implicit knowledge for cross-modal retrieval. In AAAI, volume 39, pages 17485–17493,

work page 2025
[10]

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In ICML, pages 19730–19742,

work page 2023
[11]

Microsoft coco: Com- mon objects in context

[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740–755,

2014
[14]

Image segmentation using text and image prompts

[L¨uddecke and Ecker, 2022] Timo L¨uddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, pages 7086–7096,

work page 2022
[15]

AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models

[Metzen et al., 2023] Jan Hendrik Metzen, Piyapat Saranrit- tichai, and Chaithanya Kumar Mummadi. Autoclip: Auto- tuning zero-shot classifiers for vision-language models. arXiv preprint arXiv:2309.16414,

work page Pith review arXiv 2023

Show all 26 references

[16]

Textattack: A frame- work for adversarial attacks, data augmentation, and ad- versarial training in nlp

[Morris et al., 2020] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A frame- work for adversarial attacks, data augmentation, and ad- versarial training in nlp. InEMNLP, pages 119–126,

2020
[18]

Valse: A task-independent benchmark for vision and language models centered on linguistic phe- nomena

[Parcalabescu et al., 2022] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. Valse: A task-independent benchmark for vision and language models centered on linguistic phe- nomena. In ACL, pages 8253–8280,

2022
[19]

Learning transferable visual models from nat- ural language supervision

[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748–8763,

2021
[21]

Gomez, Lukasz Kaiser, and Illia Polosukhin

[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998–6008,

2017
[22]

Image as a foreign language: Beit pretrain- ing for vision and vision-language tasks

[Wang et al., 2023] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggar- wal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretrain- ing for vision and vision-language tasks. In CVPR, pages 19175–19186,

2023
[23]

Groupvit: Semantic segmentation emerges from text supervision

[Xu et al., 2022] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pages 18134–18144,

2022
[24]

CREPE: open-domain ques- tion answering with false presuppositions

[Yu et al., 2023] Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. CREPE: open-domain ques- tion answering with false presuppositions. In ACL, pages 10457–10480,

2023
[25]

When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936,

[Yuksekgonul et al., 2022] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936,

2022 arXiv
[26]

Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models

[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592, 2023

2023 arXiv
[2014]

Roberta: A robustly optimized bert pretraining approach

[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692,

2019 arXiv
[2017]

Sugar- crepe: fixing hackable benchmarks for vision-language compositionality

[Hsieh et al., 2023] Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugar- crepe: fixing hackable benchmarks for vision-language compositionality. In NeurIPS, pages 31096–31116,

2023
[2019]

Smith, Yejin Choi, and Hannaneh Ha- jishirzi

[Liu et al., 2023] Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hannaneh Ha- jishirzi. Vera: A general-purpose plausibility estimation model for commonsense statements. In EMNLP, pages 1264–1287,

2023
[2020]

Chils: Zero- shot image classification with hierarchical label sets

[Novack et al., 2023] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero- shot image classification with hierarchical label sets. In ICML, pages 26342–26362,

2023
[2021]

Winoground: Probing vision and language models for visio-linguistic compositionality

[Thrush et al., 2022] Tristan Thrush, Ryan Jiang, Max Bar- tolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, pages 5238–5248,

2022
[2022]

spacy 2: Natural lan- guage understanding with bloom embeddings, convolu- tional neural networks and incremental parsing.To appear,

[Honnibal, 2017] Matthew Honnibal. spacy 2: Natural lan- guage understanding with bloom embeddings, convolu- tional neural networks and incremental parsing.To appear,

2017
[2023]

CyCLIP: Cyclic contrastive language-image pretraining

[Goel et al., 2022] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. CyCLIP: Cyclic contrastive language-image pretraining. In NeurIPS, volume 35, pages 6704–6719,

2022
[2024]

Going beyond nouns with vision & language models using synthetic data

[Cascante-Bonilla et al., 2023] Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, G¨ul Varol, Aude Oliva, Vicente Ordonez, Rog ´erio Feris, and Leonid Karlinsky. Going beyond nouns with vision & language models using syntheti...

2023
[2025]

Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation

[Li et al., 2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, pages 12888–12900,

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Distill- ing knowledge from text-to-image generative models im- proves visio-linguistic reasoning in clip

[Basu et al., 2024] Samyadeep Basu, Shell Xu Hu, Maziar Sanjabi, Daniela Massiceti, and Soheil Feizi. Distill- ing knowledge from text-to-image generative models im- proves visio-linguistic reasoning in clip. InEMNLP, pages 6105–6113,

work page 2024

[2] [6]

Cross-modal common representation learning by hybrid transfer network

[Huang et al., 2017] Xin Huang, Yuxin Peng, and Mingkuan Yuan. Cross-modal common representation learning by hybrid transfer network. In IJCAI, pages 1893–1900,

work page 2017

[3] [7]

Structure-clip: Towards scene graph knowledge to en- hance multi-modal structured representations

[Huang et al., 2024] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge to en- hance multi-modal structured representations. In AAAI, volume 38, pages 2417–2425,

work page 2024

[4] [8]

Adaptive prompt-based semantic embedding with inspire potential of implicit knowledge for cross-modal retrieval

[Huang et al., 2025] Xin Huang, Shilong Wang, Tong Jia, Zhihang Gou, and Jingjing Li. Adaptive prompt-based semantic embedding with inspire potential of implicit knowledge for cross-modal retrieval. In AAAI, volume 39, pages 17485–17493,

work page 2025

[5] [10]

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In ICML, pages 19730–19742,

work page 2023

[6] [11]

Microsoft coco: Com- mon objects in context

[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740–755,

2014

[7] [14]

Image segmentation using text and image prompts

[L¨uddecke and Ecker, 2022] Timo L¨uddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, pages 7086–7096,

work page 2022

[8] [15]

AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models

[Metzen et al., 2023] Jan Hendrik Metzen, Piyapat Saranrit- tichai, and Chaithanya Kumar Mummadi. Autoclip: Auto- tuning zero-shot classifiers for vision-language models. arXiv preprint arXiv:2309.16414,

work page Pith review arXiv 2023