Pith. sign in

REVIEW 3 major objections 6 minor 60 references

Language-Image Alignment with Fixed Text Encoders

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen 7B text encoder can replace jointly trained CLIP text towers for compositional alignment.

desk verdict A useful empirical study of frozen LLM text encoders for image alignment, but the headline claim overreaches because the CLIP baseline has a 63M text tower while LIFT uses a 7B one, so freezing and capacity are never separated. read the letter →

arxiv 2506.04209 v1 pith:J7K2I5GK submitted 2025-06-04 cs.CV

Jingfeng Yang , Ziyang Wu , Yue Zhao , Yi Ma This is my paper
classification cs.CV
keywords language-imagealignmentcontrastivelearningfrozentextencoderLLMembeddingscompositionalunderstandinglongcaptionsinverseeffectvisualrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper questions the standard assumption in vision-language pretraining that text and image encoders must be trained jointly from scratch. It proposes LIFT, which freezes a large language model (LLM) used as a text encoder and trains only the image encoder to align with precomputed text embeddings. The central claim is that this simplified framework outperforms vanilla CLIP on compositional understanding tasks and on long synthetic captions, while reducing training cost. If true, it suggests that joint contrastive training is not necessary for strong language-image alignment, and that the semantic richness of LLM embeddings can be transferred to visual representations.

What carries the argument

The central mechanism is the offline pre-computation of text embeddings: captions are embedded once through a frozen LLM-based text encoder (NV-Embed-V2, which uses a latent attention layer and two-stage contrastive instruction tuning), and only the image encoder and a 2-layer MLP projection head are optimized. The alignment objective is the standard CLIP contrastive loss, later replaced in an ablation by a simple cosine similarity loss on positive pairs only. Because the text encoder is frozen, training FLOPs and memory for text drop to O(1) amortized cost per sample, making long-caption training computationally feasible despite the 7B-parameter text tower.

What would settle it

Train a CLIP-style model with a 7B text encoder jointly with a ViT-B/16 image encoder on the same 1.28B DataComp-1B samples and compare it against LIFT under matched compute and training budget: if the jointly trained large-text CLIP matches or exceeds LIFT's compositional and long-caption scores, the claim that joint training is unnecessary collapses.

Watch

Extended reading notes

Core claim

The paper claims that a frozen LLM-based text encoder (specifically NV-Embed-V2, a 7B-parameter embedding model) provides text embeddings good enough to guide visual representation learning. Training only a ViT image encoder and a small projection head against these fixed embeddings yields image representations that beat a jointly trained vanilla CLIP baseline on seven SugarCrepe compositional tasks by an average of 6.8% (short captions) and 7.9% (long captions), and on zero-shot retrieval and classification by 11.0% average when trained on long captions. The authors attribute the gains to the LLM's superior encoding of compositional information and its robustness to the homogeneous syntax of synthetic captions, an effect they call the inverse effect. They also show that contrastive fine-tuning of the text encoder is necessary, while a simple cosine similarity loss can replace the contrastive loss for many tasks with only retrieval performance dropping.

Load-bearing premise

The paper assumes that LIFT's advantage over CLIP comes from using a frozen LLM text encoder and its alignment framework, rather than from the much larger text encoder that was never trained jointly with the image encoder.

Editorial extensions

If this is right

  • If correct, large-scale vision-language pretraining no longer needs to spend compute on training a text encoder from scratch, cutting FLOPs by roughly 25–36% and memory by 7–13% in the configurations tested.
  • The approach sidesteps CLIP's known weakness on compositional understanding, suggesting that the LLM's autoregressive training better encodes word order, spatial relations, and attribute bindings than contrastively trained text encoders.
  • Training on long synthetic captions, which currently degrade CLIP due to the inverse effect, becomes a strength for LIFT, yielding gains of 11.0% on zero-shot retrieval and classification.
  • The simple cosine similarity loss, which removes the need for negative pairs and large batches, is sufficient for compositional tasks and downstream LMM benchmarks, though it sacrifices zero-shot retrieval discriminability.
  • The findings open a design space where the text encoder is chosen for its semantic quality independently of the image encoder, and the two are never trained jointly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's comparison does not isolate the effect of freezing from the effect of text-encoder capacity: NV-Embed-V2 has 7B parameters while the OpenCLIP text tower has 63M, so some of LIFT's gains may stem from scale rather than from the frozen-training paradigm itself.
  • A testable extension would be to jointly train a CLIP variant with a 7B text encoder under comparable compute; if it matches LIFT, the claim that joint training is unnecessary would need revision, whereas if it underperforms, it would support the frozen-framework hypothesis.
  • The offline-embedding design hints at a practical workflow where a single powerful text model serves as the fixed semantic substrate for many image encoders, enabling rapid iteration on the vision side without repeated text encoding.
  • The paper's limitation on swap-object/swap-attribute tasks suggests that further gains may require alignment objectives beyond contrastive or cosine losses, possibly ones that enforce higher-order statistical matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LIFT, a language-image alignment framework that keeps a pre-trained LLM-based text encoder (NV-Embed-V2, 7B) frozen and trains only the image encoder and a projection head, using either a contrastive loss or a simple cosine-similarity loss. The authors compare LIFT against an OpenCLIP ViT-B/16 baseline trained on DataComp-1B and Recap-DataComp-1B captions, reporting evaluations on SugarCrepe compositional tasks, LLaVA downstream tasks, zero-shot ImageNet classification, and COCO/Flickr retrieval, along with ablations over text encoders and losses. The central claim is that frozen LLM text embeddings can guide visual learning as well as or better than jointly trained CLIP-style text towers, especially for compositional understanding and long synthetic captions, while reducing per-step training cost.

Significance. If the central claim holds, the paper offers a practically valuable simplification of language-image pretraining: it decouples text-side semantic quality from joint contrastive training, reduces per-step training FLOPs and memory, and improves compositional and long-caption performance. The paper's strengths include its broad empirical coverage, consistent experimental settings across LIFT and the baseline, the explicit ablation isolating contrastive fine-tuning of the text encoder, the loss-function comparison, and the stated plan to release code and checkpoints. However, the current evidence does not separate text-encoder capacity from the freezing decision, and the efficiency accounting omits the offline text-embedding pass, so the two headline claims are not yet fully established.

major comments (3)
  1. [Sec. 4.2, Tables 1-4] The comparison confounds text-encoder capacity with freezing. LIFT uses NV-Embed-V2 (7B, frozen), while the OpenCLIP baseline's text tower has 63M parameters and is trained from scratch. The paper itself attributes part of the 11.0% long-caption zero-shot gap to "the expressive power of text encoders" (Sec. 4.2), which is a capacity effect, not a freezing effect. As presented, the experiments do not establish that freezing rather than capacity is responsible for LIFT's gains, nor that joint training is unnecessary. Please add a matched-capacity control, for example a 7B text encoder trained jointly with the image encoder on the same data, or a within-architecture comparison of frozen versus trainable text encoder, and discuss how the results change.
  2. [Sec. 3, Figs. 3-4, Sec. 4] The efficiency claim omits the offline text-embedding cost. The paper reports per-sample training FLOPs and memory and notes that eight H800 GPUs can embed 100M captions per day, so the offline pass over 1.28B captions takes roughly 13 days, comparable to the reported 12-day training run. If captions are reused over multiple epochs or across multiple training runs, the cost amortizes, but the paper does not state the number of epochs or the total compute including the offline pass. Please report total compute including offline encoding, state the amortization assumptions, and clarify the sense in which LIFT achieves O(1) amortized complexity.
  3. [Sec. 4.1, Tables 1-4] The headline claim "LIFT outperforms CLIP" is tested against a single vanilla OpenCLIP ViT-B/16 baseline with one hyperparameter set. The experiments do not include other CLIP-style baselines (e.g., SigLIP, a larger text tower, or OpenCLIP with tuned hyperparameters) or multiple seeds with standard deviations. Because the comparison is central to the paper's conclusions, please broaden the baseline set or explicitly restrict the claim to "outperforms the OpenCLIP ViT-B/16 baseline used here," and report variance where feasible.
minor comments (6)
  1. [Introduction] The sentence "we prove that large language models already provide good enough text embeddings" is stronger than the evidence supports; suggest "demonstrate" or "provide evidence that."
  2. [Sec. 4.2] In the sentence "the CLIP's text encoder in our implementation has 63M parameters," the article "the" before "CLIP's" should be removed.
  3. [Tables 1-4] The header "Sample Seen" is unclear; it appears to combine dataset name and sample count. Please rename to something like "Data / Samples."
  4. [Sec. A.4, Figs. 3-4] The FLOPs calculation uses the global max caption token length as a proxy for the average per-batch max caption token length; please state explicitly that the reported numbers are approximations and indicate the expected deviation.
  5. [Sec. 4.4, Table 6] The DataComp 426M row corresponds to a training scale not described in the experimental settings; please specify the dataset split and sample source for this ablation.
  6. [Appendix A.1] The limitation that CLIP variants may exhibit more favorable scaling behavior and that selectively unfreezing last layers of the LLM can help is important; consider moving this caveat to the main text or conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LIFT's comparisons are externally evaluated and the fixed text encoder is an independent tool, not a fitted target.

full rationale

The paper's derivation chain is empirical and self-contained. LIFT freezes an externally pretrained LLM text encoder (NV-Embed-V2) and trains only the image encoder and projection head on image-caption pairs; the evaluation metrics (SugarCrepe, MMBench, ImageNet, COCO/Flickr retrieval) are never used to set training constants or to select the text encoder, so no 'prediction' reduces by construction to a fitted input. The headline comparisons against OpenCLIP do rely on a capacity confound (63M-parameter CLIP text tower vs 7B frozen NV-Embed-V2), and Sec. 4.2 explicitly attributes part of the 11.0% long-caption gap to 'the expressive power of text encoders' rather than to freezing; however, this is an internal-validity and fairness concern, not a circularity: the central claim would still be a falsifiable empirical statement under a matched-capacity control. There is no self-definitional equation, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain: the only coauthored citation [50] is used for background on CLIP-based LMM limitations, not to justify LIFT's results. The paper's own limitation section (A.1) candidly notes that CLIP may scale better and that unfreezing LLM layers could improve scalability, further confirming that the authors do not rely on a self-citation to force their conclusion. The amortized O(1) text-embedding cost claim is an analytic consequence of precomputation, not a circular definition of success. Overall, the study is a straightforward empirical comparison with a genuine but non-circular confound.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the evaluation benchmarks. The design choices (batch size, learning rate, weight decay, projection dimension) are standard training constants shared between LIFT and the CLIP baseline. The paper introduces no new entities or theoretical constructs; it relies on prior models (NV-Embed-V2, ViT) as external tools. The main assumptions are domain-level: the chosen text embedding space, the datasets, the baseline validity, and the benchmarks' ability to measure compositional understanding.

assumptions (4)
  • domain assumption The semantic space of NV-Embed-V2 text embeddings is a good target space for aligning images with natural language.
    The entire LIFT method assumes the precomputed LLM embeddings preserve the semantic distinctions needed for image-text matching, without validating the space against alternative target spaces; Sec 3.
  • domain assumption Training on DataComp-1B and Recap-DataComp-1B approximates the web-scale distribution relevant for CLIP-style pretraining.
    The empirical conclusions are drawn from two specific datasets; Sec 4 and A.6 assume they are representative.
  • domain assumption The authors' OpenCLIP baseline, trained with the same hyperparameters, is a valid 'CLIP' comparison for the paper's claims.
    The paper generalizes to 'CLIP' from one vanilla ViT-B/16 implementation; Sec 4.1.
  • domain assumption SugarCrepe and LLaVA benchmarks accurately measure compositional understanding and downstream usefulness.
    The central improvements are measured on these benchmarks; Sec 4.1 assumes they capture real compositional capability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language-Image Alignment with Fixed Text Encoders." pith.science (2026). https://pith.science/paper/J7K2I5GK

@misc{pith2026250604209,
  author       = {Pith},
  title        = {Pith review of: Language-Image Alignment with Fixed Text Encoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7K2I5GK}},
  note         = {Machine review of arXiv:2506.04209}
}
read the original abstract

Currently, the most dominant approach to establishing language-image alignment is to pre-train text and image encoders jointly through contrastive learning, such as CLIP and its variants. In this work, we question whether such a costly joint training is necessary. In particular, we investigate if a pre-trained fixed large language model (LLM) offers a good enough text encoder to guide visual representation learning. That is, we propose to learn Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by training only the image encoder. Somewhat surprisingly, through comprehensive benchmarking and ablation studies, we find that this much simplified framework LIFT is highly effective and it outperforms CLIP in most scenarios that involve compositional understanding and long captions, while achieving considerable gains in computational efficiency. Our work takes a first step towards systematically exploring how text embeddings from LLMs can guide visual learning and suggests an alternative design choice for learning language-aligned visual representations.

Figures

Figures reproduced from arXiv: 2506.04209 by the authors.

Figure 1
Figure 1. The qualitative comparisons between LIFT and CLIP [43]. The first line shows the caption or option selected by LIFT, and the second line shows the one selected by CLIP. In every case, LIFT selects the correct one, while CLIP does not. We observe that LIFT compensates for CLIP’s shortcomings in tasks involving compositional information (e.g., spatial locations, object-attribute associations, object-object relations).… view at source ↗
Figure 2
Figure 2. The pipeline of LIFT, which adopts a dual-encoder architecture similar to CLIP [43]. LIFT uses an LLM-based text encoder f text to pre-compute the embedding z T for each text sample T offline. During training, we solely update the image encoder f img θ and the projection head f head ϕ to align image embeddings with the pre-computed text embeddings by optimizing an alignment objective. 2 Related Work 2.1 Language-Ima… view at source ↗
Figure 3
Figure 3. The estimated training FLOPs per text￾image sample for CLIP [43] and LIFT trained with average per-batch max caption length 77 and 128. CLIP(77) CLIP(128) LIFT (77,128) 10 20 30 40 50 14 15 13 20 22 19 36 37 33 Memory Usage (GB) ViT-S ViT-B ViT-L [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The original captions (top) and their negative counterparts (bottom) from two SugarCrepe [16] tasks: replace relation (left) and swap attribute (right). calculated between a batch of normalized z T and z I as: Lcontrastive = − 1 2B X B i=1 " log exp(z T i · z I i /τ ) …
Figure 6
Figure 6. Figure 6: The examples of syntactically similar but semantically different caption pairs from Recap-DataComp￾1B [25]. The synthetic captions follow the template “A {Adj.} {Noun} with {N. Phrase} {Verb} {Location}”. CLIP’s [43] text encoder often assigns higher scores to caption …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 45 canonical work pages

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  2. [2]

    R. M. amd Ye Liu, S. R. Joty, C. Xiong, Y . Zhou, and S. Yavuz. Sfr-embedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024

    work page 2024

  3. [3]

    BehnamGhader, V

    P. BehnamGhader, V . Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy. Llm2vec: Large language models are secretly powerful text encoders, 2024

    work page 2024

  4. [4]

    Chen and K

    X. Chen and K. He. Exploring simple siamese representation learning, 2020

    work page 2020

  5. [5]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y . Qiao, and J. Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    2024

  6. [6]

    Cherti, R

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page 2818–2829. IEEE, June 2023

    work page 2023

  7. [7]

    Chiang, Z

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  8. [8]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    work page 2009

Show all 60 references
  1. [9]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    2019

  2. [10]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    2021

  3. [11]

    C. Fu, P. Chen, Y . Shen, Y . Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y . Wu, and R. Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  4. [12]

    S. Y . Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, E. Orgad, R. Entezari, G. Daras, S. Pratt, V . Ramanujan, Y . Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, S. S...

    2023

  5. [13]

    Geirhos, J.-H

    R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, Nov. 2020

    2020

  6. [14]

    Grill, F

    J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020

    2020

  7. [15]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifr...

    2022

  8. [16]

    Hsieh, J

    C.-Y . Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. InThirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  9. [17]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  10. [18]

    Huang, Q

    Z. Huang, Q. Ye, B. Kang, J. Feng, and H. Fan. Classification done right for vision-language pre-training, 2024

    2024

  11. [19]

    Jaegle, F

    A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention, 2021

    2021

  12. [20]

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. V . Le, Y . Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. 10

    2021

  13. [21]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023

    2023

  14. [22]

    J. Kim, S. Lee, J. Kwon, S. Gu, Y . Kim, M. Cho, J. yong Sohn, and C. Choi. Linq-embed-mistral:elevating text retrieval with improved gpt data through task-specific control and quality refinement. Linq AI Research Blog, 2024

    2024

  15. [23]

    C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping. Nv-embed: Improved techniques for training llms as generalist embedding models, 2025

    2025

  16. [24]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  17. [25]

    X. Li, H. Tu, M. Hui, Z. Wang, B. Zhao, J. Xiao, S. Ren, J. Mei, Q. Liu, H. Zheng, Y . Zhou, and C. Xie. What if we recaption billions of web images with llama-3?, 2024

    2024

  18. [26]

    X. Li, Z. Wang, and C. Xie. An inverse scaling law for clip training, 2023

    2023

  19. [27]

    Y . Li, Y . Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models, 2023

    2023

  20. [28]

    T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015

    2015

  21. [29]

    H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning, 2024

    2024

  22. [30]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning, 2023

    2023

  23. [31]

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin. Mmbench: Is your multi-modal model an all-around player?, 2024

    2024

  24. [32]

    Y . Liu, X. Li, Z. Wang, B. Zhao, and C. Xie. Clips: An enhanced clip framework for learning with synthetic captions, 2024

    2024

  25. [33]

    Y . Liu, K. Wang, W. Shao, P. Luo, Y . Qiao, M. Z. Shou, K. Zhang, and Y . You. Mllms-augmented visual-language representation learning, 2024

    2024

  26. [34]

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    2022

  27. [35]

    Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna. Crepe: Can vision-language foundation models reason compositionally?, 2023

    2023

  28. [36]

    Maninis, K

    K.-K. Maninis, K. Chen, S. Ghosh, A. Karpur, K. Chen, Y . Xia, B. Cao, D. Salz, G. Han, J. Dlabal, D. Gnanapragasam, M. Seyedhosseini, H. Zhou, and A. Araujo. Tips: Text-image pretraining with spatial awareness, 2025

    2025

  29. [37]

    Mehta, M

    S. Mehta, M. Horton, F. Faghri, M. H. Sekhavat, M. Najibi, M. Farajtabar, O. Tuzel, and M. Rastegari. Catlip: Clip-level visual recognition accuracy with 2.7x faster pre-training on web-scale image-text data, 2024

    2024

  30. [38]

    J. X. Morris, W. Zhao, J. T. Chiu, V . Shmatikov, and A. M. Rush. Language model inversion, 2023

    2023

  31. [39]

    N. Mu, A. Kirillov, D. Wagner, and S. Xie. Slip: Self-supervision meets language-image pre-training, 2021

    2021

  32. [40]

    Muennighoff, N

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. Mteb: Massive text embedding benchmark, 2023

    2023

  33. [41]

    M. F. Naeem, Y . Xian, X. Zhai, L. Hoyer, L. V . Gool, and F. Tombari. Silc: Improving vision language pretraining with self-distillation, 2023

    2023

  34. [42]

    Neelakantan, T

    A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy, J. Heidecke, P. Shyam, B. Power, T. E. Nekoul, G. Sastry, G. Krueger, D. Schnurr, F. P. Such, K. Hsu, M. Thompson, T. Khan, T. Sherbakov, J. Jang, P. Welinder, and L. We...

    2022

  35. [43]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  36. [44]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023

    2023

  37. [45]

    Robinson, L

    J. Robinson, L. Sun, K. Yu, K. Batmanghelich, S. Jegelka, and S. Sra. Can contrastive learning avoid shortcut solutions?, 2021

    2021

  38. [46]

    Singh, V

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read, 2019

    2019

  39. [47]

    Stone, H

    A. Stone, H. Soltau, R. Geirhos, X. Yi, Y . Xia, B. Cao, K. Chen, A. Ogale, and J. Shlens. Learning visual composition through improved semantic guidance, 2025

    2025

  40. [48]

    Thrush, R

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality, 2022

    2022

  41. [49]

    S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, Z. Wang, R. Fergus, Y . LeCun, and S. Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    2024

  42. [50]

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    2024

  43. [51]

    Tschannen, A

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, an...

    2025

  44. [52]

    van den Oord, Y

    A. van den Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding, 2019

    2019

  45. [53]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023

    2023

  46. [54]

    A. J. Wang, K. Q. Lin, D. J. Zhang, S. W. Lei, and M. Z. Shou. Too large; data reduction for vision-language pre-training, 2023

    2023

  47. [55]

    L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. Improving text embeddings with large language models, 2024

    2024

  48. [56]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  49. [57]

    Young, A

    P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the Association for Computational Linguistics, 2:67–78, 2014

    2014

  50. [58]

    Yuksekgonul, F

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision-language models be- have like bags-of-words, and what to do about it? InInternational Conference on Learning Representations, 2023

    2023

  51. [59]

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  52. [60]

    Pub" hangs above several liquor bottles A sign with the word

    K. Zheng, Y . Zhang, W. Wu, F. Lu, S. Ma, X. Jin, W. Chen, and Y . Shen. Dreamlip: Language-image pre-training with long captions, 2024. 12 A Appendix A.1 Limitations As discussed in the experiments section, LIFT ’s ability to capture compositional information is not yet compl...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.