Pith. sign in

REVIEW 3 major objections 6 minor 62 references

Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that the pre-training objective, not model size or data volume, determines how much syntax vision-language text encoders acquire.

desk verdict A worthwhile layerwise syntax-probing study whose core finding—contrastive-only training leaves VLM text encoders with little linearly accessible dependency syntax—survives the obvious confounds, though a nonlinear probe control and released configuration are needed before the causal story is airtight. read the letter →

arxiv 2412.08111 v1 pith:I5TKNXZW submitted 2024-12-11 cs.CV cs.CLcs.LG

Sri Harsha Dumpala , David Arps , Sageev Oore , Laura Kallmeyer , Hassan Sajjad This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords syntaxprobingvision-languagemodelstextencoderscontrastivelearningmaskedlanguagemodelingdependencyparsingUniversalDependencieslayer-wiseanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the text encoders of vision-language models encode far less syntactic structure than the text-only language models they are usually compared with, and that the cause is the pre-training objective rather than architecture, parameter count, or data volume. Using a linear probe that reads Universal Dependency trees out of frozen representations, the authors find that CLIP, trained with a sentence-level contrastive loss, scores dramatically lower than RoBERTa and even loses syntactic information as layers deepen. FLAVA, which adds masked-language-modeling to contrastive loss, nearly matches RoBERTa, indicating that token-level objectives are what make syntax linearly accessible. If true, this isolates a concrete, fixable bottleneck in multimodal pretraining and helps explain why VLM text encoders behave like bag-of-words models.

What carries the argument

DepProbe is a linear probing classifier that decodes labeled, directed dependency trees from frozen word representations: a matrix $L$ labels each word's incoming dependency relation, and a matrix $B$ projects representations into a low-dimensional "syntactic subspace" whose distances are trained to mirror distances in the dependency tree. The probe is trained on the EWT Universal Dependencies treebank and evaluated with labeled attachment score, unlabeled attachment score, and related metrics. The argument is carried by comparing probes across models that vary one factor at a time — pre-training objective (CLIP versus FLAVA versus uni-modal models), parameter count (151 million to 1.2 billion), data volume (400 million to 5 billion pairs), and layer depth — so that the objective is the only factor that tracks the observed score gaps.

What would settle it

Train two text encoders with identical architecture and data, one with contrastive loss only and one with an added masked-language-model objective, and compare DepProbe scores: if the contrastive-only model matches the masked-language-model model's labeled attachment score, the pre-training-objective claim is refuted. Alternatively, re-run the probe with per-model hyperparameter tuning and a nonlinear readout; if CLIP's scores rise to RoBERTa's level, the gap was a linear-separability artifact.

Watch

Extended reading notes

Core claim

The central claim is that uni-modal language model text encoders acquire syntactic knowledge more effectively than vision-language model text encoders, and that this gap is driven by the pre-training objective. On the EWT treebank with DepProbe at layer 6, RoBERTa-base reaches a labeled attachment score of 64.43, FLAVA reaches 60.83, while CLIP-ViT-B/32 reaches only 26.25; scaling CLIP from 151 million to 1.2 billion parameters or from 400 million to 5 billion image-text pairs does not close the gap. CLIP's probe performance degrades from layer to layer, whereas RoBERTa and FLAVA peak at middle layers. Sentence-level models trained on natural language inference behave like CLIP at the final layer, reinforcing the conclusion that global sentence objectives suppress local syntactic structure.

Load-bearing premise

The whole comparison assumes DepProbe's scores measure syntactic knowledge equally across models with different tokenizers, embedding geometries, and training objectives, so part of CLIP's low score could be a measurement artifact rather than missing syntax.

Editorial extensions

If this is right

  • Adding a masked-language-model or other token-level objective to contrastive vision-language pretraining should substantially restore syntactic encoding, close to unimodal levels.
  • Scaling up parameter count or image-text data for contrastive-only models will not, by itself, improve syntax; the paper's CLIP variants show flat or worse scores.
  • Syntactic probing of text encoders can serve as an intrinsic, image-free diagnostic for multimodal pretraining quality.
  • In text-to-image pipelines that use CLIP-style encoders, prompts whose meaning depends on word order, such as "A cat chases a dog," will remain unreliable unless the text encoder is retrained with token-level objectives.
  • Layer-wise, the final layers of contrastive and sentence-level encoders are the least syntactic, so interventions should target the top of the encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: finetune a CLIP text encoder with a masked-language-model objective on its original caption data and check whether DepProbe scores rise; if they do, the objective attribution is causal rather than correlational.
  • The same probe setup could predict text-to-image binding failures: prompts with high dependency distance between subject and object should fail more often, linking the measured score gap to observable image errors.
  • The authors' linear-probe lens may understate CLIP's syntax if the information exists in a nonlinear form; a parser-based or nonlinear evaluation could find more syntax than DepProbe reports, without changing the conclusion that it is not linearly accessible.
  • The similarity between sentence-level language models and CLIP at the final layer suggests that any objective that pools words into a global sentence vector — contrastive or natural-language-inference — tends to strip local dependency structure at the output layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether VLM text encoders encode syntactic structure, using DepProbe to predict Universal Dependency trees from frozen representations of CLIP, FLAVA, RoBERTa, MiniLM, Sentence-RoBERTa, and Sentence-MiniLM. It reports large gaps between CLIP and ULM text encoders at middle layers (e.g., LAS 26.25 vs 64.43 at layer 6), near-ULM performance for FLAVA, no recovery from larger CLIP models or more data, final-layer degradation for sentence-level models, and stable results on the SPUD nonsensical control. The paper concludes that the pre-training objective is the primary determinant of syntactic knowledge in VLM text encoders, with token-level objectives such as MLM being beneficial.

Significance. If the causal attribution holds, the paper provides an intrinsic evaluation that can explain compositionality failures of CLIP and motivates auxiliary token-level objectives. The measurement has strong points: DepProbe is an external probe, public checkpoints are used, the train/test gap in Table 6 rules out probe overfitting, and the SPUD nonce-sentence evaluation in Table 7 separates syntactic from lexical co-occurrence information. The layer-wise tables and per-relation breakdowns are detailed and useful. The causal claim, however, goes beyond the current experimental design, and the probe-fairness hypothesis is a genuine threat to the interpretation of the headline LAS gap.

major comments (3)
  1. [Sec. 4.1, Tables 1 and 7] The central claim that the pre-training objective is the main determinant of syntactic knowledge is not established by the CLIP-versus-FLAVA comparison. FLAVA differs from CLIP not only in having an MLM objective but also in total parameters (358M vs 151M), training data volume and domain (70M image-text pairs vs 400M, per Appendix B), and additional auxiliary objectives (MMM and MIM; Sec. 3.2). The manuscript also does not state whether FLAVA's text encoder was initialized from a pretrained ULM; if it was, as in the original FLAVA release, the near-ULM syntax scores could be inherited from initialization rather than learned through MLM. The claim should therefore be softened to a correlation with the objective, or supported by a controlled comparison such as a CLIP variant trained with an added MLM objective or an ablation of FLAVA's auxiliary losses.
  2. [Sec. 3.1, Sec. 6] DepProbe is a linear probe (Eqs. 1-3) applied after element-wise mean pooling of subword tokens, so it measures linear separability of dependency-tree information, not syntactic information per se. CLIP's different BPE tokenizer, anisotropic representation geometry, and mean pooling can all lower a linear probe's accuracy even if the underlying syntax is nonlinearly encoded. The conclusion in Sec. 6 that CLIP 'fails to capture even basic features of the dependency tree' is therefore stronger than the evidence supports. The one-sentence report in Sec. 3.1 that a hyperparameter search gave 'no significant improvements' does not describe the search space or whether it was per-model. A nonlinear control probe (e.g., a one-hidden-layer MLP probe) or an anisotropy/whitening control is needed before attributing the 26.25-versus-64.43 LAS gap to the pre-training objective; otherwise the wording should be restricted to 'syntax is not linearly readable.'
  3. [Sec. 4.2, Table 2] The text states that LAION-CLIP-ViT-B/32 'does not show any improvement over CLIP-ViT-B/32,' but Table 2 reports layer-6 LAS/UAS/UUAS of 32.12/40.29/45.71 for LAION-CLIP versus 26.25/33.49/38.91 for CLIP-ViT-B/32, which is an improvement of roughly 5-7 points. Because no variance or multiple-seed results are reported, it is impossible to tell whether these differences are meaningful; the text should either report error bars or explicitly treat such differences as within noise, and the sentence should be corrected.
minor comments (6)
  1. [Appendix A] Appendix A ends with the incomplete sentence 'For instructions to run an example probing experiment, please' and gives no actual instructions or code repository; the promised reproducibility materials should be supplied or the appendix removed.
  2. [Sec. 3.2 vs Table 1] Section 3.2 lists masked image modeling (MIM) as one of FLAVA's auxiliary objectives, but Table 1's objective column for FLAVA lists only ITC, MLM, and MMM; the table should be made consistent with the text.
  3. [Abstract, Sec. 5] The abstract's claim that 'functional relations such as determiners are predicted well by all models' is only true for labeling accuracy (Table 5, det 95.1 for CLIP); for attachment, CLIP scores 18.1 on det versus 84.0 for RoBERTa (Table 4). The metric being summarized should be named explicitly.
  4. [Sec. 1, Fig. 7] The Stable Diffusion 3.5 illustration cannot by itself isolate the CLIP text encoder, since that pipeline also uses T5-XXL and an image decoder; if retained, it should be labeled as anecdotal motivation rather than as evidence for the text-encoder claim.
  5. [Sec. 4.1, Table 2] The blanket statement that CLIP performance drops across layers is contradicted by CLIP-XLM-RoB-Large-B/32, whose LAS increases to 39.56 at layer 12 before falling; the text should say 'most CLIP variants' or otherwise qualify the trend.
  6. [Throughout] There are several typos and incomplete references, including 'RoBEERTa' and 'inforation' in Appendix D, a missing citation for T5-XXl in Appendix E, a stray comma in the abstract's first sentence, and duplicate references [7] and [8] for the same BERT paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central comparison uses an external probe on public checkpoints, and the only author-overlap resource is a nonsensical control that is not load-bearing.

full rationale

The paper's main claim is an empirical measurement: DepProbe is an external probing tool (Muller-Eberstein et al., 2022) trained on the external EWT treebank, and the models compared are public checkpoints (CLIP, FLAVA, RoBERTa, MiniLM, and sentence-encoder variants). No parameter is fitted to the target conclusion, and no reported score is a renamed fit; the train-versus-test gap in Table 6 is an overfitting check, not a hidden re-use of test information. The only author-overlap resource is the SPUD nonce dependency treebank (Arps et al., 2024), used in Section 3.3 and Section 5 as a nonsensical control to separate syntax from lexical co-occurrence. This is an independent test set used for a supporting control experiment, not a premise that encodes the conclusion, so it does not make the derivation circular. Related-work self-citations (e.g., Dumpala et al. 2024) are contextual and not load-bearing. The possible probe-fairness confound described in the skeptic note is a validity concern about the measuring instrument, not a circularity of the paper's derivation, and therefore does not affect this verdict.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities or parameters are postulated; the result is an empirical measurement. The only free parameters belong to the DepProbe instrument inherited from Mueller-Eberstein et al., and the paper does not report their values or variance. The strongest non-empirical load is the validity of linear probing and the comparability of CLIP versus FLAVA across multiple confounds.

free parameters (2)
  • DepProbe subspace dimension b = not reported (DepProbe default, up to 128)
    Controls the capacity of the syntactic subspace; attachment scores depend on it. The paper states a hyperparameter search found no improvement but does not report the search or final value.
  • DepProbe training configuration (epochs, learning rate, seeds) = not reported
    Probe training details affect point estimates; no seeds or optimization settings are given, so variance across runs is unknown.
assumptions (4)
  • domain assumption Linear separability of dependency-tree properties under DepProbe is a valid and comparable measure of syntactic knowledge across models.
    Invoked throughout Sec. 3.1-4; no control probes on random representations or probe-potential analysis are provided.
  • domain assumption Universal Dependencies trees on EWT and SPUD capture the syntactic structure relevant to VLM text encoders.
    Sec. 3.3 chooses EWT and SPUD as ground truth; if the annotation scheme is not the right target, the measured gap could be an artifact of task choice.
  • ad hoc to paper Differences between CLIP and FLAVA can be attributed mainly to pre-training objective, not to data domain, data size, model size, or auxiliary image-side objectives.
    Sec. 3.2 and Sec. 4 compare CLIP (151M params, 400M pairs, ITC) with FLAVA (358M params, 70M pairs, ITC+MLM+MMM+MIM); the conclusion that the pre-training objective plays a more crucial role depends on this comparability.
  • domain assumption Layer indexing and subword averaging produce comparable token representations across models with different tokenizers.
    Sec. 3.1 averages subword tokens; CLIP and RoBERTa use different tokenizers, which may affect probing at corresponding layers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models." pith.science (2026). https://pith.science/paper/I5TKNXZW

@misc{pith2026241208111,
  author       = {Pith},
  title        = {Pith review of: Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5TKNXZW}},
  note         = {Machine review of arXiv:2412.08111}
}
read the original abstract

Vision-language models (VLMs), serve as foundation models for multi-modal applications such as image captioning and text-to-image generation. Recent studies have highlighted limitations in VLM text encoders, particularly in areas like compositionality and semantic understanding, though the underlying reasons for these limitations remain unclear. In this work, we aim to address this gap by analyzing the syntactic information, one of the fundamental linguistic properties, encoded by the text encoders of VLMs. We perform a thorough analysis comparing VLMs with different objective functions, parameter size and training data size, and with uni-modal language models (ULMs) in their ability to encode syntactic knowledge. Our findings suggest that ULM text encoders acquire syntactic information more effectively than those in VLMs. The syntactic information learned by VLM text encoders is shaped primarily by the pre-training objective, which plays a more crucial role than other factors such as model architecture, model size, or the volume of pre-training data. Models exhibit different layer-wise trends where CLIP performance dropped across layers while for other models, middle layers are rich in encoding syntactic knowledge.

Figures

Figures reproduced from arXiv: 2412.08111 by the authors.

Figure 1
Figure 1. Four images sampled from the StableDiffusion 3.5 model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example predictions. FLAVA and RoBERTa predict the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the layer-wise performance between RoBERTa and CLIP. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example predictions with different types of errors. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Sentence length distribution in thee filtered EWT training [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the layer-wise performance between [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Examples of additional images sampled from the StableD [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 54 canonical work pages

  1. [1]

    Is bert blind? exploring the effect of vision-and-language pre- training on visual language understanding

    Morris Alper, Michael Fiman, and Hadar Averbuch-Elor. Is bert blind? exploring the effect of vision-and-language pre- training on visual language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6778–6788, 2023. 3

    work page 2023

  2. [2]

    Probing for constituency structure in neural language models

    David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for constituency structure in neural language models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 6738–6757, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. 3

    work page 2022

  3. [3]

    Multilingual nonce dependency treebanks: Under- standing how language models represent and process syn- tactic structure

    David Arps, Laura Kallmeyer, Younes Samih, and Hassan Sajjad. Multilingual nonce dependency treebanks: Under- standing how language models represent and process syn- tactic structure. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), ...

    work page 2024

  4. [4]

    On the difference of bert-style and clip-style text encoders

    Zhihong Chen, Guiming Chen, Shizhe Diao, Xiang Wan, and Benyou Wang. On the difference of bert-style and clip-style text encoders. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 13710–13721, 2023. 3

    work page 2023

  5. [5]

    Chi, John Hewitt, and Christopher D

    Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5564–5577, Online,

    work page

  6. [6]

    Manning, Joakim Nivre, and Daniel Zeman

    Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. Universal Dependencies. Computational Linguistics, 47(2):255–308, 2021. 2, 3

    work page 2021

  7. [7]

    BERT: pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, pages 4171–4186. Associa- tion for Computa...

    work page 2019

  8. [8]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minnea...

    work page 2019

Show all 62 references
  1. [9]

    Sugar- crepe++ dataset: Vision-language model sensitivity to seman- tic and lexical alterations

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sugar- crepe++ dataset: Vision-language model sensitivity to seman- tic and lexical alterations. arXiv preprint arXiv:2406.11171,

    arXiv

  2. [10]

    Colorless green recurrent net- works dream hierarchically

    Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent net- works dream hierarchically. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    2018

  3. [11]

    Sensi- tivity of generative vlms to semantically and lexically altered prompts

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Sensi- tivity of generative vlms to semantically and lexically altered prompts. arXiv e-prints, pages arXiv–2410, 2024. 3

    2024

  4. [12]

    John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short ...

    2019

  5. [13]

    A structural probe for finding syntax in word representations

    John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pap...

    2019

  6. [14]

    What does bert learn about the structure of language? In ACL 2019- 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Ganesh Jawahar, Benoˆıt Sagot, and Djam ´e Seddah. What does bert learn about the structure of language? In ACL 2019- 57th Annual Meeting of the Association for Computational Linguistics, 2019. 1

    2019

  7. [15]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International confer- ence on machine learning, pages 4904–491...

    2021

  8. [16]

    Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Process- ing, Computational Linguistics, and Speech Recognition with Language Models. Prentice Hall, Pearson Education Interna- tional, 3rd edition, 2024. Online manuscript...

    2024

  9. [17]

    Which sentence embeddings and which layers encode syntac- tic structure? Cognitive Science, 2020

    M Alex Kelly, Yang Xu, Jes ´us Calvillo, and David Reitter. Which sentence embeddings and which layers encode syntac- tic structure? Cognitive Science, 2020. 1

    2020

  10. [18]

    Schr¨odinger’s tree—On syntax and neural language models

    Artur Kulmizev and Joakim Nivre. Schr¨odinger’s tree—On syntax and neural language models. Frontiers in Artificial Intelligence, 5, 2022. 1, 3

    2022

  11. [19]

    Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4077–4091, Online, 2020. Association for ...

    2020

  12. [20]

    Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4077–4091, 2020. 1

    2020

  13. [21]

    Do vision-language models understand compound nouns? arXiv preprint arXiv:2404.00419, 2024

    Sonal Kumar, Sreyan Ghosh, S Sakshi, Utkarsh Tyagi, and Dinesh Manocha. Do vision-language models understand compound nouns? arXiv preprint arXiv:2404.00419, 2024. 2, 16 9

    2024 arXiv

  14. [22]

    How is bert surprised? layerwise detection of lin- guistic anomalies

    Bai Li, Zining Zhu, Guillaume Thomas, Yang Xu, and Frank Rudzicz. How is bert surprised? layerwise detection of lin- guistic anomalies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat...

    2021

  15. [23]

    Align before fuse: Vision and language representation learning with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021. 2

    2021

  16. [24]

    BLIP- 2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP- 2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 1, 2

    2023

  17. [25]

    Open-vocabulary semantic segmentation with mask-adapted CLIP

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted CLIP. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 , pages 7061–

    2023

  18. [26]

    Syntactic structure from deep learning

    Tal Linzen and Marco Baroni. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195–212, 2021. 3

    2021

  19. [27]

    Roberta: A robustly optimized bert pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. 1, 4, 11

    2019

  20. [28]

    Ivanova, Idan A

    Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540, 2024. 1, 3

    2024

  21. [29]

    Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy

    Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision.Proceed- ings of the National Academy of Sciences , 117(48):30046– 30054, 2020. 3

    2020

  22. [30]

    Probing for labeled dependency trees

    Max M¨uller-Eberstein, Rob van der Goot, and Barbara Plank. Probing for labeled dependency trees. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers) , pages 7711– 7726, Dublin, Ireland, 2022. Association for Com...

    2022

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [32]

    Hierarchical text-conditional image genera- tion with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents. arXiv preprint arXiv:2204.06125, 2022. 1

    2022 arXiv

  25. [33]

    Cola: How to adapt vision-language models to compose objects localized with attributes? arXiv preprint arXiv:2305.03689, 2023

    Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A Plummer, Ranjay Krishna, and Kate Saenko. Cola: How to adapt vision-language models to compose objects localized with attributes? arXiv preprint arXiv:2305.03689, 2023. 1

    2023 arXiv

  26. [34]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics,

    2019

  27. [35]

    Pho- torealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Pho- torealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informati...

    2022

  28. [36]

    Laion-5b: An open large-scale dataset for training next gen- eration image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next gen- eration image-text models. Advances in Neural Infor...

    2022

  29. [37]

    A gold standard dependency corpus for En- glish

    Natalia Silveira, Timothy Dozat, Marie-Catherine de Marn- effe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. A gold standard dependency corpus for En- glish. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) , p...

    2014

  30. [38]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 156...

    2022

  31. [39]

    Masked language mod- eling and the distributional hypothesis: Order word matters pre-training for little

    Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language mod- eling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language ...

    2021

  32. [40]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2022

  33. [41]

    Diffusion lens: Interpreting text encoders in text-to-image pipelines

    Michael Toker, Hadas Orgad, Mor Ventura, Dana Arad, and Yonatan Belinkov. Diffusion lens: Interpreting text encoders in text-to-image pipelines. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9713–972...

  34. [42]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019. 3

    2019

  35. [43]

    Can linguistic knowledge improve multimodal alignment in vision-language pretraining? arXiv preprint arXiv:2308.12898, 2023

    Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changx- ing Ding. Can linguistic knowledge improve multimodal alignment in vision-language pretraining? arXiv preprint arXiv:2308.12898, 2023. 2, 16 10

    2023 arXiv

  36. [44]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. 4, 11

    2020

  37. [45]

    When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. 1, 2, 16

    2023

  38. [46]

    Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations

    Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221,

    arXiv

  39. [50]

    We evalu- ated the ’ViT-B/32’ variant of CLIP – ViT base model trained with a image patch size of 32 – publicly avail- able at the following HuggingFace Link

    CLIP [ 31]: 12-layered transformer encoder model trained on 400 million image-text pairs. We evalu- ated the ’ViT-B/32’ variant of CLIP – ViT base model trained with a image patch size of 32 – publicly avail- able at the following HuggingFace Link

  40. [51]

    We evaluated the FLA V A pre-trained Model available at the following HuggingFace Link • Unimodal Language Models (ULMs)

    FLA V A [38]: 12-layered transformer encoder model trained on 70 million image-text pairs. We evaluated the FLA V A pre-trained Model available at the following HuggingFace Link • Unimodal Language Models (ULMs)

  41. [52]

    We evaluated the RoBERTa-base model available at the following Hug- gingFace Link

    RoBERTa-base [27]: 12-layered transformer network trained exclusively on text data. We evaluated the RoBERTa-base model available at the following Hug- gingFace Link

  42. [53]

    We evaluated the RoBERTa-large model available at the following Hug- gingFace Link

    RoBERTa-large [27]: 24-layered transformer network trained exclusively on text data. We evaluated the RoBERTa-large model available at the following Hug- gingFace Link

  43. [54]

    We evaluated the MiniLM model avail- able at the following HuggingFace Link • Sentence Language Models (SLMs) [34]

    MiniLM [44]: 12-layered transformer network trained exclusively on text data by distilling the BERT pre- trained model. We evaluated the MiniLM model avail- able at the following HuggingFace Link • Sentence Language Models (SLMs) [34]

  44. [55]

    We eval- uated the sentence-MiniLM model available at the fol- lowing HuggingFace Link

    Sentence-MiniLM [ 34]: Sentence language model trained on top of MiniLM (explained above). We eval- uated the sentence-MiniLM model available at the fol- lowing HuggingFace Link

  45. [56]

    Sentence-RoBERTa-large [34]: Sentence language model trained on top of RoBERTa-large model (ex- plained above). We evaluated the sentence-MiniLM model available at the following HuggingFace Link Variants of CLIP Below, we provide details of the var- ious CLIP model variants ev...

  46. [57]

    Here the images are provided as input with a patch size of 32

    CLIP-ViT-B/32 [31]: ViT base model with 12 transformer layers as the backbone of the text encoder. Here the images are provided as input with a patch size of 32. This model is trained using the WebImageText dataset [ 31] consisting of 400M image-text pairs. We evaluated the CL...

  47. [58]

    This model is also trained us- ing the WebImageText dataset [31] consisting of 400M image-text pairs

    CLIP-ViT-B/16 [31]: This model is same as the CLIP- ViT-B/32 model but here the images are provided as input with a patch size of 16. This model is also trained us- ing the WebImageText dataset [31] consisting of 400M image-text pairs. We evaluated the CLIP-ViT-B/16 model publ...

  48. [59]

    We evaluated the LAION-CLIP-ViT-B/32 model publicly available at the following HuggingFace Link

    LAION-CLIP-ViT-B/32 [36]: This model is similar to the CLIP-ViT-B/32 model but this model is trained using the 2 billion image-text pairs from the LAION dataset [36]. We evaluated the LAION-CLIP-ViT-B/32 model publicly available at the following HuggingFace Link

  49. [60]

    The pre- training process utilized 5 billion image-text pairs from the LAION dataset

    CLIP-XLM-RoBERTa-base-ViT-B/32 (referred to as CLIP-XLM-RoB-Base-B/32 in Table 2 of main pa- per) [ 36]: The text encoder of this model is initial- ized with RoBERTa-base weights and subsequently pre- trained using a contrastive learning framework. The pre- training process ut...

  50. [61]

    The pre- training process utilized 5 billion image-text pairs from the LAION dataset

    CLIP-XLM-RoBERTa-large-ViT-H/14 (referred to as CLIP-XLM-RoB-Large-B/32 in Table 2 of main pa- per) [ 36]: The text encoder of this model is initial- ized with RoBERTa-large weights and subsequently pre- trained using a contrastive learning framework. The pre- training process...

  51. [62]

    and Table 13 (Layer 12). CLIP FLA V A RoBERTa Score ∆ Score ∆ Score det 10.7 64.5 60.0 15.2 75.2 cc 9.2 54.0 56.8 6.3 63.2 mark 11.9 51.0 56.5 6.4 62.9 obj 31.5 44.3 66.1 9.6 75.8 case 7.3 43.5 47.8 3.0 50.8 amod 30.2 40.7 58.5 12.4 70.9 cop 20.6 39.3 57.5 2.5 60.0 aux 36.2 37...

  52. [2020]

    Association for Computational Linguistics. 3

  53. [2022]

    Implementation For instructions to run an example probing experiment, please A.1

    1 A. Implementation For instructions to run an example probing experiment, please A.1. Probing The experiments were conducted using the public implemen- tation of DepProbe ( https://github.com/personads/ depprobe). B. Details of the Models Evaluated in this Study Below we prov...

  54. [2024]

    Association for Computational Linguistics. 2

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.