{"id":"bcfb0335-153f-4c87-a2aa-b4ec7ae3655a","arxiv_id":"2412.08111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Vision-language text encoders encode less syntactic structure than text-only encoders, and contrastive pre-training rather than model size or data volume accounts for most of the deficit.","lead":"This paper probes the text encoders of vision-language models for grammar knowledge and finds they capture far less sentence structure than text-only language models. The gap is tied mainly to the training objective: contrastive image-text training leaves syntax weak, while adding masked-language objectives restores much of it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DepProbe's linear readout may systematically underreport syntax in CLIP; a non-linear control probe is needed before attributing the gap to the contrastive objective.","rationale":"The reader's weakest assumption correctly identifies the probe as a potential source of systematic bias. I agree that this is the most load-bearing concern: the paper's headline result is a comparison of DepProbe scores across model families, and the causal attribution to pre-training objective depends on that comparison being fair. A non-linear control experiment would directly test whether CLIP's syntax is simply harder to read linearly, in which case the paper's claim about syntactic learning limitations would need to be rephrased as a claim about linear readout limitations. The reader already set the verdict to CONDITIONAL on this and related methodological issues, and I do not find a reason to move it further: the relative ordering is large, the SPUD control adds some robustness, and the SLM comparison provides independent (though not conclusive) support for the objective hypothesis. If the proposed test shows no selective improvement for CLIP, the paper's central conclusions remain plausible. I therefore recommend keeping the CONDITIONAL verdict unchanged rather than escalating to REJECT or UNVERDICTED.","tokens_in":20752,"tokens_out":5509,"duration_ms":63995,"concrete_test":"Run a control probe on layer-6 representations of CLIP-ViT-B/32, FLAVA, and RoBERTa-base using the same EWT train/test splits and evaluation, but with a 2-layer MLP (or with L2-normalized/whitened input vectors fed to DepProbe). If CLIP's LAS rises from 26.25 to near 60 while RoBERTa-base stays near 64.43, the linear readout underreports CLIP syntax and the central gap is not robust; if the relative ordering and gap size remain roughly unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison (Tables 1 and 7) rests on DepProbe, a linear probe trained on averaged subword token representations with default hyperparameters (Sec. 3.1). This assumes that syntactic information is equally linearly separable across models with different tokenizers, embedding geometries, and optimization targets. CLIP's text encoder is known to produce anisotropic, cone-shaped representations, and its BPE tokenizer produces different subword splits than RoBERTa's BPE; both can systematically lower a linear probe's accuracy even when the underlying syntax is encoded in a non-linear form. The paper's statement that a hyperparameter search yielded 'no significant improvements' is not detailed, so it is unknown whether the search was per-model or sufficiently broad. The train-test gap in Table 6 rules out overfitting but does not address cross-model sensitivity. If CLIP encodes syntax in a less linearly separable fashion, the reported LAS gap (64.43 vs 26.25 at layer 6) is a measurement artifact, and the conclusion that contrastive pre-training inhibits syntactic learning (Sec. 5, 6) is unsupported. The FLAVA comparison is additionally confounded by MMM/MIM objectives, data, and parameter count, but the probe-fairness issue is more fundamental because it also undermines the ULM-versus-VLM gap itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether VLM text encoders encode syntactic structure, using DepProbe to predict Universal Dependency trees from frozen representations of CLIP, FLAVA, RoBERTa, MiniLM, Sentence-RoBERTa, and Sentence-MiniLM. It reports large gaps between CLIP and ULM text encoders at middle layers (e.g., LAS 26.25 vs 64.43 at layer 6), near-ULM performance for FLAVA, no recovery from larger CLIP models or more data, final-layer degradation for sentence-level models, and stable results on the SPUD nonsensical control. The paper concludes that the pre-training objective is the primary determinant of syntactic knowledge in VLM text encoders, with token-level objectives such as MLM being beneficial.","tokens_in":21015,"tokens_out":12083,"duration_ms":117766,"significance":"If the causal attribution holds, the paper provides an intrinsic evaluation that can explain compositionality failures of CLIP and motivates auxiliary token-level objectives. The measurement has strong points: DepProbe is an external probe, public checkpoints are used, the train/test gap in Table 6 rules out probe overfitting, and the SPUD nonce-sentence evaluation in Table 7 separates syntactic from lexical co-occurrence information. The layer-wise tables and per-relation breakdowns are detailed and useful. The causal claim, however, goes beyond the current experimental design, and the probe-fairness hypothesis is a genuine threat to the interpretation of the headline LAS gap.","major_comments":[{"comment":"The central claim that the pre-training objective is the main determinant of syntactic knowledge is not established by the CLIP-versus-FLAVA comparison. FLAVA differs from CLIP not only in having an MLM objective but also in total parameters (358M vs 151M), training data volume and domain (70M image-text pairs vs 400M, per Appendix B), and additional auxiliary objectives (MMM and MIM; Sec. 3.2). The manuscript also does not state whether FLAVA's text encoder was initialized from a pretrained ULM; if it was, as in the original FLAVA release, the near-ULM syntax scores could be inherited from initialization rather than learned through MLM. The claim should therefore be softened to a correlation with the objective, or supported by a controlled comparison such as a CLIP variant trained with an added MLM objective or an ablation of FLAVA's auxiliary losses.","section":"Sec. 4.1, Tables 1 and 7"},{"comment":"DepProbe is a linear probe (Eqs. 1-3) applied after element-wise mean pooling of subword tokens, so it measures linear separability of dependency-tree information, not syntactic information per se. CLIP's different BPE tokenizer, anisotropic representation geometry, and mean pooling can all lower a linear probe's accuracy even if the underlying syntax is nonlinearly encoded. The conclusion in Sec. 6 that CLIP 'fails to capture even basic features of the dependency tree' is therefore stronger than the evidence supports. The one-sentence report in Sec. 3.1 that a hyperparameter search gave 'no significant improvements' does not describe the search space or whether it was per-model. A nonlinear control probe (e.g., a one-hidden-layer MLP probe) or an anisotropy/whitening control is needed before attributing the 26.25-versus-64.43 LAS gap to the pre-training objective; otherwise the wording should be restricted to 'syntax is not linearly readable.'","section":"Sec. 3.1, Sec. 6"},{"comment":"The text states that LAION-CLIP-ViT-B/32 'does not show any improvement over CLIP-ViT-B/32,' but Table 2 reports layer-6 LAS/UAS/UUAS of 32.12/40.29/45.71 for LAION-CLIP versus 26.25/33.49/38.91 for CLIP-ViT-B/32, which is an improvement of roughly 5-7 points. Because no variance or multiple-seed results are reported, it is impossible to tell whether these differences are meaningful; the text should either report error bars or explicitly treat such differences as within noise, and the sentence should be corrected.","section":"Sec. 4.2, Table 2"}],"minor_comments":[{"comment":"Appendix A ends with the incomplete sentence 'For instructions to run an example probing experiment, please' and gives no actual instructions or code repository; the promised reproducibility materials should be supplied or the appendix removed.","section":"Appendix A"},{"comment":"Section 3.2 lists masked image modeling (MIM) as one of FLAVA's auxiliary objectives, but Table 1's objective column for FLAVA lists only ITC, MLM, and MMM; the table should be made consistent with the text.","section":"Sec. 3.2 vs Table 1"},{"comment":"The abstract's claim that 'functional relations such as determiners are predicted well by all models' is only true for labeling accuracy (Table 5, det 95.1 for CLIP); for attachment, CLIP scores 18.1 on det versus 84.0 for RoBERTa (Table 4). The metric being summarized should be named explicitly.","section":"Abstract, Sec. 5"},{"comment":"The Stable Diffusion 3.5 illustration cannot by itself isolate the CLIP text encoder, since that pipeline also uses T5-XXL and an image decoder; if retained, it should be labeled as anecdotal motivation rather than as evidence for the text-encoder claim.","section":"Sec. 1, Fig. 7"},{"comment":"The blanket statement that CLIP performance drops across layers is contradicted by CLIP-XLM-RoB-Large-B/32, whose LAS increases to 39.56 at layer 12 before falling; the text should say 'most CLIP variants' or otherwise qualify the trend.","section":"Sec. 4.1, Table 2"},{"comment":"There are several typos and incomplete references, including 'RoBEERTa' and 'inforation' in Appendix D, a missing citation for T5-XXl in Appendix E, a stray comma in the abstract's first sentence, and duplicate references [7] and [8] for the same BERT paper.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the probe-fairness concern raised in review is real and should be addressed by the authors; the larger risk, in my view, is the FLAVA initialization confound. If the authors confirm that FLAVA's text encoder was initialized from a pretrained ULM, the abstract's central attribution claim will need to be substantially weakened or resupported. The paper is nevertheless a useful measurement study and I would not reject it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first layerwise dependency-probing of VLM text encoders I know, and the main result—CLIP's syntax scores collapse in later layers while MLM-trained FLAVA tracks RoBERTa—is real and worth engaging. The paper does good things: DepProbe and UD/SPUD are external, the train/test gap is small, the SPUD control shows the effect isn't lexical co-occurrence, and the CLIP variant sweep (size/data/patch) is a sensible way to test alternative explanations. The XLM-RoBERTa CLIP result is underplayed: that text encoder starts from RoBERTa-base weights and then gets contrastive training, and its layer-12 LAS drops to roughly 15 from RoBERTa's 54. That is the cleanest evidence in the paper that the objective, not architecture, drives the loss of linearly accessible syntax.\n\nSoft spots, in order. First, the probe-fairness worry. DepProbe is linear, and CLIP's representations are known to be anisotropic; the paper's one-line 'hyperparameter search found no improvements' doesn't tell us whether the search was per-model or broad enough. A nonlinear control probe, or at least per-model tuning with reported ranges and seeds, would materially firm up the central comparison. I don't think this sinks the paper—the XLM-RoBERTa before/after comparison and the SLM final-layer drop both point in the same direction—but the 'syntactic information learned' wording should be softened to 'linearly accessible syntax' until that control is done. Second, FLAVA differs from CLIP in data domain, data size, and MMM/MIM, so 'pre-training objective' is not isolated to MLM; the SLM comparison helps, but the paper should say this more carefully. Third, no error bars or significance tests, and the appendix has an incomplete sentence ('For instructions... please') and a missing reference for T5-XXL. Minor, but sloppy.\n\nWho this is for: people working on VLM compositionality, contrastive representation learning, and syntactic probing. It deserves a serious referee—conditional accept is the right verdict, with the probe-fairness control and confound discussion as the main revision asks. I'd send it out.","headline":"A worthwhile layerwise syntax-probing study whose core finding—contrastive-only training leaves VLM text encoders with little linearly accessible dependency syntax—survives the obvious confounds, though a nonlinear probe control and released configuration are needed before the causal story is airtight.","tokens_in":21550,"tokens_out":2787,"would_cite":true,"duration_ms":30077,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that the pre-training objective, not model size or data volume, determines how much syntax vision-language text encoders acquire.","keywords":["syntax probing","vision-language models","text encoders","contrastive learning","masked language modeling","dependency parsing","Universal Dependencies","layer-wise analysis"],"falsifier":"Train two text encoders with identical architecture and data, one with contrastive loss only and one with an added masked-language-model objective, and compare DepProbe scores: if the contrastive-only model matches the masked-language-model model's labeled attachment score, the pre-training-objective claim is refuted. Alternatively, re-run the probe with per-model hyperparameter tuning and a nonlinear readout; if CLIP's scores rise to RoBERTa's level, the gap was a linear-separability artifact.","tokens_in":20532,"feed_emoji":"🧠","tokens_out":4767,"duration_ms":43107,"temperature":0.7,"pith_summary":"This paper tries to establish that the text encoders of vision-language models encode far less syntactic structure than the text-only language models they are usually compared with, and that the cause is the pre-training objective rather than architecture, parameter count, or data volume. Using a linear probe that reads Universal Dependency trees out of frozen representations, the authors find that CLIP, trained with a sentence-level contrastive loss, scores dramatically lower than RoBERTa and even loses syntactic information as layers deepen. FLAVA, which adds masked-language-modeling to contrastive loss, nearly matches RoBERTa, indicating that token-level objectives are what make syntax linearly accessible. If true, this isolates a concrete, fixable bottleneck in multimodal pretraining and helps explain why VLM text encoders behave like bag-of-words models.","feed_headline":"Contrastive training, not size, starves VLM text encoders of syntax","feed_subtitle":"Probing shows contrastive-only CLIP scores 26 vs RoBERTa's 64; adding MLM nearly closes the gap.","key_machinery":"DepProbe is a linear probing classifier that decodes labeled, directed dependency trees from frozen word representations: a matrix $L$ labels each word's incoming dependency relation, and a matrix $B$ projects representations into a low-dimensional \"syntactic subspace\" whose distances are trained to mirror distances in the dependency tree. The probe is trained on the EWT Universal Dependencies treebank and evaluated with labeled attachment score, unlabeled attachment score, and related metrics. The argument is carried by comparing probes across models that vary one factor at a time — pre-training objective (CLIP versus FLAVA versus uni-modal models), parameter count (151 million to 1.2 billion), data volume (400 million to 5 billion pairs), and layer depth — so that the objective is the only factor that tracks the observed score gaps.","core_discovery":"The central claim is that uni-modal language model text encoders acquire syntactic knowledge more effectively than vision-language model text encoders, and that this gap is driven by the pre-training objective. On the EWT treebank with DepProbe at layer 6, RoBERTa-base reaches a labeled attachment score of 64.43, FLAVA reaches 60.83, while CLIP-ViT-B/32 reaches only 26.25; scaling CLIP from 151 million to 1.2 billion parameters or from 400 million to 5 billion image-text pairs does not close the gap. CLIP's probe performance degrades from layer to layer, whereas RoBERTa and FLAVA peak at middle layers. Sentence-level models trained on natural language inference behave like CLIP at the final layer, reinforcing the conclusion that global sentence objectives suppress local syntactic structure.","pith_inferences":["A testable extension: finetune a CLIP text encoder with a masked-language-model objective on its original caption data and check whether DepProbe scores rise; if they do, the objective attribution is causal rather than correlational.","The same probe setup could predict text-to-image binding failures: prompts with high dependency distance between subject and object should fail more often, linking the measured score gap to observable image errors.","The authors' linear-probe lens may understate CLIP's syntax if the information exists in a nonlinear form; a parser-based or nonlinear evaluation could find more syntax than DepProbe reports, without changing the conclusion that it is not linearly accessible.","The similarity between sentence-level language models and CLIP at the final layer suggests that any objective that pools words into a global sentence vector — contrastive or natural-language-inference — tends to strip local dependency structure at the output layer."],"forward_implications":["Adding a masked-language-model or other token-level objective to contrastive vision-language pretraining should substantially restore syntactic encoding, close to unimodal levels.","Scaling up parameter count or image-text data for contrastive-only models will not, by itself, improve syntax; the paper's CLIP variants show flat or worse scores.","Syntactic probing of text encoders can serve as an intrinsic, image-free diagnostic for multimodal pretraining quality.","In text-to-image pipelines that use CLIP-style encoders, prompts whose meaning depends on word order, such as \"A cat chases a dog,\" will remain unreliable unless the text encoder is retrained with token-level objectives.","Layer-wise, the final layers of contrastive and sentence-level encoders are the least syntactic, so interventions should target the top of the encoder."],"supporting_citations":[{"why":"Supplies the DepProbe method used to decode labeled, directed dependency trees from frozen text-encoder representations.","marker":"[30]"},{"why":"Defines CLIP and its contrastive image-text pretraining objective, the main model class under test.","marker":"[31]"},{"why":"Provides FLAVA, the vision-language model whose added masked-language-model objective nearly closes the syntactic gap to uni-modal models.","marker":"[38]"},{"why":"Supplies RoBERTa, the masked-language-model uni-modal baseline with the highest syntax probing scores.","marker":"[27]"},{"why":"Introduces sentence-level language models trained on natural language inference, used to draw the parallel with CLIP's global sentence objective.","marker":"[34]"},{"why":"Provides nonsensical but grammatical treebanks used to show that probe scores reflect syntax rather than word co-occurrence statistics.","marker":"[3]"},{"why":"Defines the Universal Dependencies annotation format and relation inventory that the probing task predicts.","marker":"[6]"},{"why":"Supplies the LAION-based CLIP variants with 2 billion and 5 billion image-text pairs, used to test the effect of data volume.","marker":"[36]"},{"why":"Provides the EWT Universal Dependencies treebank on which the probes are trained and evaluated.","marker":"[37]"}],"fun_headline_variants":["Syntax gap in VLMs: objective, not scale, is the culprit","Contrastive training leaves VLM text encoders syntax-starved","CLIP's syntax blindness: pre-training objective, not size, to blame","For syntax, language models beat VLMs—regardless of scale","Why CLIP can't see syntax: it's the training objective, not data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes DepProbe's scores measure syntactic knowledge equally across models with different tokenizers, embedding geometries, and training objectives, so part of CLIP's low score could be a measurement artifact rather than missing syntax.","fun_headline_variants_meta":{"raw":{"variants":["Syntax gap in VLMs: objective, not scale, is the culprit","Contrastive training leaves VLM text encoders syntax-starved","CLIP's syntax blindness: pre-training objective, not size, to blame","For syntax, language models beat VLMs—regardless of scale","Why CLIP can't see syntax: it's the training objective, not data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1350,"prompt_tokens":920,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":536,"tokens_out":430,"duration_ms":4583,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:12:24.123483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two text encoders with identical architecture and data, one with contrastive loss only and one with an added masked-language-model objective, and compare DepProbe scores: if the contrastive-only model matches the masked-language-model model's labeled attachment score, the pre-training-objective claim is refuted. Alternatively, re-run the probe with per-model hyperparameter tuning and a nonlinear readout; if CLIP's scores rise to RoBERTa's level, the gap was a linear-separability artifact.","supporting_citations":[{"cited_title":"Probing for labeled dependency trees","cited_arxiv_id":null,"evidence_quote":"Supplies the DepProbe method used to decode labeled, directed dependency trees from frozen text-encoder representations."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Defines CLIP and its contrastive image-text pretraining objective, the main model class under test."},{"cited_title":"Flava: A foundational language and vision alignment model","cited_arxiv_id":null,"evidence_quote":"Provides FLAVA, the vision-language model whose added masked-language-model objective nearly closes the syntactic gap to uni-modal models."},{"cited_title":"Roberta: A robustly optimized bert pretraining approach, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies RoBERTa, the masked-language-model uni-modal baseline with the highest syntax probing scores."},{"cited_title":"Multilingual nonce dependency treebanks: Under- standing how language models represent and process syn- tactic structure","cited_arxiv_id":null,"evidence_quote":"Provides nonsensical but grammatical treebanks used to show that probe scores reflect syntax rather than word co-occurrence statistics."},{"cited_title":"Manning, Joakim Nivre, and Daniel Zeman","cited_arxiv_id":null,"evidence_quote":"Defines the Universal Dependencies annotation format and relation inventory that the probing task predicts."},{"cited_title":"Laion-5b: An open large-scale dataset for training next gen- eration image-text models","cited_arxiv_id":null,"evidence_quote":"Supplies the LAION-based CLIP variants with 2 billion and 5 billion image-text pairs, used to test the effect of data volume."},{"cited_title":"A gold standard dependency corpus for En- glish","cited_arxiv_id":null,"evidence_quote":"Provides the EWT Universal Dependencies treebank on which the probes are trained and evaluated."}],"review_version":1}