{"id":"4b9e5965-35b9-40dc-bf42-9bb676793ba8","arxiv_id":"2412.09353","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ordering word prediction by a dependency tree instead of left-to-right improves vision-language compositional understanding across five benchmarks.","lead":"The authors train a vision-language model decoder to generate captions in an order dictated by a sentence's grammatical dependency tree, not left to right. This 'causally ordered' training beats prior methods on five compositional benchmarks, even when those used much larger training sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No blind-model ablation: COGT's benchmark gains could stem from a language prior rather than visual compositional grounding.","rationale":"The reader's weakest assumption was causal sufficiency of the dependency tree, which is a modeling idealization explicitly acknowledged in App. A. My concern is distinct and more directly tied to the central empirical claim: the evaluation may be confounded by the decoder's language prior. The paper uses the blind-LM argument to drop COCO Order and Flickr Order from ARO, yet does not run the same no-vision control for its own model on the benchmarks it reports. This is not an ad-hoc critique; it is the standard validity check that the authors themselves invoke for other tasks. If the no-vision COGT achieves high accuracy, the headline claim of outperforming all SOTA compositional approaches by a large margin would be overstated, because the advantage could be grammaticality or structural preference rather than image-grounded compositional reasoning. The concern is testable with a single controlled experiment, and it does not require rejecting the method's internal consistency or the ablations. I therefore keep the reader's CONDITIONAL verdict: acceptance should be contingent on this blind-model control. I mark agreement as partial because the reader's rationale touches on evaluation issues (error bars, mixed baselines) but identifies a different weakest assumption; the dependency tree is central in both cases, but the failure mode I highlight is about the absence of a no-vision baseline, not about parser misspecification.","tokens_in":28488,"tokens_out":10789,"duration_ms":129398,"concrete_test":"Train an identical COGT decoder with the same dependency-guided factorization but no cross-attention to visual features Z (or at inference, replace Z with a constant vector), and evaluate the same five benchmarks. Compare this no-vision variant against the full COGT model and against a standard blind LM baseline. If the no-vision variant retains high accuracy (e.g., above 70% on SugarCrepe or ARO), the reported gains are substantially language-prior artifacts rather than evidence of compositional visual understanding. Additionally, run a control with randomized dependency trees at inference to determine how much of the score depends on the specific parse versus the partial-order mechanism itself.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is not the causal-sufficiency of the parse tree (App. A), but that the reported benchmark scores reflect image-text alignment rather than the decoder's language prior. The decoder is a 64M-parameter generative model trained on COCO captions, and at inference it scores each candidate caption by log P(W|PA(W),S,Z). The paper excludes COCO Order and Flickr Order because a 'blind' LM with no image solves them at ~99%, but it never applies the same ablation to its own model on the remaining ARO, SugarCrepe, VL-Checklist, ColorSwap, or FG-OVD tasks. If COGT's decoder is itself a strong blind LM on these tasks—for example preferring syntactically deeper trees or more natural continuations—then a large fraction of the reported +17.7 points over Fully-Parallel and the SOTA margins could come from a linguistic prior, not from compositional grounding. The dependency-guided factorization (Eq. 1) may be acting as a better language model, not a better vision-language model. The internal ablations in Tab. 1 compare only against other generative decoders with visual conditioning and do not isolate the contribution of Z. This is a sharper threat to the central claim than the acknowledged causal-sufficiency idealization because it targets the validity of the evaluation itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes COGT, a method that adapts a frozen VLM visual encoder with a small decoder trained to predict caption words in an order defined by a dependency-tree-based CGM. Equation (1) factorizes the joint caption probability using, for each word, its dependency-tree ancestors, its syntactic type, and all visual features; at inference, candidate captions are scored by this factorization using a level-order traversal of the dependency tree. Experiments on ARO, SugarCrepe, VL-Checklist, ColorSwap, and a newly proposed FG-OVD retrieval benchmark compare COGT with Fully-Parallel, Sequential-AR, Mixed, and prior compositional methods, reporting large gains, including over Cap and CapPa, which were trained on much larger data. Ablations address parser choice, mask-specific syntactic tokens, and the number of visual feature layers, and downstream linear-probing results show no degradation of the frozen encoder's features.","tokens_in":28743,"tokens_out":5282,"duration_ms":55435,"significance":"COGT is conceptually clean, and the internal ablations in Tables 1, 2, and 7 are well designed: they isolate the prediction order, the parser, the syntactic masks, and the visual feature layers. The explicit statement of the causal-sufficiency assumption in Section 3 and Appendix A, together with the public release of code and weights, is a further strength. If the benchmark gains are robust, the method would be an important demonstration that syntactic dependency structure provides a useful inductive bias for compositional VLM evaluation while training on far less data than Cap/CapPa.","major_comments":[{"comment":"The central comparison reports single runs with no standard deviations, confidence intervals, or significance tests. The abstract's claim that COGT \"significantly outperforms all the state-of-the-art compositional approaches by a large margin\" and the text's \"significant margin\" in Section 4.1 are therefore not statistically grounded; some task-level differences are small, for example SugarCrepe Swap is 98.26 for COGT and 98.96 for Fully-Parallel in Table 1. Please report means and standard deviations over at least three seeds, or a paired bootstrap over test samples, for the main tables.","section":"Section 4.1, Tables 1 and 2"},{"comment":"The paper excludes COCO Order and Flickr Order because a \"blind\" LM can solve them at about 99%, but it never applies the same control to its own decoder on the benchmarks it reports. COGT scores candidate captions with a 64M decoder trained on COCO captions via Eq. (4), so a language prior from well-formed dependency structures could explain part of the gain over Fully-Parallel and Sequential-AR. Please add a no-image ablation, for example replacing Z with a constant or with features from a different image, on ARO, SugarCrepe, VL-Checklist, ColorSwap, and FG-OVD, and report the resulting drop. The FG-OVD Trivial results in Table 10, where COGT-CLIP underperforms CLIP zero-shot, suggest COGT is not simply a stronger blind LM on all tasks, but the control is still needed to attribute the reported gains to visual compositional grounding.","section":"Section 4, first paragraph; Section 3.1 Inference"},{"comment":"The causal-sufficiency assumption that the dependency tree's ancestors, the syntactic type, and the visual features are sufficient parents for each word is stated explicitly in Appendix A but is not tested against alternative graph structures. The parser ablation in Table 2 varies parser quality within the same dependency-tree formalism, so it does not establish that the specific causal structure, rather than the sparsity of the factorization, is responsible for the gains. A concrete test is to train COGT with a random dependency tree, a deliberately reversed tree, or a constituency-tree order; if the gains persist, the conclusion should be reframed as a sparsity benefit rather than a causal-structure benefit. This is a correctness-risk concern, not an internal inconsistency, because the paper acknowledges the assumption.","section":"Section 3, Eq. (1); Appendix A"}],"minor_comments":[{"comment":"There are several typos: \"significatively\" in Related Work, \"rispectively\" in Appendix C.4, \"InstrctBLIP\" in Appendix C.3, and \"comuunicates\" in the Related Work discussion of CoVLM.","section":"Throughout"},{"comment":"The figure caption does not explain the meaning of the dashed lines, colors, or the distinction between the dependency-guided attention layer and the cross-attention layer; please add a legend or a more detailed caption.","section":"Figure 2"},{"comment":"The caption states that x, x, and x* indicate the first, second, and third best results, but these markers do not appear in the rendered table; please either add the markers to the table or remove the sentence.","section":"Table 3 caption"},{"comment":"The FG-OVD construction says that each object is associated with about ten negative captions, but it does not specify how the average is computed across the Trivial, Easy, Medium, and Hard tasks or whether the sampling is stratified by object class; adding this information would improve reproducibility.","section":"Appendix B"},{"comment":"The mapping from the parser output to the 45 syntactic categories of V is implicit; please state explicitly how relations such as \"root\" and words with multiple relations are handled, and how the universal dependency labels from Silveira et al. (2014) are matched to the parser output.","section":"Section 3.1"},{"comment":"The Winoground and MMVP results are reported without confidence intervals, and the MMVP discussion notes the small sample size; given that MMVP has only 135 samples, I suggest reporting exact binomial confidence intervals or treating those numbers as illustrative only.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I found no evidence of misconduct or circular evaluation; the internal ablations are carefully controlled and the code release is a plus. The main risk is interpretive: the reported margins may overstate visual grounding without a no-image control and without variance estimates. I recommend major_revision rather than rejection because both gaps are fixable within the manuscript's scope and the central derivation is sound. The paper's ICLR publication status does not change my assessment of the arXiv version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious paper about using dependency parses to define a decoding order for a small generative decoder on top of a frozen CLIP encoder, trained for compositional VL understanding. The core idea is new in this combination, and the internal ablations are the strongest part. They compare COGT against fully-parallel, sequential-AR, and a CapPa-style mixed objective using the same decoder size, same frozen encoder, and same COCO training data. The ordering is consistent: COGT beats them across the board, and the component ablations (parser choice, mask-specific syntactic tokens, two-layer visual features) are well designed. That is real evidence the mechanism works as claimed. They also release code and weights, and the FG-OVD retrieval benchmark is a useful addition to the evaluation suite.\n\nNow the soft spots, in proportion. First, no standard deviations or multiple seeds anywhere. The abstract's 'large margin' is not statistically grounded; on VL-Checklist Relation, COGT actually loses to its own Mixed baseline by two points. Second, the comparison with Cap and CapPa is confounded: those models were trained from scratch on 1B image-text pairs, while COGT sits on a frozen CLIP visual encoder that already carries a large pretraining corpus. Saying COGT 'improves over methods trained using much larger datasets' is misleading. Third is the stress-test concern I take seriously: the paper excludes COCO Order and Flickr Order because a blind LM solves them, but never runs the same blind-LM ablation on ARO, SugarCrepe, VL-CheckList, ColorSwap, or FG-OVD. A dependency-ordered decoder is plausibly a better language model, and if most of its score comes from a linguistic prior rather than visual grounding, the central claim weakens. The internal ablations show visual features matter—dropping the penultimate layer costs 4.75 points average—but that does not isolate the prior contribution on these specific benchmarks. This is not fatal, but it is unresolved. Finally, the causal-sufficiency of the parse tree is assumed, not validated against alternative graph structures; the paper acknowledges this explicitly, but it remains a limitation.\n\nBottom line: the method is clever, the implementation is careful, and the internal evidence supports the ordering among training objectives. The external SOTA claims are overstated, and the missing blind-LM ablation should be done. This paper is worth a serious referee—I would engage with it, but I would push for error bars, a blind-decoder baseline on all benchmarks, and a fairer description of the data-efficiency comparison.","headline":"A well-controlled internal comparison shows the dependency-ordered decoder helps, but the SOTA claims outrun the evidence: no error bars, an unvalidated causal-sufficiency assumption, and no blind-LM ablation on the main benchmarks.","tokens_in":29279,"tokens_out":2695,"would_cite":true,"duration_ms":29937,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ordering caption generation by a dependency tree, instead of left-to-right or fully parallel, makes vision-language models learn true word relations and outperform much larger models.","keywords":["compositional understanding","vision-language models","causal graphical models","dependency parsing","generative training","image-text retrieval","fine-grained attributes","CLIP adaptation"],"falsifier":"Take the same COGT decoder and compare parser-based trees against gold-standard human dependency parses and against deliberately corrupted trees on a fixed benchmark; if COGT's gains are caused by correct causal structure, accuracy should track parse quality, and random or wrong trees should erase the gap.","tokens_in":28308,"feed_emoji":"🧩","tokens_out":5628,"duration_ms":52916,"temperature":0.7,"pith_summary":"The paper argues that vision-language models fail at composition because they generate or compare captions in an order—left to right, or all at once—that forces words to depend on the wrong things. It proposes COGT (Causally-Ordered Generative Training), which builds a causal graphical model from an off-the-shelf dependency parser and generates each word conditioned only on its ancestors in the dependency tree, its syntactic category, and the image features. This removes word-to-word correlations that are artifacts of sentence order rather than meaning. The paper reports that this training objective outperforms previous compositional methods on five benchmarks, including methods pre-trained on orders of magnitude more data.","feed_headline":"Dependency-guided decoding beats billion-scale vision-language models","feed_subtitle":"A tiny decoder trained only on COCO outdoes far larger generative VLMs on five composition benchmarks.","key_machinery":"The central object is a dependency-guided causal graphical model built from the caption's dependency tree, with word variables $W_j$, syntactic-type variables $S_j$, and visual variables $Z_k$. The workhorse is the disentangled factorization of Eq. (1), implemented by a small decoder with Dependency Guided Attention: each masked token representing a word attends only to visible tokens of its ancestors and to all visual features, with a distinct masked token for each of 45 syntactic categories. At inference, words are predicted in level-order of the dependency tree, so sibling adjectives like \"small\" and \"yellow\" are conditionally independent given \"head\".","core_discovery":"On the paper's own terms, the discovery is that the ordering of words in generation is not neutral: it is the main carrier of spurious correlation. Factorizing $P(W_1,\\dots,W_n \\mid Z)$ as $\\prod_j P(W_j \\mid \\mathrm{Pa}(W_j))$, with $\\mathrm{Pa}(W_j)$ the ancestors of $W_j$ in the dependency tree together with its syntactic type and all visual features, is a better trade-off between fully autoregressive and fully parallel prediction. COGT uses this factorization both at training and at inference, scoring candidate captions by the same product, and reports state-of-the-art results on ARO, SugarCrepe, VL-Checklist, ColorSwap, and the newly proposed FG-OVD benchmark while training on only about 100K COCO samples.","pith_inferences":["A natural testable extension is to learn the graph itself instead of trusting the parser, for example by training a lightweight head to predict parent sets; the paper does not try this.","The same level-order likelihood could be used for caption generation rather than only retrieval scoring; the paper frames COGT as a scoring method and does not explore decoding for image captioning.","The categorical masked tokens encode syntax as a prior; one could also condition on semantic roles or scene graphs, which might capture cross-branch dependencies the dependency parser misses.","Because the visual features are fixed and shared, COGT might combine additively with methods that improve the visual encoder, such as dense captions or diffusion-based negative mining; the paper leaves this combination unstated."],"forward_implications":["A generative objective that respects syntactic structure can substitute for expensive hard-negative mining in compositional training.","The recipe is data-efficient: training on about 100K COCO captions suffices to surpass methods trained on roughly a billion image-text pairs, suggesting the objective, not corpus size, carries much of the compositional signal.","The method transfers across backbones: freezing a CLIP, XVLM, or InstructBLIP encoder and attaching the causal decoder yields leading benchmark results in each case.","Composition-focused training does not have to degrade generic representation quality: linear probing on image classification benchmarks stays intact or improves.","Parser quality is a lever: using better dependency parsers improves downstream accuracy, so advances in parsing should translate directly into compositional gains."],"supporting_citations":[{"why":"Documents the bag-of-words failure of vision-language models and supplies the ARO benchmark plus the NegCLIP hard-negative baseline that COGT must beat.","marker":"(Yuksekgonul et al., 2023)"},{"why":"Introduces Cap and CapPa generative pre-training, the autoregressive and parallel factorizations COGT contrasts with, and the billion-pair training scale COGT surpasses.","marker":"(Tschannen et al., 2023)"},{"why":"Provides the Deep Biaffine dependency parser whose trees define the parent sets in the causal graph.","marker":"(Dozat & Manning, 2016)"},{"why":"Supplies the causal graphical model formalism and the disentangled factorization that Eq. (1) instantiates.","marker":"(Schölkopf et al., 2021)"},{"why":"Defines the 45 syntactic categories used as conditioning variables for the masked tokens.","marker":"(Silveira et al., 2014)"}],"fun_headline_variants":["Dependency tree fixes VLM bag-of-words blindness","Causal decoding beats billion-param vision-language models","Small model with causal order wins composition tests","Partial-order generation lifts VLM compositional skill","Dependency-guided generation outperforms huge VLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire gain depends on the assumption that the dependency parser's tree really lists the only text-to-text influences that matter: if true semantic dependencies cross branches, or the parser is wrong, the factorization is misspecified and the advantage could shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Dependency tree fixes VLM bag-of-words blindness","Causal decoding beats billion-param vision-language models","Small model with causal order wins composition tests","Partial-order generation lifts VLM compositional skill","Dependency-guided generation outperforms huge VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1251,"prompt_tokens":887,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":503,"tokens_out":364,"duration_ms":4638,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:22.134843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same COGT decoder and compare parser-based trees against gold-standard human dependency parses and against deliberately corrupted trees on a fixed benchmark; if COGT's gains are caused by correct causal structure, accuracy should track parse quality, and random or wrong trees should erase the gap.","supporting_citations":[],"review_version":1}