{"id":"0f527849-4392-4ce3-9251-3ca358deb5da","arxiv_id":"2412.13636","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Large vision-language models score below 50 percent on a new cross-level compositional consistency benchmark, and a progressive meta-optimization training framework improves consistency on several VQA and TVG baselines.","lead":"Vision-language models, even very large ones, are inconsistent when asked to generalize to novel word and phrase combinations at several levels of complexity at once. This paper introduces a progressive training method and a new visual question answering benchmark, GQA-CCG, that improve such consistency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The longest-phrase-length complexity proxy in Eq. (1) is unvalidated against the actual composition-novelty structure of GQA-CCG, so the claimed simple-to-complex mechanism—and the consistency gains attributed to it—may be an artifact of question-length reweighting.","rationale":"I agree with the reader that the complexity proxy is the weakest load-bearing assumption. The headline empirical result—CFR+MLO lifting consistency from 46.46% to 49.27% while keeping GQA test-dev accuracy stable—could still hold even if the proxy is imperfect, but the paper's central explanatory claim (progressive simple-to-complex consistency learning) would be unsupported if longest-phrase length is not a valid complexity measure. The dataset-count ambiguity noted by the reader is real and should be checked against the released code, but it is secondary to the conceptual validity of the curriculum. Given that the reported gains are consistent across five VQA baselines and one TVG baseline, and that the proxy could be empirically validated or replaced, the appropriate verdict remains CONDITIONAL; my read does not change the reader's verdict.","tokens_in":14957,"tokens_out":15016,"duration_ms":144446,"concrete_test":"Use CFR or MAC as the backbone; construct three alternative validation-set partitionings with the same K and size distribution as Eq. (1): (a) parse-tree depth or number of composition nodes as L(Q); (b) true novelty level of each training sample's pp/pw/ww compositions with respect to the training set; (c) a random partition with no complexity ordering. Train the identical MLO procedure under each and evaluate Cons and per-level accuracy on GQA-CCG. If (a) or (b) do not preserve the reported gains, or if (c) reproduces them, the longest-phrase-length proxy is load-bearing; if gains persist only with the proxy-based ordering, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 'Validation Set Construction' defines L(Q) as the length of the longest phrase of Q, and Eq. (1) partitions the training set into validation sets Dv_i ordered by cumulative counts of L(Q). The paper asserts that 'compositional complexity is proportional to the length of the longest phrase in a question,' but this identification does the causal work: the multilevel optimization in Eq. (3) is supposed to make the model fit progressively harder compositions. GQA-CCG's difficulty, however, is defined by novel combinations of seen phrases and words (Eq. (8)), not by surface length. A question containing the novel phrase-phrase composition 'golden dog'+'white cat' can have L(Q)=2, while a long relative-clause question can have a much larger L(Q) with no novel composition. If L(Q) only proxies sentence length, the ordered validation sets are not teaching compositions from simple to complex; the meta-weight-nets are then merely reweighting by question length, and the consistency gains in Table 1 would not support the paper's progressive-composition explanation. Table 4 shows only that the S→C order beats C→S; it does not establish that the underlying measure tracks compositional complexity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new evaluation criterion for compositional generalization in vision-and-language models, namely consistency across phrase-phrase, phrase-word, and word-word levels of novel compositions. It proposes a meta-learning framework (MLO) that partitions the training set into validation sets ordered by an approximate compositional-complexity measure (longest phrase length), uses separate meta-weight-nets to reweight samples in each validation set, and optimizes the model and the meta-weight-nets in a multilevel, simple-to-complex fashion. The authors construct a new VQA benchmark, GQA-CCG, with 8,702 triplets of questions at the three composition levels, define a triplet-level consistency metric, and report experiments on five VQA baselines and one temporal-video-grounding baseline, along with ablations and IID generalization checks on GQA.","tokens_in":15229,"tokens_out":5847,"duration_ms":55268,"significance":"If the results are trustworthy, the paper identifies a genuinely new and measurable failure mode of current vision-and-language models: even large pretrained models achieve only around 40% consistency across composition levels, and the proposed MLO framework consistently improves both consistency and per-level accuracy on multiple baselines while retaining IID accuracy. The released GQA-CCG dataset and the consistency metric are concrete contributions that could support future work. However, the significance is currently conditional: the dataset statistics contain an unresolved internal inconsistency, the central complexity proxy is unvalidated against the actual composition-novelty structure, and the reported gains lack error bars or significance testing.","major_comments":[{"comment":"The reported dataset totals are internally inconsistent. The text states that DCCG consists of 8,702 triplets and 18,983 samples, but 8,702 × 3 = 26,106, and the three per-level counts (5,125 + 8,102 + 5,756) sum exactly to 18,983. If a triplet always contains one phrase-phrase, one phrase-word, and one word-word sample, the triplet count and the per-level counts cannot differ as reported. Since Eq. (9) computes consistency at the triplet level and Table 1 reports per-level accuracy, the reader cannot determine whether the consistency metric and the per-level accuracies are computed over the same data. Please provide an exact breakdown of unique samples and triplet memberships, or correct the arithmetic, so the benchmark statistics are auditable.","section":"GQA-CCG Dataset (paragraph after Manual Review)"},{"comment":"The paper asserts that 'compositional complexity is proportional to the length of the longest phrase in a question' and uses this to order validation sets from simple to complex. No evidence is given that L(Q) tracks the levels of novel compositions defined in Eq. (8). A question can contain a short phrase-phrase novel composition such as 'golden dog' + 'white cat' with a small L(Q), while a long relative-clause question can have a large L(Q) with no novel composition. The multilevel optimization in Eq. (3) is only meaningful as a simple-to-complex curriculum if the validation sets actually increase in compositional difficulty. Please validate the proxy (e.g., by reporting the distribution of L(Q) across the three GQA-CCG levels, or by showing that the S→C advantage in Table 4 persists when validation sets are built from explicit composition-novelty annotations rather than phrase length).","section":"Validation Set Construction (Eq. (1) and surrounding text)"},{"comment":"All experimental results are single-run; no error bars, multiple seeds, or significance tests are reported. The headline consistency improvements are modest in absolute terms (e.g., MAC from 30.82% to 34.10%; CFR from 46.46% to 49.27%), and for models trained with different random seeds such differences could plausibly arise from run-to-run variance. Please report mean ± standard deviation over at least three seeds, or provide bootstrap confidence intervals for the main tables, so the claimed improvements can be distinguished from noise.","section":"Experimental results (Tables 1-4)"},{"comment":"The nested formulation in Eq. (3) is notationally confusing: the loops are written as if each LOOP is a constrained argmin over one meta-weight-net, but the actual training alternates parameter optimization and meta optimization, and the meta-weight-nets are updated sequentially. The paper should clarify the precise algorithm (e.g., pseudocode or a clear step-by-step description) and state how the approximation via the implicit function theorem interacts with the sequential updates. This is important for reproducibility, especially because the supplementary details are referenced but not included in this version.","section":"Multilevel Optimization (Eq. (3))"}],"minor_comments":[{"comment":"In the second paragraph, 'we use the the train balanced split' contains a duplicated 'the'.","section":"GQA-CCG Dataset"},{"comment":"In Automatic Generation, 'we fill associated infos' should be 'we fill the associated information'.","section":"Sample Generation Pipeline"},{"comment":"The model names 'LLaV A-1.5-Xtuner' and 'LLaV A-1.6' should be written without the space, as 'LLaVA-1.5-Xtuner' and 'LLaVA-1.6'.","section":"Table 1 and Experimental Settings"},{"comment":"The conclusion says the dataset enables 'qualitative evaluation', which appears to be a typo for 'quantitative evaluation'.","section":"Conclusion"},{"comment":"The notation 'LOOPK : ...' followed by 'LOOP2 : s.t. ...' is hard to read as a nested optimization; consider rewriting as a sequence of explicit optimization problems or an algorithmic loop.","section":"Eq. (3)"},{"comment":"The phrase 'varies in parameters' should be 'varying in parameter count' or 'with different numbers of parameters'.","section":"Compositional Generalization Performance"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is interesting, but the dataset arithmetic inconsistency and the unvalidated complexity proxy directly affect the credibility of the headline claims. These are fixable, but they must be resolved before publication. I would also strongly encourage the authors to add multiple-seed results; without them, the 2-3 point consistency gains are hard to evaluate. The paper should not be accepted in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper adds a useful and genuinely new evaluation axis to the compositional generalization literature—consistency of performance across phrase-phrase, phrase-word, and word-word novel compositions—plus a new benchmark (GQA-CCG), a consistency metric, and a meta-learning framework. The empirical story is mostly solid: five VQA baselines and one TVG baseline all improve on both consistency and per-level accuracy when the framework is applied, and the gains are not at the expense of IID accuracy on GQA. I'd give the authors credit for the dataset construction: they filter novel phrase-phrase compositions, manually annotate a seed set, generate derived questions with GPT-3.5, then postprocess and manually review. That's careful work.\n\nThe main soft spot is the claim that 'compositional complexity is proportional to the length of the longest phrase in a question.' That assumption is doing real work in the validation set construction and the simple-to-complex curriculum. But the dataset defines difficulty by novel combinations of seen words/phrases, not by surface length. A question with a novel phrase-phrase composition like 'golden dog' + 'white cat' can have a short longest phrase, while a long relative-clause question may contain no novel composition at all. So the ordering used to train the meta-weight-nets may be capturing sentence length rather than compositional complexity. The ablation showing S→C beats C→S is consistent with the method's story, but it doesn't validate the proxy. It's worth a pointed question to the authors, though it doesn't sink the paper—the consistency gains could come from sample reweighting, but they are still there.\n\nA second, smaller issue: the dataset numbers don't add up in an obvious way. The text says 8702 triplets and then reports 5125 (pp), 8102 (pw), and 5756 (ww) samples, which sum to 18983. Three times 8702 is 26106. Either the per-level counts are not all triplets, or some triplets are incomplete. The paper doesn't explain this. Also, no error bars or multiple seeds anywhere, so the reported improvements (e.g., CFR consistency 46.46→49.27) are point estimates.\n\nBottom line: this is a useful paper for anyone working on compositional generalization or VQA evaluation. The consistency metric and dataset are the strongest contributions; the training framework is plausible but rests on a questionable complexity proxy. I'd send it to a serious referee, expecting the proxy and the dataset arithmetic to be fixed or defended. It deserves engagement.","headline":"Introduces a genuinely new evaluation axis—cross-level compositional consistency—with a new benchmark and a model-agnostic training framework; the empirical gains are consistent across baselines, but the complexity proxy and dataset numbers need scrutiny.","tokens_in":15738,"tokens_out":2823,"would_cite":true,"duration_ms":24211,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-and-language models are inconsistent across levels of novel composition, and a meta-learning curriculum improves that consistency without sacrificing accuracy.","keywords":["compositional generalization","consistency","visual question answering","meta-learning","multilevel optimization","sample weighting","temporal video grounding","GQA-CCG"],"falsifier":"A direct test would be to rerun the framework with the validation sets reordered by a different complexity proxy, such as human difficulty ratings, parse-tree depth, or the number of novel concept pairs. If the simple-to-complex ordering no longer outperforms complex-to-simple or simultaneous weighting, the claimed curriculum mechanism is not what drives the consistency gains.","tokens_in":14767,"feed_emoji":"🧩","tokens_out":7557,"duration_ms":65666,"temperature":0.7,"pith_summary":"Existing vision-and-language models are good at some novel compositions and bad at others, but the paper argues that the deeper problem is consistency: when a model handles a hard phrase-phrase composition, it should also handle the simpler phrase-word and word-word compositions derived from it. Most current models, including large pretrained ones, answer barely 40% of such derived triplets consistently, even when they score well on individual levels. The paper introduces GQA-CCG, a VQA dataset of 8,702 triplet questions, and a consistency metric that counts triplets answered entirely correctly. It then proposes a meta-learning framework, MLO, that trains a model by progressively fitting validation sets of increasing compositional complexity; across five VQA baselines and one temporal-video-grounding baseline, MLO improves both per-level accuracy and consistency, with the strongest result (CFR+MLO) raising consistency from 46.46% to 49.27% while keeping overall accuracy above 74% and GQA test-dev accuracy stable.","feed_headline":"Meta-learning lifts cross-level composition consistency","feed_subtitle":"Five VQA baselines and a video grounding model improve on GQA-CCG without sacrificing standard accuracy.","key_machinery":"The machinery is a multilevel optimization loop over K validation sets $\\{\\mathcal{D}_v^i\\}_{i=1}^K$, partitioned from the training set by an approximate complexity score: the length of the longest phrase in the question, with Eq. (1) balancing similar complexity against similar set size. Each validation set has its own meta-weight-net, a small fully connected network with sigmoid output, that takes the question embedding and outputs a sample weight $w_d$. During parameter optimization the model minimizes the weighted training loss $\\mathcal{L}_t$; during meta optimization the weight networks are updated sequentially, from the simplest to the most complex validation set, so that later weight networks fit sets whose harder compositions presuppose the earlier ones. Reversing the order (complex to simple) yields almost no gain, which the paper reads as evidence that the ordering, not the reweighting alone, carries the effect.","core_discovery":"The paper's central claim is that cross-level consistency is a real, separately measurable property of compositional generalization, distinct from per-level accuracy, and that it can be improved by explicit simple-to-complex curriculum. On its new GQA-CCG benchmark, models with up to 37B parameters achieve only about 40% consistency, showing that even models that master many novel compositions do not reliably master the simpler compositions that are their building blocks. The proposed MLO framework divides the training set into K validation sets of increasing complexity, assigns each a meta-weight-net that produces per-sample weights, and optimizes the model and the weight networks in a nested loop so that the validation sets are fit in order from simple to complex. The results show consistent gains in accuracy and consistency across attention-based, graph-based, module-network, and pretrained VQA baselines, and across temporal video grounding, with no loss on the IID GQA test set.","pith_inferences":["The consistency metric could serve as a probe for systematicity in large language and vision-language models: a classifier over incorrect triplet patterns might reveal whether failures concentrate at the phrase-word bridge or the word-word base, giving a diagnostic that accuracy alone cannot provide.","Because the complexity proxy is longest-phrase length, an immediate test is whether other proxies, such as parse-tree depth or number of novel concept pairs, reproduce the ordering effects; if gains persist under different proxies, the curriculum mechanism is robust, and if not, the gains may come from reweighting rather than from simple-to-complex teaching.","A natural extension is to apply the same nested-loop reweighting to generative tasks, measuring consistency of instructed outputs rather than correctness of multiple-choice answers.","The near-40% ceiling for large pretrained models suggests that scale and broad pretraining do not by themselves buy cross-level consistency; the paper's results point to training-time curricula as a cheaper complement."],"forward_implications":["If consistency is a distinct axis of compositional generalization, evaluations should report triplet-level consistency alongside per-level accuracy, since models can score high on individual levels while failing the consistency test.","The simple-to-complex ordering is load-bearing: swapping it for complex-to-simple removes the gains, so training curricula for compositional tasks should respect the derivation order of compositions.","The framework transfers across model families and tasks, so consistency improvement is not tied to a single architecture or to visual question answering alone.","The IID results on GQA indicate that on these baselines the consistency gains do not trade off against standard in-distribution accuracy, which is the failure mode of many OOD-oriented methods."],"supporting_citations":[{"why":"GQA, the source dataset whose splits are reorganized and re-annotated into the GQA-CCG benchmark.","marker":"(Hudson and Manning 2019)"},{"why":"benepar parser that extracts the phrases used for complexity scoring and novelty detection.","marker":"(Kitaev, Cao, and Klein 2019)"},{"why":"Meta-weight-net, the per-sample weighting module that each validation set's network is adapted from.","marker":"(Shu et al. 2019)"},{"why":"Implicit-differentiation approximation that makes the meta-gradient computation tractable in the multilevel loop.","marker":"(Lorraine, Vicol, and Duvenaud 2020)"},{"why":"CFR, the strongest VQA baseline; its +MLO result anchors the main consistency improvement from 46.46% to 49.27%.","marker":"(Nguyen et al. 2022)"},{"why":"MAC, the attention-based baseline used in the validation-set ordering ablations.","marker":"(Hudson and Manning 2018)"},{"why":"Charades-CG, the temporal video grounding benchmark used to show the framework transfers outside VQA.","marker":"(Li et al. 2022)"},{"why":"Cognitive-load rationale cited for why simple-to-complex ordering, rather than the reverse, should help learning.","marker":"(Plass, Moreno, and Brünken 2010)"}],"fun_headline_variants":["GQA-CCG benchmark reveals models' cross-level consistency gap","Meta-learning curriculum improves composition consistency","37B-parameter models only 40% on consistency benchmark","Simple-to-complex meta-learning lifts cross-level consistency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the length of the longest phrase in a question is a valid proxy for how hard it is to compose the concepts in that question; if that ordering is wrong, the framework is reweighting samples rather than actually teaching compositions from simple to complex.","fun_headline_variants_meta":{"raw":{"variants":["GQA-CCG benchmark reveals models' cross-level consistency gap","Meta-learning curriculum improves composition consistency","37B-parameter models only 40% on consistency benchmark","Simple-to-complex meta-learning lifts cross-level consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2358,"prompt_tokens":949,"completion_tokens":1409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1345}},"tokens_in":565,"tokens_out":1409,"duration_ms":12034,"temperature":1.0,"reasoning_tokens":1345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:56:35.657913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to rerun the framework with the validation sets reordered by a different complexity proxy, such as human difficulty ratings, parse-tree depth, or the number of novel concept pairs. If the simple-to-complex ordering no longer outperforms complex-to-simple or simultaneous weighting, the claimed curriculum mechanism is not what drives the consistency gains.","supporting_citations":[{"cited_title":"A.; and Manning, C","cited_arxiv_id":null,"evidence_quote":"GQA, the source dataset whose splits are reorganized and re-annotated into the GQA-CCG benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"benepar parser that extracts the phrases used for complexity scoring and novelty detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Meta-weight-net, the per-sample weighting module that each validation set's network is adapted from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Implicit-differentiation approximation that makes the meta-gradient computation tractable in the multilevel loop."},{"cited_title":"X.; Do, T.; Tran, H.; Tjiputra, E.; Tran, Q","cited_arxiv_id":null,"evidence_quote":"CFR, the strongest VQA baseline; its +MLO result anchors the main consistency improvement from 46.46% to 49.27%."},{"cited_title":"A.; and Manning, C","cited_arxiv_id":null,"evidence_quote":"MAC, the attention-based baseline used in the validation-set ordering ablations."},{"cited_title":"L.; Moreno, R.; and Br \\\"u nken, R","cited_arxiv_id":null,"evidence_quote":"Cognitive-load rationale cited for why simple-to-complex ordering, rather than the reverse, should help learning."}],"review_version":1}