{"id":"d3eb3400-8b03-49e1-9847-adefaac1c9bf","arxiv_id":"2412.15588","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"NeSyCoCo improves compositional generalization in vision-language reasoning by combining LLM-generated programs, normalized differentiable predicate scores, and shared word-embedding-conditioned predicate functions.","lead":"NeSyCoCo is a neuro-symbolic system that uses an LLM to turn visual questions into logical programs, then grounds each predicate with a shared neural network guided by word embeddings. It reports top accuracy on ReaSCAN and CLEVR-CoGenT and shows partial zero-shot transfer to synonym concepts in a new CLEVR-SYN test.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLEVR-SYN may replace concepts only in programs, not in the natural-language queries, so the zero-shot 'novel concepts' claim may be untested end-to-end.","rationale":"The reader's weakest assumption correctly identifies the GloVe-based semantic transfer as a fragile part of the argument. My concern sharpens this: the actual evaluation protocol for CLEVR-SYN is ambiguous about whether novel concepts appear in the language input or only in the programs. If the latter, the paper's central 'linguistic lexical variety' claim is not tested end-to-end; the LLM program generator is bypassed and only the predicate-grounding module is evaluated. This is a load-bearing issue because the abstract and Section 4.4 tie the benchmark to 'faced with new concepts,' which in a neuro-symbolic pipeline should include the language-to-program component. The second part of the concern is that the paper reports correlation scores instead of per-substitution accuracies, and its own Table 6 shows several near-zero correlations, so the aggregate 73.4% hard-split accuracy cannot be interpreted as uniformly robust generalization. These issues do not invalidate the ReaSCAN and CLEVR-CoGenT results, which have independent support from ablations and statistical tests, so a conditional verdict remains appropriate. The concrete test would settle whether the zero-shot claim should be downgraded or restricted to the grounding module only.","tokens_in":14583,"tokens_out":3916,"duration_ms":39155,"concrete_test":"Inspect the released CLEVR-SYN generation script to determine whether the regex replacement applies to the question text as well as the program text. Then run NeSyCoCo end-to-end on CLEVR-SYN under two conditions: (a) original queries paired with the synonym-replaced programs, and (b) synonym-replaced queries paired with either original or replaced programs, with per-synonym accuracy and frequency reported. If accuracy in condition (b) is substantially lower than the reported 92.1/81.2/73.4, or if per-synonym accuracy is near chance for cube-box, sphere-ball, or brown-chocolate, the central claim of robust zero-shot generalization to novel concepts is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims 'robust performance with novel concepts in the CLEVR-SYN benchmark.' Section 4.4 describes CLEVR-SYN as replacing concepts 'in programs' via regular-expression search on CLEVR programs, but the text never states that the corresponding natural-language queries are also modified. In the NeSyCoCo pipeline, the language-to-program module (LLM) reads the query and generates the program; if only the ground-truth programs are rewritten while queries retain the original words, the LLM never encounters the novel synonyms. The evaluation then isolates whether the shared predicate FFN can ground unseen GloVe embeddings, but it does not test whether the full system understands novel linguistic expressions. This matters because 'handling linguistic lexical variety' is presented as a central contribution. Additionally, Table 6 reports Pearson correlations between original and replaced predicate scores rather than per-substitution accuracy; the low correlations for cube-box (0.06), sphere-ball (-0.02), and brown-chocolate (0.20) could translate into large accuracy losses on queries containing those predicates, losses that are hidden in the aggregate hard-split number of 73.4%. The paper's own analysis concedes that only 9 of 15 substitutions yield correlations above 0.6, so the robustness claim is materially weaker than the headline suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NeSyCoCo, a neuro-symbolic vision-language reasoning framework that combines an LLM-based language-to-program module, dependency parsing of input queries, shared predicate networks conditioned on frozen word embeddings, and normalized soft composition operations. The method is evaluated on ReaSCAN, CLEVR-CoGenT, CLEVR, CLEVR-Ref/Puzzle/RPM, and a newly introduced CLEVR-SYN benchmark. The central claims are state-of-the-art accuracy on ReaSCAN and CLEVR-CoGenT, and robust zero-shot generalization to novel but semantically related concepts on CLEVR-SYN, with reported three-run means and standard deviations.","tokens_in":14847,"tokens_out":4311,"duration_ms":38051,"significance":"If the reported results hold, the paper makes a useful contribution: replacing per-predicate networks with a shared FFN over distributed word embeddings is a simple and plausible mechanism for grounding unseen predicates, and the normalized soft composition operations address a known weakness in LEFT's use of unbounded logits. The availability of code, three-run statistics, ablations, and an analysis of score distributions are strengths. However, the significance is currently limited by evaluation gaps: the CLEVR-SYN benchmark appears to modify programs rather than natural-language queries, the CLEVR extension tables report NeSyCoCo only with ground-truth programs, and the state-of-the-art claim is not supported on every split. These issues affect the paper's headline claims and should be addressed before the results can be fully assessed.","major_comments":[{"comment":"The CLEVR-SYN benchmark is described as replacing concepts 'in programs' via regular-expression search, but the paper never states that the corresponding natural-language queries are also modified. In the NeSyCoCo pipeline, the language-to-program module reads the query and generates the program; if only ground-truth programs are rewritten while queries retain original words, the full system never encounters the novel synonyms. The evaluation then isolates whether the shared predicate FFN can ground unseen GloVe embeddings, but it does not test whether the system understands novel linguistic expressions. Because 'Handling Language Variety' is a central contribution and the abstract claims 'robust performance with novel concepts,' this gap is load-bearing. The authors should either modify the queries as well and run the full pipeline, or explicitly reframe the claim as a module-level test of predicate grounding.","section":"Section 4.4, CLEVR-SYN construction and Table 7"},{"comment":"For the CLEVR extension tasks, NeSyCoCo is reported only as 'NeSyCoCo + GT programs' (100% on Ref, 95% on Puzzles, 100% on RPM), while LEFT is reported both with GT programs and with LLM programs. This comparison does not support a claim that the full NeSyCoCo pipeline, including its language-to-program component, improves on LEFT for these tasks. Without a 'NeSyCoCo + LLM programs' row, the reader cannot tell whether the 95% Puzzle result is due to the proposed reasoning/grounding changes or to the use of ground-truth programs. Please report the full-pipeline results on these extensions or restrict the claim to the GT-program setting.","section":"Section 4.3, Table 5"},{"comment":"The abstract claims 'state-of-the-art results on the ReaSCAN and CLEVR-CoGenT compositional generalization benchmarks,' but the reported numbers do not support this on every split. On ReaSCAN, Syntax Guided Transformer achieves 97.3% on A2 and 99.6% on A1 and A3, compared with NeSyCoCo's 94.1%, 99.1%, and 98.5% respectively. On CLEVR-CoGenT Split A, MDETR achieves 99.7% versus NeSyCoCo's 99.6%. The claim should be qualified to the average ReaSCAN accuracy and CLEVR-CoGenT Split B, or the authors should discuss why competitiveness on some splits is sufficient for the stated claim.","section":"Section 4.2, Tables 2 and 3"},{"comment":"The per-substitution Pearson correlations in Table 6 are near zero or negative for several substitutions: cube-box 0.06, sphere-ball -0.02, red-burgundy 0.15, and brown-chocolate 0.20. The paper acknowledges that only 9 of 15 substitutions have correlation above 0.6, but the aggregate hard-split accuracy of 73.4% may hide large failures on queries containing these low-correlation predicates. To support the robustness claim, please report accuracy separately for queries involving low-correlation substitutions, or provide per-substitution accuracy in addition to the correlation analysis.","section":"Section 4.4, Table 6 and hard-split analysis"}],"minor_comments":[{"comment":"The row labeled 'w/o Emb. w/o Soft Reasoner' has exactly the same values and standard deviations as the row labeled 'LEFT†.' If this is intentional, the table and caption should explain why the two rows coincide; otherwise it appears to be a copy-paste error that undermines the readability of the ablation study.","section":"Table 2"},{"comment":"The introduction states that NeSyCoCo 'achieves state-of-the-art results on the ReaSCAN compositional generalization and CLEVR-Puzzle benchmarks,' but the experiments do not report a 'CLEVR-Puzzle' benchmark as a single named dataset; they report CLEVR-Puzzle as one of the CLEVR extensions in Table 5. The wording should be aligned with the experimental sections.","section":"Introduction and Section 4.2"},{"comment":"The phrase 'improves composition and interoperability' in the discussion of soft composition appears to be a typo for 'interpretability'; please correct it.","section":"Section 3.3"},{"comment":"The dataset name 'CLVER' appears twice in Table 9; it should be 'CLEVR.' There are also several typos such as 'CLVER-CoGenT' in the same table.","section":"Table 9 and throughout"},{"comment":"The Ethical Statement appropriately acknowledges that predicate generalization depends on the choice of distributed representation, but this limitation should be connected more explicitly to the CLEVR-SYN results in the main text, since the low correlations in Table 6 are instances of that dependence.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper's core architecture is plausible and the ReaSCAN/CoGenT results are interesting, but the CLEVR-SYN evaluation as described does not test the full system's handling of novel linguistic expressions, and the CLEVR extension results are reported only with ground-truth programs. These issues are fixable with additional experiments or reframing, but they are load-bearing for the paper's stated contributions. I would recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible LEFT follow-up, not a breakthrough. The three modifications are concrete and the ablations show each component earns its keep. Dependency-parsed prompting helps most on the complex C splits, the shared embedding-conditioned predicate FFN beats predicate-specific FFNs, and normalizing predicate scores before soft composition fixes a real scale problem. The numbers are three-run means with standard deviations, and the main comparisons are against external baselines. That is honest, reproducible work, and CLEVR-SYN is a useful new evaluation set even with the flaws below.\n\nNow the soft spots. First, the abstract's SOTA claim is too broad. On ReaSCAN, NeSyCoCo beats LEFT on every split but loses to Syntax-Guided Transformer on A1, A2, and A3. On CLEVR-CoGenT split A it is not SOTA either. Saying 'state-of-the-art on average' or 'on split B' would be accurate; the blanket claim overreaches. Second, the CLEVR extension table evaluates NeSyCoCo only with ground-truth programs, so we never see how the LLM program generator behaves on those tasks. That is a missing comparison, not a fatal flaw. Third, and most important, I think the stress-test note is right about CLEVR-SYN. Section 4.4 says concepts in programs are replaced via regular-expression search on CLEVR programs, but never says the corresponding natural-language queries are modified. If only the programs change, the LLM never encounters the novel synonyms; the test measures the shared predicate FFN's ability to ground unseen GloVe embeddings, not the system's ability to understand novel words. Since handling linguistic lexical variety is advertised as a central contribution, this distinction matters. To the authors' credit, Table 6 and the surrounding analysis admit that only 9 of 15 substitutions have correlation above 0.6, and cube-box and sphere-ball are near zero. That is a genuinely honest self-assessment, which makes the 'robust performance with novel concepts' claim hard to defend as stated. The benchmark should either rewrite the queries end-to-end or be repositioned as a predicate-grounding probe.\n\nWho is this for? People working on neuro-symbolic VQA and compositional generalization who want a practical recipe and a new evaluation probe. It deserves a serious referee. The engineering is clean, the ablations are informative, and the CLEVR-SYN protocol issue is fixable with clarification or a small extension. I would accept it for review and request the missing LLM-program numbers and an exact description of what is replaced in CLEVR-SYN.","headline":"NeSyCoCo is a credible incremental extension of LEFT with a useful new benchmark, but the CLEVR-SYN protocol likely bypasses the language-to-program module, so the zero-shot 'novel concepts' claim is weaker than the abstract suggests.","tokens_in":15384,"tokens_out":3743,"would_cite":true,"duration_ms":30121,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neuro-symbolic composer that lets ordinary word embeddings stand in for logic predicates reaches new state-of-the-art results on compositional generalization benchmarks, with an explicit soft-logic executor handling the composition.","keywords":["compositional generalization","neuro-symbolic reasoning","vision-language reasoning","dependency parsing","word embeddings","soft logic composition","CLEVR benchmarks","ReaSCAN"],"falsifier":"Run the trained model on CLEVR-SYN after replacing GloVe with a scrambled-but-fixed embedding table: if hard-split accuracy stays near the reported 73.4 percent, the mechanism does not depend on semantic structure in the embeddings, while a drop toward the LEFT baseline would show the zero-shot claim is bounded by embedding quality. A lighter test is to restrict the hard split to synonym pairs whose GloVe cosine similarity is below 0.4 and check whether accuracy falls to chance.","tokens_in":14377,"feed_emoji":"🧩","tokens_out":7127,"duration_ms":62141,"temperature":0.7,"pith_summary":"NeSyCoCo sets out to make neuro-symbolic visual reasoning compositional without a fixed dictionary of predicates. It uses an LLM to turn a query into a logical program, adds a dependency parse of the query to the prompt, and represents every predicate by a frozen word embedding rather than a learned one-hot symbol. A single shared network scores objects and relations from the embedding and visual features, and normalized scores are combined by soft differentiable logic. On standard generalization benchmarks the paper reports the best average grounding accuracy on ReaSCAN and the best split-B accuracy on CLEVR-CoGenT, and its new CLEVR-SYN benchmark shows it can ground some unseen synonyms in zero shots. The reason to care is that this points toward agents that combine familiar concepts with new words instead of needing retraining.","feed_headline":"Predicate embeddings lift compositional reasoning to 97.5 percent","feed_subtitle":"A soft-logic neuro-symbolic composer beats prior baselines on ReaSCAN and CLEVR-CoGenT and grounds unseen synonyms zero-shot.","key_machinery":"The load-bearing mechanism is a single shared multimodal feed-forward network that consumes two inputs: a predicate's distributed word representation (a 300-dimensional GloVe vector) and the visual feature of an object or object pair, and outputs a sigmoid-normalized score in [0,1]. Because the predicate enters as a vector rather than as an index into a table of per-predicate networks, synonyms and paraphrases with similar embeddings activate the same learned function. The accompanying soft logic executor then combines these normalized scores using multiplication for 'and', max for 'exists', min for 'forall', min-max linear normalization for 'iota', and a sum for 'count', replacing the raw-logit min and softmax operations of the earlier framework. Dependency parsing of the query, fed to the LLM as extra context, makes the generated symbolic programs line up with the query's grammatical structure.","core_discovery":"The paper's central claim is that a neuro-symbolic reasoner can generalize compositionally without a predefined predicate set by letting language itself define the predicates. Every predicate token from the LLM-generated program is mapped through an off-the-shelf word embedding, concatenated with visual features, and scored by one shared feed-forward network; that normalized score replaces raw logits. Composition is then done with soft operations: multiplication for conjunction, max for existential and relational 'and', min for universal quantification, bounded linear normalization for variable assignment, and sums for counting. The paper reports 97.5 percent average grounding accuracy on ReaSCAN against 92.3 for the previous LEFT baseline, 78.8 percent on CLEVR-CoGenT split B against 76.2, and 73.4 percent on the hard split of its new CLEVR-SYN synonym benchmark against 49.5, with a paired t-test p=0.0026 showing the soft reasoner contributes significantly. The paper itself qualifies this with its own correlation table: only 9 of 15 synonym pairs score above 0.6, so the claim is that generalization works as far as the frozen embeddings preserve meaning.","pith_inferences":["Because the shared network sees predicate embeddings rather than one-hot symbols, the same architecture could in principle be retrained with any embedding family; the paper's own appendix data already suggest contextual encoders behave differently from static ones.","A natural stress test would swap the benchmark's synonym pairs for antonyms or confusable attributes with similar embeddings, such as left versus right, since a model that relies on embedding proximity could fail loudly there.","The CLEVR-SYN correlation analysis suggests a concrete design rule: zero-shot transfer via frozen embeddings should only be trusted when the embedding cosine similarity exceeds roughly 0.4, matching the break-off point in the paper's Figure 5.","Because the perception module is inherited from the baseline, the improvements are measured at the reasoning layer; combining this predicate-embedding mechanism with a stronger visual encoder is an untested combination."],"forward_implications":["If the ReaSCAN results hold, the largest gains appear in the C splits, which test relative clauses and 'that is' constructions, where normalized predicate scores and dependency-parsed prompts matter most.","On CLEVR-CoGenT split B, the method moves past the 76.2 percent ceiling shared by both LEFT and MDETR, suggesting the soft reasoner rather than the visual encoder drives the improvement.","Novel-concept grounding works zero-shot for synonyms whose embeddings are close to the training predicate and fails for the others, making embedding choice part of the generalization story.","Soft, normalized composition removes the scale mismatch between predicate scores, so conjunctions can be trained jointly instead of being dominated by the minimum of raw logits.","The CLEVR-SYN benchmark itself provides a reusable test for whether a neuro-symbolic method can ground unseen but similar concepts without retraining."],"supporting_citations":[{"why":"Provides the LEFT framework this method builds on: LLM-generated logical programs, the predicate-scoring architecture replaced here, and the baseline numbers compared throughout.","marker":"Hsu et al. 2024"},{"why":"Supplies the CLEVR benchmark and its programs, which train the predicates and seed CLEVR-SYN and the CoGenT splits.","marker":"Johnson et al. 2017a"},{"why":"Supplies the ReaSCAN benchmark whose seven splits define the compositional generalization test and the grounding metric.","marker":"Wu et al. 2021"},{"why":"Provides the frozen GloVe word vectors used to represent every predicate in the shared network, the core of the zero-shot synonym mechanism.","marker":"Pennington, Socher, and Manning 2014"},{"why":"Provides the spaCy dependency parse of each query that is concatenated into the LLM prompt for program generation.","marker":"Honnibal and Montani 2017"},{"why":"Supplies Mask R-CNN object detection whose bounding-box features form the perception input to predicate scoring.","marker":"He et al. 2017"},{"why":"Supplies the LLaMA-3.1 70B backbone used to translate queries with dependency context into symbolic programs.","marker":"Dubey, Jauhri, and Others 2024"},{"why":"Provides MDETR, the end-to-end baseline compared on CLEVR and CLEVR-CoGenT.","marker":"Kamath et al. 2021"}],"fun_headline_variants":["Soft logical composition generalizes better on vision-language tasks","Language-defined predicates enable zero-shot synonym grounding","Neuro-symbolic composer beats baselines without predefined predicates","97.5% accuracy via soft predicate scoring, no hard-coded logic","LLM-suggested predicates ground unseen synonyms via word embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen off-the-shelf word embedding carries enough of each predicate's meaning that a network trained only on the original vocabulary will also give sensible scores to unseen synonyms; the paper's own CLEVR-SYN correlations show this is only partly true, with cube/box at 0.06 and sphere/ball at -0.02.","fun_headline_variants_meta":{"raw":{"variants":["Soft logical composition generalizes better on vision-language tasks","Language-defined predicates enable zero-shot synonym grounding","Neuro-symbolic composer beats baselines without predefined predicates","97.5% accuracy via soft predicate scoring, no hard-coded logic","LLM-suggested predicates ground unseen synonyms via word embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1295,"prompt_tokens":988,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":604,"tokens_out":307,"duration_ms":3664,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:17:13.006130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained model on CLEVR-SYN after replacing GloVe with a scrambled-but-fixed embedding table: if hard-split accuracy stays near the reported 73.4 percent, the mechanism does not depend on semantic structure in the embeddings, while a drop toward the LEFT baseline would show the zero-shot claim is bounded by embedding quality. A lighter test is to restrict the hard split to synonym pairs whose GloVe cosine similarity is below 0.4 and check whether accuracy falls to chance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LEFT framework this method builds on: LLM-generated logical programs, the predicate-scoring architecture replaced here, and the baseline numbers compared throughout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the frozen GloVe word vectors used to represent every predicate in the shared network, the core of the zero-shot synonym mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the spaCy dependency parse of each query that is concatenated into the LLM prompt for program generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MDETR, the end-to-end baseline compared on CLEVR and CLEVR-CoGenT."}],"review_version":1}