NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization

Danial Kamali; Elham J. Barezi; Parisa Kordjamshidi

REVIEW 4 major objections 5 minor 52 references

NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neuro-symbolic composer that lets ordinary word embeddings stand in for logic predicates reaches new state-of-the-art results on compositional generalization benchmarks, with an explicit soft-logic executor handling the composition.

desk verdict NeSyCoCo is a credible incremental extension of LEFT with a useful new benchmark, but the CLEVR-SYN protocol likely bypasses the language-to-program module, so the zero-shot 'novel concepts' claim is weaker than the abstract suggests. read the letter →

arxiv 2412.15588 v1 pith:BB5X5LM5 submitted 2024-12-20 cs.CL

Danial Kamali , Elham J. Barezi , Parisa Kordjamshidi This is my paper · ORCID

classification cs.CL

keywords compositionalgeneralizationneuro-symbolicreasoningvision-languagedependencyparsingwordembeddingssoftlogiccompositionCLEVRbenchmarksReaSCAN

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NeSyCoCo sets out to make neuro-symbolic visual reasoning compositional without a fixed dictionary of predicates. It uses an LLM to turn a query into a logical program, adds a dependency parse of the query to the prompt, and represents every predicate by a frozen word embedding rather than a learned one-hot symbol. A single shared network scores objects and relations from the embedding and visual features, and normalized scores are combined by soft differentiable logic. On standard generalization benchmarks the paper reports the best average grounding accuracy on ReaSCAN and the best split-B accuracy on CLEVR-CoGenT, and its new CLEVR-SYN benchmark shows it can ground some unseen synonyms in zero shots. The reason to care is that this points toward agents that combine familiar concepts with new words instead of needing retraining.

What carries the argument

The load-bearing mechanism is a single shared multimodal feed-forward network that consumes two inputs: a predicate's distributed word representation (a 300-dimensional GloVe vector) and the visual feature of an object or object pair, and outputs a sigmoid-normalized score in [0,1]. Because the predicate enters as a vector rather than as an index into a table of per-predicate networks, synonyms and paraphrases with similar embeddings activate the same learned function. The accompanying soft logic executor then combines these normalized scores using multiplication for 'and', max for 'exists', min for 'forall', min-max linear normalization for 'iota', and a sum for 'count', replacing the raw-logit min and softmax operations of the earlier framework. Dependency parsing of the query, fed to the LLM as extra context, makes the generated symbolic programs line up with the query's grammatical structure.

What would settle it

Run the trained model on CLEVR-SYN after replacing GloVe with a scrambled-but-fixed embedding table: if hard-split accuracy stays near the reported 73.4 percent, the mechanism does not depend on semantic structure in the embeddings, while a drop toward the LEFT baseline would show the zero-shot claim is bounded by embedding quality. A lighter test is to restrict the hard split to synonym pairs whose GloVe cosine similarity is below 0.4 and check whether accuracy falls to chance.

Watch

Extended reading notes

Core claim

The paper's central claim is that a neuro-symbolic reasoner can generalize compositionally without a predefined predicate set by letting language itself define the predicates. Every predicate token from the LLM-generated program is mapped through an off-the-shelf word embedding, concatenated with visual features, and scored by one shared feed-forward network; that normalized score replaces raw logits. Composition is then done with soft operations: multiplication for conjunction, max for existential and relational 'and', min for universal quantification, bounded linear normalization for variable assignment, and sums for counting. The paper reports 97.5 percent average grounding accuracy on ReaSCAN against 92.3 for the previous LEFT baseline, 78.8 percent on CLEVR-CoGenT split B against 76.2, and 73.4 percent on the hard split of its new CLEVR-SYN synonym benchmark against 49.5, with a paired t-test p=0.0026 showing the soft reasoner contributes significantly. The paper itself qualifies this with its own correlation table: only 9 of 15 synonym pairs score above 0.6, so the claim is that generalization works as far as the frozen embeddings preserve meaning.

Load-bearing premise

The load-bearing premise is that a frozen off-the-shelf word embedding carries enough of each predicate's meaning that a network trained only on the original vocabulary will also give sensible scores to unseen synonyms; the paper's own CLEVR-SYN correlations show this is only partly true, with cube/box at 0.06 and sphere/ball at -0.02.

Editorial extensions

If this is right

If the ReaSCAN results hold, the largest gains appear in the C splits, which test relative clauses and 'that is' constructions, where normalized predicate scores and dependency-parsed prompts matter most.
On CLEVR-CoGenT split B, the method moves past the 76.2 percent ceiling shared by both LEFT and MDETR, suggesting the soft reasoner rather than the visual encoder drives the improvement.
Novel-concept grounding works zero-shot for synonyms whose embeddings are close to the training predicate and fails for the others, making embedding choice part of the generalization story.
Soft, normalized composition removes the scale mismatch between predicate scores, so conjunctions can be trained jointly instead of being dominated by the minimum of raw logits.
The CLEVR-SYN benchmark itself provides a reusable test for whether a neuro-symbolic method can ground unseen but similar concepts without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the shared network sees predicate embeddings rather than one-hot symbols, the same architecture could in principle be retrained with any embedding family; the paper's own appendix data already suggest contextual encoders behave differently from static ones.
A natural stress test would swap the benchmark's synonym pairs for antonyms or confusable attributes with similar embeddings, such as left versus right, since a model that relies on embedding proximity could fail loudly there.
The CLEVR-SYN correlation analysis suggests a concrete design rule: zero-shot transfer via frozen embeddings should only be trusted when the embedding cosine similarity exceeds roughly 0.4, matching the break-off point in the paper's Figure 5.
Because the perception module is inherited from the baseline, the improvements are measured at the reasoning layer; combining this predicate-embedding mechanism with a stronger visual encoder is an untested combination.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a credible LEFT follow-up, not a breakthrough. The three modifications are concrete and the ablations show each component earns its keep. Dependency-parsed prompting helps most on the complex C splits, the shared embedding-conditioned predicate FFN beats predicate-specific FFNs, and normalizing predicate scores before soft composition fixes a real scale problem. The numbers are three-run means with standard deviations, and the main comparisons are against external baselines. That is honest, reproducible work, and CLEVR-SYN is a useful new evaluation set even with the flaws below.

Now the soft spots. First, the abstract's SOTA claim is too broad. On ReaSCAN, NeSyCoCo beats LEFT on every split but loses to Syntax-Guided Transformer on A1, A2, and A3. On CLEVR-CoGenT split A it is not SOTA either. Saying 'state-of-the-art on average' or 'on split B' would be accurate; the blanket claim overreaches. Second, the CLEVR extension table evaluates NeSyCoCo only with ground-truth programs, so we never see how the LLM program generator behaves on those tasks. That is a missing comparison, not a fatal flaw. Third, and most important, I think the stress-test note is right about CLEVR-SYN. Section 4.4 says concepts in programs are replaced via regular-expression search on CLEVR programs, but never says the corresponding natural-language queries are modified. If only the programs change, the LLM never encounters the novel synonyms; the test measures the shared predicate FFN's ability to ground unseen GloVe embeddings, not the system's ability to understand novel words. Since handling linguistic lexical variety is advertised as a central contribution, this distinction matters. To the authors' credit, Table 6 and the surrounding analysis admit that only 9 of 15 substitutions have correlation above 0.6, and cube-box and sphere-ball are near zero. That is a genuinely honest self-assessment, which makes the 'robust performance with novel concepts' claim hard to defend as stated. The benchmark should either rewrite the queries end-to-end or be repositioned as a predicate-grounding probe.

Who is this for? People working on neuro-symbolic VQA and compositional generalization who want a practical recipe and a new evaluation probe. It deserves a serious referee. The engineering is clean, the ablations are informative, and the CLEVR-SYN protocol issue is fixable with clarification or a small extension. I would accept it for review and request the missing LLM-program numbers and an exact description of what is replaced in CLEVR-SYN.

Referee Report

4 major / 5 minor

Summary. The paper proposes NeSyCoCo, a neuro-symbolic vision-language reasoning framework that combines an LLM-based language-to-program module, dependency parsing of input queries, shared predicate networks conditioned on frozen word embeddings, and normalized soft composition operations. The method is evaluated on ReaSCAN, CLEVR-CoGenT, CLEVR, CLEVR-Ref/Puzzle/RPM, and a newly introduced CLEVR-SYN benchmark. The central claims are state-of-the-art accuracy on ReaSCAN and CLEVR-CoGenT, and robust zero-shot generalization to novel but semantically related concepts on CLEVR-SYN, with reported three-run means and standard deviations.

Significance. If the reported results hold, the paper makes a useful contribution: replacing per-predicate networks with a shared FFN over distributed word embeddings is a simple and plausible mechanism for grounding unseen predicates, and the normalized soft composition operations address a known weakness in LEFT's use of unbounded logits. The availability of code, three-run statistics, ablations, and an analysis of score distributions are strengths. However, the significance is currently limited by evaluation gaps: the CLEVR-SYN benchmark appears to modify programs rather than natural-language queries, the CLEVR extension tables report NeSyCoCo only with ground-truth programs, and the state-of-the-art claim is not supported on every split. These issues affect the paper's headline claims and should be addressed before the results can be fully assessed.

major comments (4)

[Section 4.4, CLEVR-SYN construction and Table 7] The CLEVR-SYN benchmark is described as replacing concepts 'in programs' via regular-expression search, but the paper never states that the corresponding natural-language queries are also modified. In the NeSyCoCo pipeline, the language-to-program module reads the query and generates the program; if only ground-truth programs are rewritten while queries retain original words, the full system never encounters the novel synonyms. The evaluation then isolates whether the shared predicate FFN can ground unseen GloVe embeddings, but it does not test whether the system understands novel linguistic expressions. Because 'Handling Language Variety' is a central contribution and the abstract claims 'robust performance with novel concepts,' this gap is load-bearing. The authors should either modify the queries as well and run the full pipeline, or explicitly reframe the claim as a module-level test of predicate grounding.
[Section 4.3, Table 5] For the CLEVR extension tasks, NeSyCoCo is reported only as 'NeSyCoCo + GT programs' (100% on Ref, 95% on Puzzles, 100% on RPM), while LEFT is reported both with GT programs and with LLM programs. This comparison does not support a claim that the full NeSyCoCo pipeline, including its language-to-program component, improves on LEFT for these tasks. Without a 'NeSyCoCo + LLM programs' row, the reader cannot tell whether the 95% Puzzle result is due to the proposed reasoning/grounding changes or to the use of ground-truth programs. Please report the full-pipeline results on these extensions or restrict the claim to the GT-program setting.
[Section 4.2, Tables 2 and 3] The abstract claims 'state-of-the-art results on the ReaSCAN and CLEVR-CoGenT compositional generalization benchmarks,' but the reported numbers do not support this on every split. On ReaSCAN, Syntax Guided Transformer achieves 97.3% on A2 and 99.6% on A1 and A3, compared with NeSyCoCo's 94.1%, 99.1%, and 98.5% respectively. On CLEVR-CoGenT Split A, MDETR achieves 99.7% versus NeSyCoCo's 99.6%. The claim should be qualified to the average ReaSCAN accuracy and CLEVR-CoGenT Split B, or the authors should discuss why competitiveness on some splits is sufficient for the stated claim.
[Section 4.4, Table 6 and hard-split analysis] The per-substitution Pearson correlations in Table 6 are near zero or negative for several substitutions: cube-box 0.06, sphere-ball -0.02, red-burgundy 0.15, and brown-chocolate 0.20. The paper acknowledges that only 9 of 15 substitutions have correlation above 0.6, but the aggregate hard-split accuracy of 73.4% may hide large failures on queries containing these low-correlation predicates. To support the robustness claim, please report accuracy separately for queries involving low-correlation substitutions, or provide per-substitution accuracy in addition to the correlation analysis.

minor comments (5)

[Table 2] The row labeled 'w/o Emb. w/o Soft Reasoner' has exactly the same values and standard deviations as the row labeled 'LEFT†.' If this is intentional, the table and caption should explain why the two rows coincide; otherwise it appears to be a copy-paste error that undermines the readability of the ablation study.
[Introduction and Section 4.2] The introduction states that NeSyCoCo 'achieves state-of-the-art results on the ReaSCAN compositional generalization and CLEVR-Puzzle benchmarks,' but the experiments do not report a 'CLEVR-Puzzle' benchmark as a single named dataset; they report CLEVR-Puzzle as one of the CLEVR extensions in Table 5. The wording should be aligned with the experimental sections.
[Section 3.3] The phrase 'improves composition and interoperability' in the discussion of soft composition appears to be a typo for 'interpretability'; please correct it.
[Table 9 and throughout] The dataset name 'CLVER' appears twice in Table 9; it should be 'CLEVR.' There are also several typos such as 'CLVER-CoGenT' in the same table.
[Ethical Statement] The Ethical Statement appropriately acknowledges that predicate generalization depends on the choice of distributed representation, but this limitation should be connected more explicitly to the CLEVR-SYN results in the main text, since the low correlations in Table 6 are instances of that dependence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are evaluated on external benchmarks (ReaSCAN, CLEVR-CoGenT, CLEVR) against external baselines, and the only self-created benchmark (CLEVR-SYN) is a falsifiable transfer test, not a fitted input.

full rationale

NeSyCoCo's derivation chain is largely self-contained. It builds on LEFT (Hsu et al. 2024), an external method, and its three proposed components (dependency-parsed prompting, shared FFN with GloVe predicate embeddings, and soft composition) are each ablated on external benchmarks in Table 2; the soft reasoner's contribution is tested with a paired t-test (p = 0.0026). The CLEVR-CoGenT and ReaSCAN results are measured against external baselines (LEFT, MDETR, GroCoT, Syntax-Guided Transformer), so the state-of-the-art claims do not reduce to the authors' own prior results. The CLEVR-SYN benchmark is the only self-created element. It tests whether a shared predicate FFN, trained on original CLEVR predicates, can score unseen synonym predicates through frozen GloVe embeddings. This is an empirical transfer test, not a construction: Table 6 shows that several substitutions fail (cube-box rho = 0.06, sphere-ball rho = -0.02, brown-chocolate rho = 0.20), so the benchmark can falsify the claimed robustness. One validity caveat is that CLEVR-SYN replaces concepts in programs by regular-expression search rather than in the natural-language queries, so the language-to-program module is not exercised on novel lexical items; however, that is an evaluation-coverage limitation, not circularity, because the paper's stated claim concerns predicate-level zero-shot grounding and the accuracy numbers are not forced by the benchmark's construction. The paper's Ethical Statement also acknowledges that predicate generalization depends on the choice of distributed representation, further supporting that the result is empirical. Self-citations (e.g., Kamali and Kordjamshidi 2023 for dependency parsing) are not load-bearing; the dependency-parsing component is ablated via the 'w/o Emb. w/o Soft Reasoner' row in Table 2. No circular step reduces a prediction to its input by definition.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical systems paper, so it does not derive its results from axioms. The load-bearing assumptions are architectural and representational: GloVe embeddings must capture synonymy, the LLM must produce correct programs most of the time, Mask R-CNN feature extraction must be adequate, and soft logic operations must approximately preserve logical meaning. The main free parameters are the learned weights and hyperparameters of the shared predicate network.

free parameters (3)

Shared predicate FFN weights = Learned (14.2-14.3M total parameters)
All predicate scores are produced by this network; its weights are fitted to training data and determine whether unseen synonyms generalize, but the paper does not analyze how many or which examples are needed.
Learning rate (per dataset) = Grid searched in {1e-3, 1e-4, 1e-5} for ReaSCAN/CoGenT; {1e-2, 1e-3, 1e-4, 1e-5} for CLEVR
Chosen by validation performance; not derived from theory. Reported in Appendix B Table 9.
Architecture hyperparameters (FFN sizes, batch size, epochs, curriculum) = FFN [1024,512,256,128,1]; batch 32; epochs 100; curriculum on
Manual choices that affect results; no sensitivity analysis reported.

assumptions (4)

standard math A product t-norm and the complement operator are valid fuzzy-logic interpretations of conjunction and negation.
Table 1 replaces min with product and negation with 1-x; no proof that these preserve the semantics of the logical program, and no analysis of cases where scores are near zero.
domain assumption GloVe 6B-300D embeddings are a sufficient semantic representation for predicates and their synonyms.
The method and CLEVR-SYN evaluation rely on this; Table 6 shows it fails for several substitutions (cube-box, sphere-ball), so it is a load-bearing assumption.
domain assumption Mask R-CNN object proposals and relation features provide a complete enough scene representation for the benchmarks.
The perception module in Section 3.2 is taken as given; errors in object detection would propagate into predicate scores.
domain assumption The LLM plus dependency parsing produces programs that are semantically equivalent to the intended query often enough for the reported accuracies.
The authors state in the Ethical Statement that 'unresolved semantic errors remain a significant challenge'; the main text does not quantify the program error rate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization." pith.science (2026). https://pith.science/paper/BB5X5LM5

@misc{pith2026241215588,
  author       = {Pith},
  title        = {Pith review of: NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BB5X5LM5}},
  note         = {Machine review of arXiv:2412.15588}
}

read the original abstract

Compositional generalization is crucial for artificial intelligence agents to solve complex vision-language reasoning tasks. Neuro-symbolic approaches have demonstrated promise in capturing compositional structures, but they face critical challenges: (a) reliance on predefined predicates for symbolic representations that limit adaptability, (b) difficulty in extracting predicates from raw data, and (c) using non-differentiable operations for combining primitive concepts. To address these issues, we propose NeSyCoCo, a neuro-symbolic framework that leverages large language models (LLMs) to generate symbolic representations and map them to differentiable neural computations. NeSyCoCo introduces three innovations: (a) augmenting natural language inputs with dependency structures to enhance the alignment with symbolic representations, (b) employing distributed word representations to link diverse, linguistically motivated logical predicates to neural modules, and (c) using the soft composition of normalized predicate scores to align symbolic and differentiable reasoning. Our framework achieves state-of-the-art results on the ReaSCAN and CLEVR-CoGenT compositional generalization benchmarks and demonstrates robust performance with novel concepts in the CLEVR-SYN benchmark.

Figures

Figures reproduced from arXiv: 2412.15588 by the authors.

**Figure 2.** Language to program conversion procedure. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Differentiable predicate function in NeSy [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Boxplot comparing concept scores of LEFT and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Relationship between cosine similarity of word [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 39 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Awadalla, A.; Gao, I.; Gardner, J.; Hessel, J.; Hanafy, Y.; Zhu, W.; Marathe, K.; Bitton, Y.; Gadre, S.; Sagawa, S.; et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

arXiv 2023
[4]

Benesty, J.; Chen, J.; Huang, Y.; and Cohen, I. 2009. Pearson Correlation Coefficient, 1--4. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-642-00296-0

work page 2009
[5]

Beurer-Kellner, L.; Fischer, M.; and Vechev, M. 2023. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI): 1946--1969

work page 2023
[6]

A.; and Yu, T

Cheng, Z.; Xie, T.; Shi, P.; Li, C.; Nadkarni, R.; Hu, Y.; Xiong, C.; Radev, D.; Ostendorf, M.; Zettlemoyer, L.; Smith, N. A.; and Yu, T. 2023. Binding Language Models in Symbolic Languages. ICLR, abs/2210.02875

arXiv 2023
[7]

Csordás, R.; Irie, K.; and Schmidhuber, J. 2021. The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

work page 2021
[8]

Dubey, A.; Jauhri, A.; and Others. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

arXiv 2024

Show all 52 references

[9]

R.; Guo, Q.; Uszok, A.; Nafar, A.; Raisi, E.; and Kordjamshidi, P

Faghihi, H. R.; Guo, Q.; Uszok, A.; Nafar, A.; Raisi, E.; and Kordjamshidi, P. 2021. DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning

2021
[10]

R.; Nafar, A.; Uszok, A.; Karimian, H.; and Kordjamshidi, P

Faghihi, H. R.; Nafar, A.; Uszok, A.; Karimian, H.; and Kordjamshidi, P. 2024. Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language. In Besold, T. R.; d'Avila Garcez, A.; Jimenez-Ruiz, E.; Confalonieri, R.; Madhyastha, P.; and Wagner, B., eds., Neural-Symb...

2024
[11]

Gao, T.; Huang, Q.; and Mooney, R. 2020. Systematic Generalization on g SCAN with Language Conditioned Embedding

2020
[12]

R.; Zhang, Y.; Uszok, A.; and Kordjamshidi, P

Guo, Q.; Faghihi, H. R.; Zhang, Y.; Uszok, A.; and Kordjamshidi, P. 2020. Inference-Masked Loss for Deep Structured Output Learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI 2020)

2020
[13]

Gupta, T.; and Kembhavi, A. 2023. Visual Programming: Compositional visual reasoning without training . 14953--14962

2023
[14]

He, K.; Gkioxari, G.; Doll \' a r, P.; and Girshick, R. B. 2017. Mask R-CNN . CoRR, abs/1703.06870

2017 arXiv
[15]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

2016
[16]

Honnibal, M.; and Montani, I. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear

2017
[17]

Hsu, J.; Mao, J.; Tenenbaum, J.; and Wu, J. 2024. What’s left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36

2024
[18]

Hsu, J.; Mao, J.; and Wu, J. 2023. DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage. Transactions on machine learning research

2023
[19]

Hupkes, D.; Dankers, V.; Mul, M.; and Bruni, E. 2020. Compositionality Decomposed: How do Neural Networks Generalise? (Extended Abstract). In Bessiere, C., ed., Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , 5065--5069. In...

2020
[20]

Jiang, B.; Luo, R.; Mao, J.; Xiao, T.; and Jiang, Y. 2018. Acquisition of Localization Confidence for Accurate Object Detection

2018
[21]

Jiang, Y.; and Bansal, M. 2021. Inducing Transformer ' s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6253--6265. Online and Punta Cana, Dominican Republ...

2021
[22]

L.; and Girshick, R

Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. 2017 a . CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In CVPR

2017
[23]

L.; and Girshick, R

Johnson, J.; Hariharan, B.; van der Maaten, L.; Hoffman, J.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. 2017 b . Inferring and Executing Programs for Visual Reasoning

2017
[24]

Kamali, D.; and Kordjamshidi, P. 2023. Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments. In Hupkes, D.; Dankers, V.; Batsuren, K.; Sinha, K.; Kazemnejad, A.; Christodoulopoulos, C.; Cotterell, R.; and Bruni, E., eds., ...

2023
[25]

Kamath, A.; Singh, M.; LeCun, Y.; Synnaeve, G.; Misra, I.; and Carion, N. 2021. MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. 1760--1770

2021
[26]

Kuo, Y.-L.; Katz, B.; and Barbu, A. 2021. Compositional Networks Enable Systematic Generalization for Grounded Language Understanding. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Findings of the Association for Computational Linguistics: EMNLP 2021, 216--2...

2021
[27]

L.; Jiang, H.; Peng, Y.; Wang, L.; Lin, X.; Tu, C.-C.; Savani, Y.; Fu, D.; Ou, J.; Zhou, D.; Ma, T.; and Liang, P

Li, X. L.; Jiang, H.; Peng, Y.; Wang, L.; Lin, X.; Tu, C.-C.; Savani, Y.; Fu, D.; Ou, J.; Zhou, D.; Ma, T.; and Liang, P. 2023. Llava: Large Language and Vision Assistant. arXiv preprint arXiv:2305.11495

2023 arXiv
[28]

Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge

2024
[29]

Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692

2019 arXiv
[30]

B.; and Wu, J

Mao, J.; Gan, C.; Kohli, P.; Tenenbaum, J. B.; and Wu, J. 2019. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision . In International Conference on Learning Representations

2019
[31]

Ontañón, S.; Ainslie, J.; Cvicek, V.; and Fisher, Z. 2021. Making Transformers Solve Compositional Tasks. 1: 3591--3607

2021
[32]

Partee, B.; et al. 1984. Compositionality. Varieties of formal semantics, 3: 281--311

1984
[33]

Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; K \"o pf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019. PyTorch:...

2019 arXiv
[34]

Pennington, J.; Socher, R.; and Manning, C. 2014. Glove: Global Vectors for Word Representation. volume 14, 1532--1543

2014
[35]

Qiu, L.; Hu, H.; Zhang, B.; Shaw, P.; and Sha, F. 2021. Systematic Generalization on g SCAN : W hat is Nearly Solved and What is Next? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2180--2188. Online and Punta Cana, Dominican Republ...

2021
[36]

Rajaby Faghihi, H.; Nafar, A.; Zheng, C.; Mirzaee, R.; Zhang, Y.; Uszok, A.; Wan, A.; Premsri, T.; Roth, D.; and Kordjamshidi, P. 2023. GLUECons: A Generic Benchmark for Learning under Constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8): 9552--9561

2023
[37]

Ruis, L.; Andreas, J.; Baroni, M.; Bouchacourt, D.; and Lake, B. 2020. A benchmark for systematic generalization in grounded language understanding

2020
[38]

Sikarwar, A.; Patel, A.; and Goyal, N. 2022. When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 648--669. Abu Dhabi, United Arab Emirates: A...

2022
[39]

Sinha, S.; Premsri, T.; and Kordjamshidi, P. 2024. A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices. Transactions on Machine Learning Research. Survey Certification

2024
[40]

Sur\'is, D.; Menon, S.; Vondrick, C.; and . 2023. ViperGPT: Visual Inference via Python Execution for Reasoning

2023
[41]

Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Fan, Y.; Dang, K.; Du, M.; Ren, X.; Men, R.; Liu, D.; Zhou, C.; Zhou, J.; and Lin, J. 2024. Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. arXiv...

2024 arXiv
[42]

C.; and Potts, C

Wu, Z.; Kreiss, E.; Ong, D. C.; and Potts, C. 2021. Rea SCAN : Compositional Reasoning in Language Grounding. NeurIPS 2021 Datasets and Benchmarks Track

2021
[43]

Xiao, B.; Wu, H.; Xu, W.; Dai, X.; Hu, H.; Lu, Y.; Zeng, M.; Liu, C.; and Yuan, L. 2023. Florence-2: Advancing a unified representation for a variety of vision tasks. arXiv preprint arXiv:2311.06242

2023 arXiv
[44]

Xu, G.; Kordjamshidi, P.; and Chai, J. 2023. M eta R e V ision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 12224--122...

2023
[45]

Xu, G.; Kordjamshidi, P.; and Chai, J. 2024. GIPCOL: Graph-Injected Soft Prompting for Compositional Zero-Shot Learning. In IEEE/CVF Winter Conference on Applications of Computer Vision

2024
[46]

H.; Shang, X.; Yuan, Z.; Sun, Y.; and Liu, J

Xu, L.; Huang, M. H.; Shang, X.; Yuan, Z.; Sun, Y.; and Liu, J. 2023. Meta Compositional Referring Expression Segmentation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 19478--19487

2023
[47]

Yi, K.; Wu, J.; Gan, C.; Torralba, A.; Kohli, P.; and Tenenbaum, J. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31

2018
[48]

Yun, T.; Bhalla, U.; Pavlick, E.; and Sun, C. 2023. Do Vision-Language Pretrained Models Learn Composable Primitive Concepts? Transactions on Machine Learning Research

2023
[49]

Zelikman, E.; Huang, Q.; Poesia, G.; Goodman, N.; and Haber, N. 2023. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions. Advances in Neural Information Processing Systems, 36: 31466--31523

2023
[50]

Zhang, Y.; and Kordjamshidi, P. 2023. VLN-Trans: Translator for the Vision and Language Navigation Agent. In The 61st Annual Meeting Of The Association For Computational Linguistics

2023
[51]

Zhang, Y.; Ma, Z.; Li, J.; Qiao, Y.; Wang, Z.; Chai, J.; Wu, Q.; Bansal, M.; and Kordjamshidi, P. 2024. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint arXiv:2407.07035

2024 arXiv
[52]

Zhu, W.; Thomason, J.; and Jia, R. 2022. Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems. 4697--4711

2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Awadalla, A.; Gao, I.; Gardner, J.; Hessel, J.; Hanafy, Y.; Zhu, W.; Marathe, K.; Bitton, Y.; Gadre, S.; Sagawa, S.; et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

arXiv 2023

[4] [4]

Benesty, J.; Chen, J.; Huang, Y.; and Cohen, I. 2009. Pearson Correlation Coefficient, 1--4. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-642-00296-0

work page 2009

[5] [5]

Beurer-Kellner, L.; Fischer, M.; and Vechev, M. 2023. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI): 1946--1969

work page 2023

[6] [6]

A.; and Yu, T

Cheng, Z.; Xie, T.; Shi, P.; Li, C.; Nadkarni, R.; Hu, Y.; Xiong, C.; Radev, D.; Ostendorf, M.; Zettlemoyer, L.; Smith, N. A.; and Yu, T. 2023. Binding Language Models in Symbolic Languages. ICLR, abs/2210.02875

arXiv 2023

[7] [7]

Csordás, R.; Irie, K.; and Schmidhuber, J. 2021. The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

work page 2021

[8] [8]

Dubey, A.; Jauhri, A.; and Others. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

arXiv 2024