The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models

Aaditya Naik; Adam Stein; Eric Wong; Mayur Naik; Neelay Velingker

REVIEW 4 major objections 5 minor 16 references

The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that neuro-symbolic learning's perception component can be replaced by prompted foundation models, achieving the reliability and interpretability of symbolic programs without the compute, overfitting, and…

desk verdict A useful position paper with a clear three-pitfall taxonomy, but the 'generalizable' claim rests on thinner evidence than the rhetoric suggests. read the letter →

arxiv 2505.24874 v1 pith:ECVRHHRC submitted 2025-05-30 cs.LG

Adam Stein , Aaditya Naik , Neelay Velingker , Mayur Naik , Eric Wong This is my paper

classification cs.LG MSC 68T0568T0768T27

keywords neuro-symboliclearningfoundationmodelspromptinggeneralizationoverfittingsymbolhallucinationinterpretabilityprogramsynthesis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the neural part of neuro-symbolic learning no longer needs to be trained from scratch: prompted foundation models can extract the symbols, and a hand-written program does the reasoning, a setup the authors call neuro-symbolic prompting. Their experiments across five benchmarks—summing handwritten digits, evaluating handwritten arithmetic expressions, answering family-relationship questions, classifying leaf species, and answering visual questions about objects—show that as foundation models grow, their accuracy approaches or overtakes trained neuro-symbolic systems. They identify three pitfalls of the traditional training-based approach: training becomes wasteful compute, trained perception models overfit to dataset biases and degrade under small noise, and the weak supervision from the program lets models hallucinate intermediate symbols that still give the right answer. If true, the argument redirects the field's effort away from training perception models and toward automatically synthesizing the symbols and programs.

What carries the argument

The central object is the neuro-symbolic prompting pipeline: a foundation model, given a few-shot prompt, converts each raw input into discrete symbols, and a deterministic program computes the answer from those symbols. The argument's engine is the scaling comparison that plots prompted-model accuracy against parameter count and shows the gap to trained baselines shrinking as models grow. The named mechanism doing the heavy lifting is symbol hallucination: a trained perception network emits symbols not supported by the input that the program happens to combine into the correct final answer, which the paper argues explains most of the trained baselines' apparent advantage on the leaf, family-relationship, and visual-question benchmarks. This mechanism extends prior work on reasoning shortcuts. The paper's forward-looking target is program-symbol synthesis: automatically producing the program and symbol vocabulary for a new task, which it identifies as the field's remaining frontier.

What would settle it

A concrete check: rerun the sum-of-digits experiment with Gaussian noise at standard deviations 0.01, 0.03, 0.06, and 0.12, using three independently written prompt templates for the foundation model while keeping the few-shot examples fixed. If the trained baseline's accuracy drop at any noise level is smaller than the prompted model's worst-template drop, or if any prompt template makes the prompted model lose as much accuracy as the trained baseline, the data-pitfall claim is falsified. A second check: take the leaf images where the trained method is correct and ask fresh human annotators, shown only the images, to classify margin, shape, and texture; if a majority agree with the trained method's symbols, the symbol-hallucination claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that replacing the trained neural perception component of a neuro-symbolic system with a prompted foundation model—termed neuro-symbolic prompting—delivers the reliability of program execution and the interpretability of intermediate symbols that motivated neuro-symbolic learning, while avoiding the costs and generalization failures of training. The authors show that on a text-based family-relationship benchmark and a visual question-answering benchmark the largest prompted models already outperform the trained neuro-symbolic baseline, and on handwritten-digit-sum and leaf-classification benchmarks the accuracy gap shrinks as model size grows, with the remaining gap attributable to the baselines' overfitting and symbol hallucination rather than to better symbol extraction. Under 3% Gaussian noise, prompted models lose roughly 1–2% accuracy while trained baseline models lose up to 21%, and a human evaluation indicates that when the trained method gets the right answer on leaf images, a majority of respondents disagree with the intermediate symbols it used. The paper further shows, on two benchmarks with increasing problem complexity, that neuro-symbolic prompting maintains accuracy while pure chain-of-thought prompting degrades, and concludes that the open problem is no longer learning symbols but synthesizing the right symbols and program.

Load-bearing premise

The argument rests on the assumption that the five chosen benchmarks, the hand-written prompts, and the single 3% Gaussian-noise shift fairly represent both paradigms; if a different prompt template, a different noise level, or a different distribution shift changed which method wins, the three pitfalls would not be robust.

Editorial extensions

If this is right

Practitioners can assemble a neuro-symbolic system by prompting a frontier foundation model for symbols and running a hand-written program, skipping end-to-end differentiable training entirely.
Research effort should shift from training perception modules to automatically synthesizing the symbolic program and symbol vocabulary that connect foundation-model outputs to task answers.
Trained perception models should be validated under distribution shift before their reported accuracy is trusted, since in-distribution accuracy can reflect dataset memorization and symbol hallucination rather than genuine concept learning.
On reasoning tasks with increasing complexity (more steps, longer programs), the reliability advantage of a real symbolic program over pure chain-of-thought prompting grows.
Finetuning or distilling a foundation model into a smaller specialized network remains a sensible middle ground for resource-constrained or specialized domains, since it starts from a capable base instead of from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that if the scaling trend continues, the remaining leaf-classification gap will close as multimodal models improve, making the benchmark no longer discriminate between the paradigms.
A testable extension of the symbol-hallucination finding: for any neuro-symbolic system lacking symbol ground truth, sample cases with correct final answers and ask human annotators whether the intermediate symbols are faithful; low agreement flags the program pitfall.
Generalizing the data-pitfall logic beyond neuro-symbolic learning, any trained perception model's in-distribution accuracy can be inflated by dataset memorization, so robustness reporting under distribution shift should accompany every accuracy claim.
A natural next experiment the paper does not run: have a foundation model synthesize the program itself, then measure how often the generated program plus prompted symbols matches held-out answers, operationalizing the frontier the paper identifies.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper is a position paper with experiments, not a settled empirical result. The core idea, combining prompted foundation models with symbolic programs, is not new (SatLM, PAL, Logic-LM, Vieira all do this), but the paper gives it a name and a systematic treatment. The three-pitfall taxonomy, compute, data, and program, is the real contribution. The compute pitfall is well supported by scaling curves showing the gap narrowing as model size grows. The program pitfall connects cleanly to the reasoning-shortcuts literature, and the human evaluation for Leaf is a nice, honest touch. The paper also concedes in its Alternate Views section that finetuning is a viable middle ground, which is more balanced than most position papers. The soft spot is the data pitfall, which carries most of the 'generalizable' weight. The evidence is a single 3% Gaussian-noise perturbation on four image datasets. That measures robustness to one synthetic shift, not generalization to the dataset-specific biases neuro-symbolic training supposedly overfits to, such as MNIST digit style, leaf species distribution, or CLEVR object frequencies. The stress-test note is right: one noise level and no error bars (greedy decoding, temperature 0, no seeds) make the gap between -0.02 and -0.03, or -0.06 and -0.07, unknowable. Missing cells for ISED and GPT-4o further weaken the comparison. The symbol-hallucination metric is also conditional on the subset where the trained method is right and the prompted method is wrong, which inflates the finding; the CLUTRR 0.00 is stark but likely based on very few examples. The human evaluation is small (10 participants, 15 items per property). That said, the paper is a position paper, and the experiments are illustrative rather than a formal proof. The central claim, that trained perception modules are increasingly unnecessary in neuro-symbolic systems, is plausible and aligns with the scaling trends. It just does not settle the question. The code is available, which helps reproducibility. I would send this to peer review. A good referee can push for a broader shift suite, variance estimates, and filled-in baseline cells. As a reader, take it as a hypothesis-generating argument, not the final word.

Referee Report

4 major / 5 minor

Summary. This position paper argues that replacing the trained neural perception modules of classical neuro-symbolic (NeSy) systems with prompted foundation models, in a configuration the authors call 'neuro-symbolic prompting,' can achieve the core benefits of NeSy learning--program reliability and symbol interpretability--without the costs and generalization failures of training. The argument is built on three claimed pitfalls of NeSy training: the compute pitfall (training specialized perception models is increasingly unnecessary as foundation models scale), the data pitfall (NeSy-trained models overfit to dataset particularities, evidenced by larger accuracy drops under Gaussian noise), and the program pitfall (NeSy-trained models often hallucinate intermediate symbols while still reaching correct final answers). The authors evaluate several open and proprietary foundation models against Scallop and ISED on five benchmarks (Sum5, HWF5, CLUTRR, Leaf, CLEVR), and add a human evaluation on the Leaf dataset plus qualitative examples. They conclude that future NeSy research should focus on automatic program and symbol synthesis rather than training perception models.

Significance. If the empirical claims were fully supported, the paper would be a significant contribution to the neuro-symbolic learning community: it reframes a long-standing research agenda away from training perception components and toward program/symbol synthesis, and it consolidates a large amount of evidence across many foundation-model families on standard NeSy benchmarks. The paper is honest in scope as a position paper, it ships code, and the human evaluation of the Leaf attributes is a useful methodological step. However, the strength of the conclusions currently exceeds the evidence: the main generalization claim rests on a single, narrow perturbation, several model/dataset cells are missing, and the prompted results have no variance estimates. The direction is plausible and the paper is likely to stimulate discussion, but the load-bearing empirical support needs substantial strengthening before the central claims can be accepted.

major comments (4)

[Section 2.2, Table 1, Figure 4] The 'data pitfall' claim that NeSy training 'encourages overfitting to dataset particularities' is supported only by a single robustness experiment: 3% Gaussian noise on four image datasets. This measures robustness to one synthetic perturbation, not the generalization to dataset-specific biases (MNIST writing style, leaf species distribution, CLEVR object frequencies) that the paper's own qualitative examples invoke. The absence of error bars for the prompted results (Appendix B states greedy decoding, temperature 0) makes the differences for Sum5 (-0.02 vs -0.03) and Leaf (-0.06 vs -0.07) statistically uninterpretable. I would need a broader shift suite (rotation, background, style, held-out classes), multiple noise levels, and variance estimates before the 'generalizable' conclusion in Section 7 can be credited.
[Section 2.1, Figure 2, Table B.1] The claim that 'as model size increases, the gap between neuro-symbolic training and neuro-symbolic prompting increasingly vanishes' is not consistently supported by the data. For HWF5 the best prompted model (Gemini 2.0 Flash, 0.710) remains far below Scallop (0.966), for Leaf the best (GPT-4o, 0.509) is far below Scallop (0.811), and InternVL2.5-78B scores 0.000 on HWF5, which is worse than its 38B variant (0.140). The figure presents a visual trend without any quantitative trend test, and the missing ISED cells (HWF5, CLUTRR, CLEVR) and GPT-4o cells (four datasets) prevent a complete comparison. The compute pitfall is qualitatively plausible, but the stated 'gap shrinks with scale' claim is too strong relative to the reported numbers.
[Section 2.3, Table 1, Appendix C] The symbol-hallucination metric is defined only over the subset of cases where the NeSy training method is correct and the NeSy prompting method is wrong. This conditional measure cannot support the conclusion that the overall performance advantage of Scallop over prompting 'does not come from more correct intermediate symbol prediction,' because it ignores cases where both methods are correct, both are wrong, or prompting is correct. The number of samples in the conditional subset is not reported; for CLUTRR, where Scallop accuracy is only 0.400 and the reported value is 0.00, the subset may be very small. The human evaluation on Leaf uses 10 raters on 45 items with Cohen's kappa values as low as 0.05; interpreting this as evidence that the attributes are 'not apparent from the image' is fragile without per-item variance and a larger rater pool.
[Section 3.1, Figure 6] The 'program reliability' opportunity is demonstrated with a single foundation model (Gemini-2.0-Flash) on two datasets (CLUTRR and CLEVR), again without error bars. Since the paper's central recommendation is to adopt neuro-symbolic prompting generally, this evidence should be extended to at least a subset of the model family used in Section 2, or the scope of the claim should be narrowed. Without variance estimates, the plotted curves in Figure 6 could overlap under different decoding or prompting choices, which weakens the claim that neuro-symbolic prompting 'achieves consistently high accuracy.'

minor comments (5)

[Table 1] The abbreviations 'NStrain' and 'NSprompt' are used in the table but not spelled out in the caption; please define them in the caption or the text.
[Appendix B, first paragraph] The sentence 'we use greedy decoding (temperature 0) so there are no error bars for neuro-symbolic prompting methods' is misleading: greedy decoding removes sampling randomness for a fixed prompt/input, but it does not remove variance due to API nondeterminism, prompt-order effects, or test-set resampling. At minimum, the paper should acknowledge that the reported prompted accuracies are point estimates from a single run.
[Figure 1] The three-panel figure is helpful, but the middle panel (pure prompting) and the right panel (neuro-symbolic prompting) differ only by the addition of a 'Program' box; the visual distinction could be made stronger, for example by highlighting the non-trained nature of the middle path.
[Section 2.2, Figure 3] The MNIST example attributes Scallop's correct prediction to 'memorized this particular image' but no evidence (e.g., training-set membership) is provided. If the claim is about memorization rather than learned bias, a quantitative check would be useful; otherwise the qualitative example should be phrased more cautiously.
[Section 2.3] The term 'symbol hallucination' is introduced to cover both reasoning shortcuts and cases where the program itself is imperfect; the paper should state explicitly whether the term is intended to unify these distinct phenomena or to name a new one, since the former interpretation conflicts with the existing literature on reasoning shortcuts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical claims are benchmark comparisons against external, code-reproducible baselines; self-citations are present but not load-bearing.

full rationale

This is a position paper whose central claims—that prompted foundation models can replace trained perception modules in neuro-symbolic systems, and that trained systems overfit and hallucinate symbols—are supported by benchmark comparisons and human evaluation rather than by a formal derivation. The nearest potential issue is that the baselines Scallop [Hua+21] and ISED [Sol+24] are co-authored by one of the present authors, and Dolphin [Nai+24] is by a present author. These citations are not load-bearing: Scallop and ISED are external, code-reproduced baseline systems used as critical comparisons, and Dolphin appears only in related work. No equation is reverse-engineered to fit the thesis; the three 'pitfalls' are interpretations of Table 1 and Figures 2-5, not quantities defined in terms of the conclusions. Appendix B explicitly states 'we use greedy decoding (temperature 0) so there are no error bars for neuro-symbolic prompting methods,' and the data-pitfall row rests on a single 3% Gaussian-noise level with a missing CLUTRR cell; these are evidential limitations, not circularity. The human evaluation of Leaf symbols is an independent check of the hallucination claim. I therefore find no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted model parameters, but several design choices (noise level, few-shot counts, human evaluation subset) are hand-selected and influence results. The main axioms are that the five benchmarks are representative of neuro-symbolic tasks, that greedy decoding gives a fair evaluation, that the prompt templates are fair implementations of the paradigm, and that accuracy captures the relevant notion of generalization. No new entities are postulated.

free parameters (3)

Gaussian noise level for robustness test = 3% (single level in Table 1; Figure 4 scans up to 0.25 std dev)
Hand-chosen threshold for the data-pitfall experiment. Results at other noise levels are not reported in the main comparison, and the paper uses this one value to claim prompt models are more robust.
Few-shot example counts per dataset = Sum5: 5; HWF5 digits: 5; HWF5 operator: 4; CLUTRR: 2; CLEVR: 2; Leaf margin: 5; shape: 9; texture: 3
Prompt templates include an arbitrary number of examples chosen by the authors without ablation. Performance of foundation models can depend strongly on these counts.
Human evaluation sample size per property = 15 questions per property, 10 annotators
Small hand-picked sample; the hallucination claim's precision is limited.

assumptions (4)

domain assumption The five benchmarks (Sum5, HWF5, CLUTRR, Leaf, CLEVR) are representative of neuro-symbolic tasks.
The paper generalizes from these five datasets to the whole neuro-symbolic learning paradigm. Section 2 uses them to support the three pitfalls.
domain assumption Greedy decoding (temperature 0) is a fair and sufficient evaluation protocol for prompted foundation models.
Appendix B states no error bars because of greedy decoding, treating a single run as deterministic. This assumes prompts and models are deterministic enough and that sampling variance is negligible.
ad hoc to paper The prompt templates and few-shot example counts are representative of how these foundation models should be used.
The prompts in Appendix A are designed by the authors; the choice of few-shot counts is not systematically tuned or justified. Results depend on these choices.
domain assumption Accuracy on the benchmark test sets is the primary measure of generalization.
The paper equates higher accuracy with better generalization, and the data-pitfall argument relies on accuracy drops under noise. Differential accuracy may not capture all aspects of overfitting.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models." pith.science (2026). https://pith.science/paper/ECVRHHRC

@misc{pith2026250524874,
  author       = {Pith},
  title        = {Pith review of: The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ECVRHHRC}},
  note         = {Machine review of arXiv:2505.24874}
}

read the original abstract

Neuro-symbolic learning was proposed to address challenges with training neural networks for complex reasoning tasks with the added benefits of interpretability, reliability, and efficiency. Neuro-symbolic learning methods traditionally train neural models in conjunction with symbolic programs, but they face significant challenges that limit them to simplistic problems. On the other hand, purely-neural foundation models now reach state-of-the-art performance through prompting rather than training, but they are often unreliable and lack interpretability. Supplementing foundation models with symbolic programs, which we call neuro-symbolic prompting, provides a way to use these models for complex reasoning tasks. Doing so raises the question: What role does specialized model training as part of neuro-symbolic learning have in the age of foundation models? To explore this question, we highlight three pitfalls of traditional neuro-symbolic learning with respect to the compute, data, and programs leading to generalization problems. This position paper argues that foundation models enable generalizable neuro-symbolic solutions, offering a path towards achieving the original goals of neuro-symbolic learning without the downsides of training from scratch.

Figures

Figures reproduced from arXiv: 2505.24874 by the authors.

**Figure 2.** Performance of neuro-symbolic prompting for four benchmarks as model size increases compared [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Examples of the data pitfall from four benchmarks showing cases where the neuro-symbolic prompting method using Gemini makes an “error”. In contrast, the neuro-symbolic training method is “correct.” These predictions, marked as errors from neuro-symbolic prompting, reflect cases where some symbols are ambiguous or hard to determine from the input. In contrast, the neuro-symbolic training method appears to have memor… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Neuro-symbolic trained neural models memorize dataset biases rather than learn general concepts like foundation models. correct for the cases where neuro-symbolic prompting with Gemini gets the wrong answer. Results are shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Examples of the program pitfall. All examples reflect errors made by neuro-symbolic prompting where the neuro-symbolic training method (Scallop) produced the correct answer. We see that the neurosymbolic training method reaches the correct answer for the wrong reasons…

**Figure 6.** Figure 6: Performance of end-to-end prompting of Gemini-2.0-Flash compared to neuro-symbolic prompting [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Examples of ChatGPT code execution. Generated code is executed for solving the problem, but we [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages

[1]

Phi-3 technical report: A highly capable language model locally on your phone

[Abd+24] Marah Abdin et al. “Phi-3 technical report: A highly capable language model locally on your phone”. In: arXiv preprint arXiv:2404.14219 (2024). [Ach+23] Josh Achiam et al. “Gpt-4 technical report”. In: arXiv preprint arXiv:2303.08774 (2023). [Amo+16] Dario Amodei et al. “Concrete problems in AI safety”. In: arXiv preprint arXiv:1606.06565 (2016)....

arXiv 2024
[3]

Pretrained Transformers Improve Out-of-Distribution Robustness

URL: https : //openreview.net/forum?id=Ep0TtjVoap. [Hen+20] Dan Hendrycks et al. “Pretrained Transformers Improve Out-of-Distribution Robustness”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky et al. Online: Association for Computational Linguistics, July 2020, pp. 2744–2751. DOI: 10.18653...

arXiv 2021
[5]

MNIST handwritten digit database

URL: https://openreview.net/ forum?id=S1gSrruVd4. [LCB10] Yann LeCun, Corinna Cortes, and CJ Burges. “MNIST handwritten digit database”. In: ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2 (2010). [Li+20] Qing Li et al. “Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning”. In: I...

work page 2010
[9]

Faithful Chain-of-Thought Reasoning

2024, pp. 10635–10644. [Lyu+23] Qing Lyu et al. “Faithful Chain-of-Thought Reasoning”. In:The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023)

work page 2024
[10]

Dolphin: A Programmable Framework for Scalable Neurosymbolic Learn- ing

URL: https : //openreview.net/forum?id=pDcM1k7mgZ. [Nai+24] Aaditya Naik et al. “Dolphin: A Programmable Framework for Scalable Neurosymbolic Learn- ing”. In: arXiv preprint arXiv:2410.03348 (2024). [Ope25] OpenAI. “Computer-Using Agent: Introducing a universal interface for AI to interact with the digital world”. In: (2025). URL: https://openai.com/index...

arXiv 2024
[11]

by Houda Bouamor, Juan Pino, and Kalika Bali

Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 3806–3824. DOI: 10 . 18653 / v1 / 2023 . findings - emnlp.248. URL: https://aclanthology.org/2023.findings-emnlp.248/. [Pro24] Prolific. Prolific. https://www.prolific.com. Accessed: 2025-05-22. London, UK,

work page 2023
[12]

Drum: End-to-end differentiable rule mining on knowledge graphs

URL: https://openreview.net/forum?id=F6RPYDUIZr. [Sad+19] Ali Sadeghian et al. “Drum: End-to-end differentiable rule mining on knowledge graphs”. In: Advances in neural information processing systems 32 (2019). [Sin+19] Koustuv Sinha et al. “CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text”. In: Proceedings of the 2019 Conference on Empiri...

arXiv 2019
[13]

Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations

URL: https://openreview. net/forum?id=QXQY58xU25. [SSK21] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. “Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 3619–3629. [Sun+21] Jiankai Sun et a...

work page 2021

Show all 16 references

[16]

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

[Yue+24] Xiang Yue et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 9556–9567. 14 A Prompts The prompt we use for the foun...

2024
[155]

Gemini: a family of highly capable multimodal models

Proceedings of Machine Learning Research. PMLR, 16–18 Nov 2021, pp. 21–30. URL: https://proceedings.mlr.press/v155/sun21a.html. [Tea+23] Gemini Team et al. “Gemini: a family of highly capable multimodal models”. In: arXiv preprint arXiv:2312.11805 (2023). [Tea25] Qwen Team. Qw...

2023 arXiv
[202]

Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts

Proceedings of Machine Learning Research. PMLR, 23–29 Jul 2023, pp. 23915–23936. URL: https : / / proceedings . mlr . press / v202 / marconato23a.html. 12 [Mar+23b] Emanuele Marconato et al. “Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoni...

2023
[2015]

Knowledge-based artificial neural networks

URL: https : / / landscapeplants.aub.edu.lb/. [TS94] Geoffrey G Towell and Jude W Shavlik. “Knowledge-based artificial neural networks”. In: Artificial intelligence 70.1-2 (1994), pp. 119–165. [VSK24] Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. “LLMs Still Can’t...

1994 arXiv
[2020]

Learn to Explain Efficiently via Neural Logic Inductive Learning

Ed. by Christian Bessiere. ijcai.org, 2020, pp. 1755–1762. DOI: 10.24963/IJCAI.2020/243. URL: https://doi.org/10.24963/ijcai.2020/243. [YS20] Yuan Yang and Le Song. “Learn to Explain Efficiently via Neural Logic Inductive Learning”. In: International Conference on Learning Rep...

2020 doi
[2023]

Deepproblog: Neural probabilistic logic programming

[Man+18] Robin Manhaeve et al. “Deepproblog: Neural probabilistic logic programming”. In: Advances in neural information processing systems 31 (2018). [Mao+19] Jiayuan Mao et al. “The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Superv...

2018
[2024]

Language models are few-shot learners

URL: https://openreview.net/forum?id=5VtI484yVy. [Bro+20] Tom Brown et al. “Language models are few-shot learners”. In:Advances in neural information processing systems 33 (2020), pp. 1877–1901. [Cha+21] Swarat Chaudhuri et al. “Neurosymbolic programming”. In: Foundations and ...

2020 arXiv
[2025]

Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

URL: https : / / openreview . net / forum ? id = 0K1OaL6XuK. [Joh+17] Justin Johnson et al. “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 29...

2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Phi-3 technical report: A highly capable language model locally on your phone

[Abd+24] Marah Abdin et al. “Phi-3 technical report: A highly capable language model locally on your phone”. In: arXiv preprint arXiv:2404.14219 (2024). [Ach+23] Josh Achiam et al. “Gpt-4 technical report”. In: arXiv preprint arXiv:2303.08774 (2023). [Amo+16] Dario Amodei et al. “Concrete problems in AI safety”. In: arXiv preprint arXiv:1606.06565 (2016)....

arXiv 2024

[2] [3]

Pretrained Transformers Improve Out-of-Distribution Robustness

URL: https : //openreview.net/forum?id=Ep0TtjVoap. [Hen+20] Dan Hendrycks et al. “Pretrained Transformers Improve Out-of-Distribution Robustness”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky et al. Online: Association for Computational Linguistics, July 2020, pp. 2744–2751. DOI: 10.18653...

arXiv 2021

[3] [5]

MNIST handwritten digit database

URL: https://openreview.net/ forum?id=S1gSrruVd4. [LCB10] Yann LeCun, Corinna Cortes, and CJ Burges. “MNIST handwritten digit database”. In: ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2 (2010). [Li+20] Qing Li et al. “Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning”. In: I...

work page 2010

[4] [9]

Faithful Chain-of-Thought Reasoning

2024, pp. 10635–10644. [Lyu+23] Qing Lyu et al. “Faithful Chain-of-Thought Reasoning”. In:The 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023)

work page 2024

[5] [10]

Dolphin: A Programmable Framework for Scalable Neurosymbolic Learn- ing

URL: https : //openreview.net/forum?id=pDcM1k7mgZ. [Nai+24] Aaditya Naik et al. “Dolphin: A Programmable Framework for Scalable Neurosymbolic Learn- ing”. In: arXiv preprint arXiv:2410.03348 (2024). [Ope25] OpenAI. “Computer-Using Agent: Introducing a universal interface for AI to interact with the digital world”. In: (2025). URL: https://openai.com/index...

arXiv 2024

[6] [11]

by Houda Bouamor, Juan Pino, and Kalika Bali

Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 3806–3824. DOI: 10 . 18653 / v1 / 2023 . findings - emnlp.248. URL: https://aclanthology.org/2023.findings-emnlp.248/. [Pro24] Prolific. Prolific. https://www.prolific.com. Accessed: 2025-05-22. London, UK,

work page 2023

[7] [12]

Drum: End-to-end differentiable rule mining on knowledge graphs

URL: https://openreview.net/forum?id=F6RPYDUIZr. [Sad+19] Ali Sadeghian et al. “Drum: End-to-end differentiable rule mining on knowledge graphs”. In: Advances in neural information processing systems 32 (2019). [Sin+19] Koustuv Sinha et al. “CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text”. In: Proceedings of the 2019 Conference on Empiri...

arXiv 2019

[8] [13]

Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations

URL: https://openreview. net/forum?id=QXQY58xU25. [SSK21] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. “Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 3619–3629. [Sun+21] Jiankai Sun et a...

work page 2021