REVIEW 3 major objections 19 references
Can VLMs Reason Robustly? A Neuro-Symbolic Investigation
T0 review · 3 major / 0 minor · reviewed 2026-07-13 · grok-4.5
Pith's one-line read Fine-tuned VLMs hit high in-distribution accuracy but miss the rules under object-count shifts; compiling those rules into circuits over VLM concepts yields consistently higher out-of-distribution accuracy.
desk verdict Clean controlled demo that end-to-end VLM fine-tuning fails object-count OOD while a simple VLM+SDD pipeline tracks concept accuracy; useful, scoped, not oversold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
VLC: a two-stage pipeline in which a vision-language model first recognizes discrete object concepts from the image, and a Boolean circuit (an SDD compiled from the symbolic rules) then evaluates those rules exactly over the recognized concepts to produce the answer.
What would settle it
Run the same VLC pipeline on a visual deductive task whose rules are withheld or supplied only in natural language (no symbolic compilation), or deliberately corrupt concept extraction so circuit inputs are systematically wrong; if out-of-distribution accuracy stays high, the claim that exact circuit encoding drives the robustness does not hold.
Extended reading notes
Core claim
Gradient-based end-to-end fine-tuning of VLMs does not reliably induce the underlying reasoning function: models achieve high in-distribution accuracy on visual deductive tasks yet fail under covariate shifts that change only the number of objects while leaving the rules fixed. VLC—VLM concept recognition followed by a circuit that compiles and executes the true task rules exactly—consistently attains higher out-of-distribution task accuracy than end-to-end fine-tuning, Prism, and ViperGPT across arithmetic addition, logical XOR, and relational-check rule sets.
Load-bearing premise
The true task rules must already be known in symbolic form so they can be compiled once into an exact circuit, and the VLM’s concept outputs must be parseable into the discrete binary inputs that circuit expects.
Editorial extensions
If this is right
- Structurally encoding the true reasoning function in a circuit makes task accuracy depend only on concept-recognition accuracy.
- Fine-tuning the VLM for concept recognition (not end-to-end reasoning) raises both in- and out-of-distribution concept accuracy and therefore VLC task accuracy.
- Scaling VLM size improves concept recognition but does not reliably improve end-to-end reasoning for every rule family (for example XOR).
- Black-box reasoning components—an LLM in Prism or multi-model programs in ViperGPT—produce inconsistent accuracy across distinct rule sets.
Reading between the lines
- If concept extractors were swapped for cheaper specialized detectors while keeping the same circuit interface, the robustness gains might transfer beyond foundation VLMs.
- Automatic compilation of circuits from natural-language rules—an extension the paper itself flags—would make the method usable when only verbal instructions exist.
- The controlled “more objects” shift may understate failures from appearance shifts (style, occlusion), suggesting a natural next stress test of the same pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether Vision-Language Models reason robustly under covariate shifts in which the perceptual input distribution (number of objects per image) changes while the underlying prediction rules stay fixed. Using three visual deductive reasoning tasks—MNAdd (arithmetic addition), MNLogic (logical XOR), and KandLogic (relational color-shape check)—generated via the NESY benchmark suite, the authors show that end-to-end fine-tuned VLMs reach high in-distribution accuracy but collapse or stagnate out of distribution (Table 1). Black-box neuro-symbolic baselines Prism (VLM + LLM) and ViperGPT (program generation + pretrained tools) are inconsistent across tasks. The proposed method, VLC, decouples VLM-based concept recognition from circuit-based symbolic reasoning: task rules are compiled once into an SDD that executes exactly over discrete concepts extracted from the VLM. Empirically, VLC’s task accuracy tracks concept accuracy and is more robust under object-count shifts; ablations cover recognition fine-tuning and model scaling. Code is released.
Significance. The central empirical finding—that gradient-based end-to-end fine-tuning on these tasks does not induce the underlying reasoning function under a controlled object-count covariate shift—is clearly demonstrated and is of genuine interest to the VLM and neuro-symbolic communities. The experimental design is clean: same rule, varying object count, reported means and standard deviations over five seeds, and a transparent two-stage architecture. Releasing code and stating limitations (known symbolic rules, parseable concept outputs, task-specific circuits) in Appendix A are strengths. Within the deliberately simple synthetic setting the contribution is solid; external validity to open-ended multimodal reasoning is limited by construction, which the authors largely acknowledge. The work is a useful, falsifiable stress test rather than a general theory of VLM reasoning.
major comments (3)
- Table 1 and §5.2: The claim that end-to-end fine-tuning “does not reliably induce the underlying reasoning function” is supported by the OOD drop, but the only controlled factor is object count (3→5→7). This confounds “failure to learn the rule” with known length/compositionality failures of transformers. A short control—e.g., training on mixed lengths, or evaluating chain-of-thought / scratchpad prompting on the same OOD splits—would strengthen the causal attribution to missing rule induction rather than sequence-length generalization alone.
- Table 2 (MNLogic 5dgt/7dgt, before ft): Task accuracy exceeds concept accuracy (e.g., 72.78% task vs 64.50% concept on 5dgt). Under exact circuit execution this is only possible if incorrect concept assignments still yield the correct XOR by chance. The paper should quantify or discuss this partial-correctness effect; otherwise the statement that “performance of VLC is dependent solely on the VLM’s recognition capability” (§5.2) is slightly overstated for parity-like rules.
- §5.1 Baselines / Appendix E: ViperGPT is given GPT-4o for code generation and GroundingDINO for detection, while End2end, Prism, and VLC use a 7B Qwen stack. The qualitative diagnosis of detection failures is helpful, but the quantitative comparison in Table 1 mixes model capacity and tooling. Either re-run ViperGPT with a matched open 7B code model (as partially done with CodeLlama in Appendix E) as the primary number, or clearly frame ViperGPT as an upper-bound tool-using system rather than a peer 7B baseline.
Circularity Check
No significant circularity: VLC deliberately compiles known ground-truth rules into an exact circuit; empirical OOD comparisons are independent of that construction.
full rationale
The paper's central empirical claim is that end-to-end fine-tuned VLMs achieve high in-distribution accuracy but fail under controlled object-count covariate shifts (Table 1), while VLC's task accuracy tracks concept accuracy because the SDD executes the supplied rules exactly. Encoding the true reasoning function into the circuit is an explicit design choice (Section 3, Phase II; Appendix A), not a hidden derivation that reduces a prediction to a fitted free parameter or a self-citation uniqueness theorem. Baselines (End2end FT, Prism, ViperGPT) are evaluated on the same held-out OOD splits; no equation equates a reported accuracy to a training-set fit by construction. Self-citations (e.g., Chen et al. 2025a on two-stage neuro-symbolic pipelines) supply architectural context and are not load-bearing for the OOD numbers. The acknowledged scope limits (rules known a priori in symbolic form; reliable parse of VLM concept strings) are stated openly and do not create circularity within the paper's stated claims. Score 0 is therefore appropriate.
Assumptions & free parameters
free parameters (2)
- number of few-shot examples =
5
- fine-tuning learning rate / epochs / batch schedule =
1e-5 / 5 epochs
assumptions (4)
- domain assumption Covariate shift is realized solely by changing the number of objects per image while the label-generating rule remains fixed.
- standard math Task rules admit exact compilation into Boolean circuits (specifically SDDs) that evaluate deterministically on discrete concept assignments.
- domain assumption VLM textual outputs under few-shot prompting can be parsed into the binary concept vectors required by the circuit.
- ad hoc to paper Symbolic rules (or equivalent natural-language descriptions) are supplied as input rather than induced from data.
invented entities (1)
-
VLC pipeline
Cite this review
Pith. "Pith review of Can VLMs Reason Robustly? A Neuro-Symbolic Investigation." pith.science (2026). https://pith.science/paper/SHDKD7YV
@misc{pith2026260323867,
author = {Pith},
title = {Pith review of: Can VLMs Reason Robustly? A Neuro-Symbolic Investigation},
year = {2026},
howpublished = {\url{https://pith.science/paper/SHDKD7YV}},
note = {Machine review of arXiv:2603.23867}
}
read the original abstract
Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC, a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three simple visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves higher task accuracy on out-of-distribution data than other reasoning paradigms. Code is available at https://github.com/uiuctml/VLC.
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[2]
Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024a
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024a. Weixin Chen, Simon Yu, Huajie Shao, Lui Sha, and Han Zhao. Neural probabilistic circuits: Enabling compositional and interpretable predict...
-
[3]
On the measure of intelligence.arXiv preprint arXiv:1911.01547,
François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547,
arXiv 1911
-
[4]
Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passerini, Stefano Teso, and Emanuele Mar- conato. If concept bottlenecks are the question, are foundation models the answer?arXiv preprint arXiv:2504.19774,
-
[5]
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech...
-
[6]
Neptune: A neuro-pythonic framework for tunable compositional reasoning on vision-language
Danial Kamali and Parisa Kordjamshidi. Neptune: A neuro-pythonic framework for tunable compositional reasoning on vision-language. InNeuIPS 2025 Workshop on Space in Vision, Language, and Embodied AI,
2025
-
[7]
Nicolas Lazzari, Valentina Presutti, and Antonio Vergari. To neuro-symbolic classification and beyond by compiling description logic ontologies to probabilistic circuits.arXiv preprint arXiv:2601.14894,
-
[8]
Boosting deductive reasoning with step signals in rlhf.arXiv preprint arXiv:2410.09528, 2024a
Jialian Li, Yipin Zhang, Wei Shen, Yuzi Yan, Jian Xie, and Dong Yan. Boosting deductive reasoning with step signals in rlhf.arXiv preprint arXiv:2410.09528, 2024a. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified vision-language understanding and generation. InInternational Conference ...
Show all 19 references
-
[9]
Symbol grounding in neuro-symbolic ai: A gentle introduction to reasoning shortcuts.arXiv preprint arXiv:2510.14538,
Emanuele Marconato, Samuele Bortolotti, Emile van Krieken, Paolo Morettin, Elena Umili, Antonio Vergari, Efthymia Tsamoura, Andrea Passerini, and Stefano Teso. Symbol grounding in neuro-symbolic ai: A gentle introduction to reasoning shortcuts.arXiv preprint arXiv:2510.14538,
-
[10]
Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman. Certified deductive reasoning with language models.Trans. Mach. Learn. Res., 2024,
2024
-
[11]
Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950,
14 Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950,
-
[12]
Proofwriter: Generating implications, proofs, and abduc- tive statements over natural language
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abduc- tive statements over natural language. InACL/IJCNLP, volume ACL/IJCNLP 2021 ofFindings of ACL, pp. 3621–3634. Association for Computational Linguistics,
2021
-
[13]
Easyarc: Evaluating vision language models on true visual reasoning.arXiv preprint arXiv:2506.11595,
Mert Unsal and Aylin Akkus. Easyarc: Evaluating vision language models on true visual reasoning.arXiv preprint arXiv:2506.11595,
-
[14]
Language in a bottle: Language model guided concept bottlenecks for interpretable image classification
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. InCVPR, pp. 19187–19197. IEEE, 2023a. 15 Zhengyuan Yang, Linjie Li, Kevin ...
-
[15]
Vision-language models for vision tasks: A survey
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 46(8):5625–5644, 2024a. Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh M. Susskind, and Navdeep Jaitly. How...
-
[16]
Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding.arXiv preprint arXiv:2410.21311,
Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat Seng Chua. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding.arXiv preprint arXiv:2410.21311,
-
[17]
Extracting Symbolic Rules from Natural Language.In this paper, we assume access to symbolic rules for visual deductive reasoning tasks
16 A Discussions In this section, we discuss the limitations of the proposed method and provide some potential solutions. Extracting Symbolic Rules from Natural Language.In this paper, we assume access to symbolic rules for visual deductive reasoning tasks. In practice, howeve...
2024
-
[18]
ViperGPT.In ViperGPT, GPT-4o generates a program for each task based on the templates provided in Surís et al. (2023). Once generated, the same program is applied to all test samples for that task. The generated programs frequently invoke both a detection model and a VLM. Spec...
2023
-
[19]
Specifically, we use the�������benchmark to create datasets for various visual deductive reasoning tasks
D Dataset Construction In this section, we describe the construction of the datasets. Specifically, we use the�������benchmark to create datasets for various visual deductive reasoning tasks. MNAdd.Datasets for the MNAdd task consist of images depicting two multi-digit numbers...
2002
Reviewed July 13, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.