Pith. sign in

REVIEW 3 major objections 19 references

Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

T0 review · 3 major / 0 minor · reviewed 2026-07-13 · grok-4.5

Pith's one-line read Fine-tuned VLMs hit high in-distribution accuracy but miss the rules under object-count shifts; compiling those rules into circuits over VLM concepts yields consistently higher out-of-distribution accuracy.

desk verdict Clean controlled demo that end-to-end VLM fine-tuning fails object-count OOD while a simple VLM+SDD pipeline tracks concept accuracy; useful, scoped, not oversold. read the letter →

arxiv 2603.23867 v2 pith:SHDKD7YV submitted 2026-03-25 cs.LG cs.AIcs.CV

classification cs.LGcs.AIcs.CV
keywords vision-languagemodelsvisualdeductivereasoningcovariateshiftneuro-symboliccircuitsconceptrecognitionout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether vision-language models can reason when the image distribution changes but the prediction rules stay fixed. On three simple visual deductive tasks—adding multi-digit handwritten numbers, XORing binary digits, and checking that same shapes share the same color—end-to-end fine-tuning reaches high accuracy on images with few objects yet fails when images contain more objects. Neuro-symbolic baselines that still hand reasoning to a black-box language model or to generated multi-model programs also prove inconsistent across rule families. The authors introduce VLC: a VLM recognizes object concepts, then a Boolean circuit compiled from the known rules executes them exactly. Across all three tasks, this split produces higher out-of-distribution task accuracy than end-to-end fine-tuning, Prism, and ViperGPT, showing that explicitly encoding the rule—not hoping the network induces it—is what supports reliable reasoning under controlled covariate shift.

What carries the argument

VLC: a two-stage pipeline in which a vision-language model first recognizes discrete object concepts from the image, and a Boolean circuit (an SDD compiled from the symbolic rules) then evaluates those rules exactly over the recognized concepts to produce the answer.

What would settle it

Run the same VLC pipeline on a visual deductive task whose rules are withheld or supplied only in natural language (no symbolic compilation), or deliberately corrupt concept extraction so circuit inputs are systematically wrong; if out-of-distribution accuracy stays high, the claim that exact circuit encoding drives the robustness does not hold.

Watch

Extended reading notes

Core claim

Gradient-based end-to-end fine-tuning of VLMs does not reliably induce the underlying reasoning function: models achieve high in-distribution accuracy on visual deductive tasks yet fail under covariate shifts that change only the number of objects while leaving the rules fixed. VLC—VLM concept recognition followed by a circuit that compiles and executes the true task rules exactly—consistently attains higher out-of-distribution task accuracy than end-to-end fine-tuning, Prism, and ViperGPT across arithmetic addition, logical XOR, and relational-check rule sets.

Load-bearing premise

The true task rules must already be known in symbolic form so they can be compiled once into an exact circuit, and the VLM’s concept outputs must be parseable into the discrete binary inputs that circuit expects.

Editorial extensions

If this is right

  • Structurally encoding the true reasoning function in a circuit makes task accuracy depend only on concept-recognition accuracy.
  • Fine-tuning the VLM for concept recognition (not end-to-end reasoning) raises both in- and out-of-distribution concept accuracy and therefore VLC task accuracy.
  • Scaling VLM size improves concept recognition but does not reliably improve end-to-end reasoning for every rule family (for example XOR).
  • Black-box reasoning components—an LLM in Prism or multi-model programs in ViperGPT—produce inconsistent accuracy across distinct rule sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If concept extractors were swapped for cheaper specialized detectors while keeping the same circuit interface, the robustness gains might transfer beyond foundation VLMs.
  • Automatic compilation of circuits from natural-language rules—an extension the paper itself flags—would make the method usable when only verbal instructions exist.
  • The controlled “more objects” shift may understate failures from appearance shifts (style, occlusion), suggesting a natural next stress test of the same pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper investigates whether Vision-Language Models reason robustly under covariate shifts in which the perceptual input distribution (number of objects per image) changes while the underlying prediction rules stay fixed. Using three visual deductive reasoning tasks—MNAdd (arithmetic addition), MNLogic (logical XOR), and KandLogic (relational color-shape check)—generated via the NESY benchmark suite, the authors show that end-to-end fine-tuned VLMs reach high in-distribution accuracy but collapse or stagnate out of distribution (Table 1). Black-box neuro-symbolic baselines Prism (VLM + LLM) and ViperGPT (program generation + pretrained tools) are inconsistent across tasks. The proposed method, VLC, decouples VLM-based concept recognition from circuit-based symbolic reasoning: task rules are compiled once into an SDD that executes exactly over discrete concepts extracted from the VLM. Empirically, VLC’s task accuracy tracks concept accuracy and is more robust under object-count shifts; ablations cover recognition fine-tuning and model scaling. Code is released.

Significance. The central empirical finding—that gradient-based end-to-end fine-tuning on these tasks does not induce the underlying reasoning function under a controlled object-count covariate shift—is clearly demonstrated and is of genuine interest to the VLM and neuro-symbolic communities. The experimental design is clean: same rule, varying object count, reported means and standard deviations over five seeds, and a transparent two-stage architecture. Releasing code and stating limitations (known symbolic rules, parseable concept outputs, task-specific circuits) in Appendix A are strengths. Within the deliberately simple synthetic setting the contribution is solid; external validity to open-ended multimodal reasoning is limited by construction, which the authors largely acknowledge. The work is a useful, falsifiable stress test rather than a general theory of VLM reasoning.

major comments (3)
  1. Table 1 and §5.2: The claim that end-to-end fine-tuning “does not reliably induce the underlying reasoning function” is supported by the OOD drop, but the only controlled factor is object count (3→5→7). This confounds “failure to learn the rule” with known length/compositionality failures of transformers. A short control—e.g., training on mixed lengths, or evaluating chain-of-thought / scratchpad prompting on the same OOD splits—would strengthen the causal attribution to missing rule induction rather than sequence-length generalization alone.
  2. Table 2 (MNLogic 5dgt/7dgt, before ft): Task accuracy exceeds concept accuracy (e.g., 72.78% task vs 64.50% concept on 5dgt). Under exact circuit execution this is only possible if incorrect concept assignments still yield the correct XOR by chance. The paper should quantify or discuss this partial-correctness effect; otherwise the statement that “performance of VLC is dependent solely on the VLM’s recognition capability” (§5.2) is slightly overstated for parity-like rules.
  3. §5.1 Baselines / Appendix E: ViperGPT is given GPT-4o for code generation and GroundingDINO for detection, while End2end, Prism, and VLC use a 7B Qwen stack. The qualitative diagnosis of detection failures is helpful, but the quantitative comparison in Table 1 mixes model capacity and tooling. Either re-run ViperGPT with a matched open 7B code model (as partially done with CodeLlama in Appendix E) as the primary number, or clearly frame ViperGPT as an upper-bound tool-using system rather than a peer 7B baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VLC deliberately compiles known ground-truth rules into an exact circuit; empirical OOD comparisons are independent of that construction.

full rationale

The paper's central empirical claim is that end-to-end fine-tuned VLMs achieve high in-distribution accuracy but fail under controlled object-count covariate shifts (Table 1), while VLC's task accuracy tracks concept accuracy because the SDD executes the supplied rules exactly. Encoding the true reasoning function into the circuit is an explicit design choice (Section 3, Phase II; Appendix A), not a hidden derivation that reduces a prediction to a fitted free parameter or a self-citation uniqueness theorem. Baselines (End2end FT, Prism, ViperGPT) are evaluated on the same held-out OOD splits; no equation equates a reported accuracy to a training-set fit by construction. Self-citations (e.g., Chen et al. 2025a on two-stage neuro-symbolic pipelines) supply architectural context and are not load-bearing for the OOD numbers. The acknowledged scope limits (rules known a priori in symbolic form; reliable parse of VLM concept strings) are stated openly and do not create circularity within the paper's stated claims. Score 0 is therefore appropriate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper rests on standard definitions of covariate shift, the known expressiveness of Boolean circuits/SDDs, and ordinary training choices (few-shot count, learning rate, epochs). No free parameters are fitted to produce the central robustness claim; the circuits implement the ground-truth rules by construction.

free parameters (2)
  • number of few-shot examples = 5
    Fixed at 5 for all VLM prompts; chosen by authors to stabilize output format.
  • fine-tuning learning rate / epochs / batch schedule = 1e-5 / 5 epochs
    Standard AdamW settings (lr 1e-5, 5 epochs, etc.) used for both end-to-end and recognition fine-tuning; not claimed to be optimal or theoretically derived.
assumptions (4)
  • domain assumption Covariate shift is realized solely by changing the number of objects per image while the label-generating rule remains fixed.
    Defines the experimental regime in Sections 1–2 and 5.1; standard statistical notion applied to visual object count.
  • standard math Task rules admit exact compilation into Boolean circuits (specifically SDDs) that evaluate deterministically on discrete concept assignments.
    Section 3.2; relies on classical knowledge-compilation results for SDDs / NNF circuits.
  • domain assumption VLM textual outputs under few-shot prompting can be parsed into the binary concept vectors required by the circuit.
    Phase I–II interface (Section 3); assumes format consistency that the authors enforce via in-context examples.
  • ad hoc to paper Symbolic rules (or equivalent natural-language descriptions) are supplied as input rather than induced from data.
    Stated in Section 2 and revisited as a limitation in Appendix A; central to the claimed robustness guarantee.
invented entities (1)
  • VLC pipeline
    purpose: Two-stage architecture that routes VLM concept recognition into a pre-compiled circuit for exact rule execution.
    Named method introduced by the paper; independent evidence is limited to the reported synthetic experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can VLMs Reason Robustly? A Neuro-Symbolic Investigation." pith.science (2026). https://pith.science/paper/SHDKD7YV

@misc{pith2026260323867,
  author       = {Pith},
  title        = {Pith review of: Can VLMs Reason Robustly? A Neuro-Symbolic Investigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHDKD7YV}},
  note         = {Machine review of arXiv:2603.23867}
}
read the original abstract

Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC, a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three simple visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves higher task accuracy on out-of-distribution data than other reasoning paradigms. Code is available at https://github.com/uiuctml/VLC.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

  2. [2]

    Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? InNeurIPS, 2024a. Weixin Chen, Simon Yu, Huajie Shao, Lui Sha, and Han Zhao. Neural probabilistic circuits: Enabling compositional and interpretable predict...

  3. [3]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547,

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547,

  4. [4]

    If concept bottlenecks are the question, are foundation models the answer?arXiv preprint arXiv:2504.19774,

    Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passerini, Stefano Teso, and Emanuele Mar- conato. If concept bottlenecks are the question, are foundation models the answer?arXiv preprint arXiv:2504.19774,

  5. [5]

    Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech...

  6. [6]

    Neptune: A neuro-pythonic framework for tunable compositional reasoning on vision-language

    Danial Kamali and Parisa Kordjamshidi. Neptune: A neuro-pythonic framework for tunable compositional reasoning on vision-language. InNeuIPS 2025 Workshop on Space in Vision, Language, and Embodied AI,

  7. [7]

    To neuro-symbolic classification and beyond by compiling description logic ontologies to probabilistic circuits.arXiv preprint arXiv:2601.14894,

    Nicolas Lazzari, Valentina Presutti, and Antonio Vergari. To neuro-symbolic classification and beyond by compiling description logic ontologies to probabilistic circuits.arXiv preprint arXiv:2601.14894,

  8. [8]

    Boosting deductive reasoning with step signals in rlhf.arXiv preprint arXiv:2410.09528, 2024a

    Jialian Li, Yipin Zhang, Wei Shen, Yuzi Yan, Jian Xie, and Dong Yan. Boosting deductive reasoning with step signals in rlhf.arXiv preprint arXiv:2410.09528, 2024a. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified vision-language understanding and generation. InInternational Conference ...

Show all 19 references
  1. [9]

    Symbol grounding in neuro-symbolic ai: A gentle introduction to reasoning shortcuts.arXiv preprint arXiv:2510.14538,

    Emanuele Marconato, Samuele Bortolotti, Emile van Krieken, Paolo Morettin, Elena Umili, Antonio Vergari, Efthymia Tsamoura, Andrea Passerini, and Stefano Teso. Symbol grounding in neuro-symbolic ai: A gentle introduction to reasoning shortcuts.arXiv preprint arXiv:2510.14538,

  2. [10]

    Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman. Certified deductive reasoning with language models.Trans. Mach. Learn. Res., 2024,

  3. [11]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950,

    14 Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950,

  4. [12]

    Proofwriter: Generating implications, proofs, and abduc- tive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. Proofwriter: Generating implications, proofs, and abduc- tive statements over natural language. InACL/IJCNLP, volume ACL/IJCNLP 2021 ofFindings of ACL, pp. 3621–3634. Association for Computational Linguistics,

  5. [13]

    Easyarc: Evaluating vision language models on true visual reasoning.arXiv preprint arXiv:2506.11595,

    Mert Unsal and Aylin Akkus. Easyarc: Evaluating vision language models on true visual reasoning.arXiv preprint arXiv:2506.11595,

  6. [14]

    Language in a bottle: Language model guided concept bottlenecks for interpretable image classification

    Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. InCVPR, pp. 19187–19197. IEEE, 2023a. 15 Zhengyuan Yang, Linjie Li, Kevin ...

  7. [15]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 46(8):5625–5644, 2024a. Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh M. Susskind, and Navdeep Jaitly. How...

  8. [16]

    Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding.arXiv preprint arXiv:2410.21311,

    Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat Seng Chua. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding.arXiv preprint arXiv:2410.21311,

  9. [17]

    Extracting Symbolic Rules from Natural Language.In this paper, we assume access to symbolic rules for visual deductive reasoning tasks

    16 A Discussions In this section, we discuss the limitations of the proposed method and provide some potential solutions. Extracting Symbolic Rules from Natural Language.In this paper, we assume access to symbolic rules for visual deductive reasoning tasks. In practice, howeve...

  10. [18]

    ViperGPT.In ViperGPT, GPT-4o generates a program for each task based on the templates provided in Surís et al. (2023). Once generated, the same program is applied to all test samples for that task. The generated programs frequently invoke both a detection model and a VLM. Spec...

  11. [19]

    Specifically, we use the�������benchmark to create datasets for various visual deductive reasoning tasks

    D Dataset Construction In this section, we describe the construction of the datasets. Specifically, we use the�������benchmark to create datasets for various visual deductive reasoning tasks. MNAdd.Datasets for the MNAdd task consist of images depicting two multi-digit numbers...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.