Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A text-first pipeline that converts images into independent descriptions claims new state-of-the-art results on three visual reasoning benchmarks.

desk verdict The cognitive framework and the perception-bottleneck ablation are solid, but the SOTA claims are unsupported by same-set comparisons and need major revision. read the letter →

arxiv 2501.13620 v5 pith:66L4HAMM submitted 2025-01-23 cs.CV cs.AI

Mohit Vaishnav , Tanel Tammet This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelsBongardproblemsvisualreasoningcomponentialanalysisperception-reasoningbottleneckWinogroundchain-of-thoughtpromptingbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the reason many vision-language models fail on abstract visual reasoning is not weak reasoning but weak visual perception. It introduces three evaluation paradigms, and shows that when images are first converted into detailed, task-agnostic text descriptions and a powerful language model reasons only over those descriptions, accuracy on Bongard-OpenWorld, Bongard-HOI, and Winoground rises to new state-of-the-art levels. The authors also show that feeding high-quality external descriptions to open-source VLMs sharply improves their reasoning accuracy, which they take as evidence of a perception bottleneck. If right, this reframes where future work should concentrate: improving the perceptual front end rather than the reasoning engine.

What carries the argument

Componential Analysis (CA) is the two-stage procedure that carries the argument. Stage 1 prompts a VLM to produce a detailed, hierarchical JSON description of each image—scene, objects, activities, context, visual patterns, emotional undertones, visible text, and a summary—without showing the task or captions. Stage 2 gives all thirteen descriptions to a reasoning model, labeled positive, negative, and query, and asks it to derive the rule and classify. Because reasoning sees only text, this mechanism separates perception from reasoning and lets single-image models and text-only LLMs participate in multi-image benchmarks.

What would settle it

Run the CA protocol on the full 1,001-case Bongard-OpenWorld test set and on the original Bongard-HOI split sizes, with the same GPT-4o and Gemini 2.0 pipelines, and compare against the reported 92.8% and 93.6% accuracies; if the full-set scores fall below the cited human averages or prior state of the art, the claim that CA is state of the art would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the perception-reasoning interface, not abstract reasoning, is the main bottleneck for many VLMs on these tasks. In the Componential Analysis (CA) paradigm, where images become independent structured text descriptions before any reasoning, GPT-4o reaches 92.8% and Gemini 2.0 93.6% on the Bongard-OpenWorld subset, exceeding the 91.0% human average reported by the benchmark's authors. The same paradigm yields 77.3% average accuracy for GPT-4o on Bongard-HOI and a 52.0% Winoground group score, both described as new state of the art. When high-quality external descriptions are substituted for a model's own descriptions, weaker open-source VLMs improve sharply, which the paper takes as direct evidence that their perceptual front end is the limiting stage.

Load-bearing premise

The state-of-the-art claim depends on the hand-built balanced subsets of Bongard-OpenWorld (500 cases from the first 250 samples) and Bongard-HOI (100 per split) being as hard as, and directly comparable to, the full benchmarks and the human averages reported in the original papers.

Editorial extensions

If this is right

  • End-to-end evaluations understate the reasoning ability of open-source VLMs; with external descriptions, Llama-Vision-11B rises from 53.4% to 84.17% on Bongard-OpenWorld.
  • CA turns multi-image benchmarks into text tasks, so single-image VLMs and text-only LLMs can be evaluated on Bongard problems; text-only Phi-4 reaches 91.98% with GPT-4o descriptions.
  • For GPT-4o and Gemini 2.0, accuracy increases from DVRL to DRL to CA, indicating that stage separation helps current architectures.
  • On Winoground, task-agnostic description generation outperforms context-conditioned chain-of-thought prompts, supporting decoupled perception and reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the Bongard-OpenWorld result is measured on a balanced 500-case subset drawn from the first 250 examples, the state-of-the-art claim would need re-checking on the full 1,001-case set; the same holds for the 100-per-split Bongard-HOI subsets.
  • The perception-bottleneck thesis predicts a monotone relationship between description fidelity and CA accuracy; a formal test would vary description quality continuously and measure downstream accuracy, which the paper only samples at two quality levels.
  • The same two-stage pipeline could be extended to other relational benchmarks such as Raven's Progressive Matrices; strong transfer would support the general modularity claim, whereas weak transfer would bound it to natural-image commonsense tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a cognitive-science-inspired evaluation framework for vision-language models (VLMs), consisting of three paradigms: Direct Visual Rule Learning (DVRL), Deductive Rule Learning (DRL), and Componential Analysis (CA). The framework is applied to Bongard-OpenWorld, Bongard-HOI, and Winoground. The central claim is that CA, which reasons over task-agnostic textual image descriptions, achieves new state-of-the-art results on these benchmarks and reveals a perception bottleneck in open-source VLMs. Ablation studies, where high-quality GPT-4o descriptions are supplied to other models, show large accuracy gains and enable text-only LLMs to perform visual reasoning tasks.

Significance. If the findings hold, the paper makes a useful diagnostic contribution: the three paradigms provide a structured way to separate perceptual encoding from textual reasoning, and the external-description ablation in Section 7.2 is a clean demonstration that many open VLMs are limited more by perception than by reasoning. The idea of decoupling perception via rich textual descriptions is promising and could influence both evaluation methodology and architecture design. However, the headline SOTA claims are not yet supported by the evidence as presented, because the comparisons to prior work and human baselines are made on nonstandard, unreleased subsets. The framework itself is plausible and the bottleneck finding survives the subset issues, but the empirical basis for the SOTA claim needs substantial revision.

major comments (4)
  1. [Section 6.1, Table 1; Appendix A.3.1] The claim of state-of-the-art performance on Bongard-OpenWorld is not supported by the current comparison protocol. The evaluation uses a nonstandard 500-case subset constructed from the first 250 samples with one positive and one negative query per context set, and Table A.2 shows that category '0' comprises 73% of the subset. The human average of 91% is cited from [8] for the full benchmark, not for this subset, and no prior model is evaluated on this subset. Without rerunning prior methods and human baselines on the exact same 500 cases, or releasing sample IDs so others can do so, the accuracy of 92.8-93.6% cannot be labeled SOTA or described as surpassing the human average.
  2. [Section 6.2, Table 2; Appendix A.3.2] The Bongard-HOI results have the same comparability problem. The balanced 100-per-split subsets are not the original test splits (for example, soua has 2236 positive and 1348 negative samples in the original release), and the human averages in Table 2 are taken from [9] for the full splits. No prior model results are reported on the balanced subsets. Consequently, the statement that CA outperforms prior published methods on Bongard-HOI is not empirically established; the authors should either evaluate prior baselines on the identical subsets or explicitly restrict their claims to the new subsets and release them.
  3. [Section 6.3, Table 3; Table A.4] The Winoground SOTA claim is under-supported and contains an internal inconsistency. Table 3 reports Gemini 2.0 CA scores of 71.00 Text, 48.75 Image, and 42.00 Group, while Table A.4 reports the same configuration as 71.91, 48.71, and 42.01. Table A.4 also omits the MMICL+CoCoT baseline that appears in Table 3, and it does not include a broader set of published Winoground leaderboard results, so the statement that CA 'significantly surpasses previously reported SOTA' lacks a same-protocol comparison. The authors should reconcile the two tables and provide a complete comparison against all relevant prior methods on the same 400 samples.
  4. [Sections 5.3, 7.2; Abstract] The claim that CA 'isolates reasoning from perception' is only partially realized in the primary evaluations of Tables 1 and 2, where the same model produces the descriptions and performs the reasoning; perceptual errors in that model's own descriptions can still influence the reasoning stage. Full decoupling is achieved only in the Section 7.2 ablation, where GPT-4o-generated descriptions are given to other models. The paper should either present external-description results for GPT-4o and Gemini 2.0 as well, or qualify the isolation claim so that it applies to the external-description setting rather than to the default CA protocol used for the headline numbers.
minor comments (5)
  1. [Appendix A.3.1, A.3.4] The dataset-availability text is inconsistent: A.3.1 says 'Specific sample IDs used will be released' while A.3.4 says details 'will be released upon publication'; please state clearly when and where the subsets will be released.
  2. [Section 6.3; Appendix A.4.4] The Winoground results are presented in Section 6.3, but Appendix A.4.4 incorrectly refers to 'Section 6.2' as the location of the Winoground evaluation; please correct the cross-reference.
  3. [Table 3 vs Table A.4] The duplicate reporting of the Gemini 2.0 CA Winoground scores in Tables 3 and A.4 with different values is a presentation error that must be fixed; the same configuration should have one set of numbers.
  4. [Appendix A.4.1 vs Table 5] Appendix A.4.1 lists Deepseek-r1 as 32B and 70B, but Table 5 reports 'Deepseek-r1:14b'; please align the model sizes across the appendix and the main table.
  5. [Throughout] Several minor typographical issues should be cleaned up, including the inconsistent spacing in 'LLaV A' and the stray text 'Omit the ‘`` tags' repeated in the prompts in Appendix A.5.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: reported accuracies are direct evaluations, not fitted quantities.

full rationale

The paper's derivation chain consists of running frozen VLMs and LLMs under three prompting paradigms and reporting raw classification accuracies; no parameter is fitted to the reported outcomes, no predicted quantity is defined in terms of the benchmark labels, and no benchmark claim is justified by the authors' own prior results. Equations (1)-(3) in Appendix A.4.4 merely define the standard Winoground text/image/group scores over the model's explicit matching choices, so they do not encode the ground-truth answer. The only self-citation ([56]) appears in an appendix sentence about future attention-map analysis and is not load-bearing for the SOTA or perception-bottleneck conclusions. The hand-built subsets (Appendix A.3.1-A.3.2) compared with human averages from [8,9] raise an apples-to-oranges benchmark-comparability concern but do not make any result equivalent to its input by construction. Similarly, the partial decoupling caveat that primary CA runs use the model's own descriptions is a validity limitation, explicitly addressed by the external-description ablation in Section 7.2, not a circularity. This is a non-finding: no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no mathematical free parameters or new physical entities. Its load-bearing choices are the evaluation subset, the description prompt design, and the assumption that external GPT-4o descriptions are a neutral perceptual ground truth. These are experimental design choices rather than fitted constants.

free parameters (1)
  • Bongard-OpenWorld evaluation subset
    The paper uses a hand-constructed 500-case subset from the first 250 source samples, with one positive and one negative query per context set. This selection is ad hoc and may not be representative of the full benchmark, affecting comparability to human averages and prior SOTA numbers.
assumptions (4)
  • domain assumption The 500-case Bongard-OpenWorld subset is representative of the full benchmark and comparable to the human average from [8].
    Section 4 and Appendix A.3.1 construct the subset from the first 250 samples; the paper compares its accuracies to the human average (91%) from [8], which may have been computed on a different sample.
  • domain assumption Temperature-0 API outputs are deterministic and stable across calls and model versions.
    All results are single-run accuracies without variance estimates; Appendix A.4.2 specifies temperature 0 but does not address API version drift or nondeterminism.
  • domain assumption Per-image task-agnostic descriptions do not inadvertently encode cross-image rule information.
    The CA paradigm's decoupling claim rests on descriptions being generated without knowledge of the rule set. The A.5.3.1 prompt asks for a generic description, but a powerful model may still apply implicit prior knowledge of the benchmark. Section 9 acknowledges this fragility.
  • ad hoc to paper External descriptions generated by GPT-4o are high-fidelity and independent of each reasoning model.
    Section 7.2 uses GPT-4o descriptions as a common surrogate for all models. If those descriptions contain task-specific cues or omit information needed by weaker models, the observed reasoning improvement may overstate the perception bottleneck.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs." pith.science (2026). https://pith.science/paper/66L4HAMM

@misc{pith2026250113620,
  author       = {Pith},
  title        = {Pith review of: A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/66L4HAMM}},
  note         = {Machine review of arXiv:2501.13620}
}
read the original abstract

A fundamental challenge in artificial intelligence involves understanding the cognitive mechanisms underlying visual reasoning in sophisticated models like Vision-Language Models (VLMs). How do these models integrate visual perception with abstract thought, especially when reasoning across multiple images or requiring fine-grained compositional understanding? Drawing inspiration from cognitive science, this paper introduces a structured evaluation framework using diverse visual reasoning tasks-Bongard Problems (BPs) and Winoground-to dissect the perception-reasoning interface in VLMs. We propose three distinct evaluation paradigms, mirroring human problem-solving strategies: Direct Visual Rule Learning (DVRL; holistic processing), Deductive Rule Learning (DRL; rule extraction and application), and Componential Analysis (CA; analytical decomposition via task-agnostic textual descriptions). These paradigms systematically vary cognitive load and probe processing stages. Notably, CA enables multi-image reasoning evaluation even for single-image architectures and isolates reasoning from perception by operating on textual descriptions. Applying this framework, we demonstrate that CA, leveraging powerful language models for reasoning over rich, independently generated descriptions, achieves new state-of-the-art (SOTA) performance on challenging benchmarks including Bongard-OpenWorld, Bongard-HOI, and Winoground. Ablation studies confirm reasoning improves significantly when perceptual challenges are mitigated, revealing a critical perception bottleneck. Our framework provides a valuable diagnostic tool and suggests that decoupling perception (via rich, task-agnostic description) from reasoning is a promising direction for robust and general visual intelligence.

Figures

Figures reproduced from arXiv: 2501.13620 by the authors.

Figure 1
Figure 1. Example Bongard-OpenWorld task. Left: Positive examples. Center: Negative examples. Right: Query. Rule: A group photo at a wedding reception. Query is negative. (3 of 6 examples shown per set). Our primary testbed is the Bongard-OpenWorld dataset [8]. We selected this benchmark for its use of natural, real-world images and its require￾ment for few-shot abstract reasoning based on commonsense visual concepts, alignin… view at source ↗
Figure 2
Figure 2. provides a schematic overview. Specific prompts are detailed in Appendix A.5. VLM Analysis Rule Identified Query Image Description Category (pos or neg) ` ` ` Negative Images Query Image Componential Analysis Deductive Rule Learning Direct Visual Rule Learning VLM Rule Identified VLM Analysis Rule Identified Query Image Description Category (pos or neg) Query Image Structured Image Description VLM LLM Analysis Rule … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MMTABREAL: Real-World Benchmark for Multimodal Table Understanding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    The paper releases a 500-table, 4,021-question benchmark of real-world multimodal tables and shows that leading vision-language models drop 20-40% in accuracy relative to earlier synthetic benchmarks.

Reference graph

Works this paper leans on

73 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [8]

    Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world

    Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chun Zhu, and Yizhou Wang. Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  2. [9]

    Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions

    Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19056–19065, 2022

    work page 2022

  3. [1]

    Ai, visual imagery, and a case study on the challenges posed by human intelligence tests

    Maithilee Kunda. Ai, visual imagery, and a case study on the challenges posed by human intelligence tests. Proceedings of the National Academy of Sciences, 117(47):29390–29397, 9 2020

    work page 2020

  4. [2]

    Building machines that learn and think like people

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017

    2017

  5. [3]

    What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10424, 2024

    Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10424, 2024

    arXiv 2024

  6. [4]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  7. [5]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  8. [6]

    The recognition problem

    MM Bongard. The recognition problem. tech. rep. 1968

    work page 1968

Show all 73 references
  1. [7]

    Fluid concepts and creative analogies: Computer models of the funda- mental mechanisms of thought

    Douglas R Hofstadter. Fluid concepts and creative analogies: Computer models of the funda- mental mechanisms of thought. Basic books, 1995

    1995

  2. [10]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  3. [11]

    Biederman

    I. Biederman. Recognition-by-components: A theory of human image understanding. Psycho- logical Review, 94:115–147, 1987

    1987

  4. [12]

    The psychology of proof: Deductive reasoning in human thinking

    Lance J Rips. The psychology of proof: Deductive reasoning in human thinking . Mit Press, 1994

    1994

  5. [13]

    Gluck, Russell A

    Mark A. Gluck, Russell A. Poldrack, and Szabolcs Kéri. The cognitive neuroscience of category learning. Neuroscience & Biobehavioral Reviews , 32(2):193–196, 2008. The Cognitive Neuroscience of Category Learning

    2008

  6. [14]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  7. [15]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608–3...

    2018

  8. [16]

    What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models

    Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, and Bingchen Zhao. What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  9. [17]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–25...

    2022

  10. [18]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems...

    2024

  11. [19]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Pr...

    2024

  12. [20]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023

    2023 arXiv

  13. [21]

    Fine-tuning multimodal llms to follow zero-shot demonstrative instructions

    Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2023

    2023

  14. [22]

    Mmiu: Multimodal multi-image understanding for evaluating large vision-language models

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718, 2024

    2024 arXiv

  15. [23]

    Measuring abstract reasoning in neural networks

    David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pages 511–520. PMLR, 2018

    2018

  16. [24]

    Raven: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5317–5327, 2019

    2019

  17. [25]

    Human problem solving, volume 104

    Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104. Prentice- hall Englewood Cliffs, NJ, 1972

    1972

  18. [26]

    Neural substrates of similarity and rule-based strategies in judgment

    Bettina von Helversen, Linnea Karlsson, Björn Rasch, and Jörg Rieskamp. Neural substrates of similarity and rule-based strategies in judgment. Frontiers in human neuroscience, 8:809, 2014

    2014

  19. [27]

    Complex cognition: the science of human reasoning, problem- solving, and decision-making, 2010

    Markus Knauff and Ann G Wolf. Complex cognition: the science of human reasoning, problem- solving, and decision-making, 2010

    2010

  20. [28]

    The challenges of componential analysis: Cognitive and metacognitive instruction in mathematical problem solving

    Nancy L Hutchinson. The challenges of componential analysis: Cognitive and metacognitive instruction in mathematical problem solving. Journal of learning disabilities, 25(4):249–252, 1992

    1992

  21. [29]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  22. [30]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  23. [31]

    Mul- timodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023

    2023 arXiv

  24. [32]

    Visual chain-of-thought prompting for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, and Chuang Gan. Visual chain-of-thought prompting for knowledge-based visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1254–1262, 2024. 11

    2024

  25. [33]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168–5191, 2023

    2023

  26. [34]

    Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs

    Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprint arXiv:2401.02582, 2024

    2024 arXiv

  27. [35]

    Compositional chain-of- thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024

    2024

  28. [36]

    Chatgpt 4o

    OpenAI. Chatgpt 4o. Large language model, 2024. Accessed version 2024-08-06

    2024

  29. [37]

    Gemini 2.0 flash-exp

    Google. Gemini 2.0 flash-exp. Large language model, 2024. Accessed on 2024-08-06

    2024

  30. [38]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv

  31. [39]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI Blog. Retrieved December, 20:2024, 2024

    2024

  32. [40]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  33. [41]

    Llava-llama3: A llava model fine-tuned from llama 3 instruct and clip-vit-large-patch14- 336 with sharegpt4v-pt and internvl-sft

    XTuner. Llava-llama3: A llava model fine-tuned from llama 3 instruct and clip-vit-large-patch14- 336 with sharegpt4v-pt and internvl-sft. https://github.com/XTuner/llava-llama3, 2025

    2025

  34. [42]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024

    2024 arXiv

  35. [43]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  36. [44]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  37. [45]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  38. [46]

    Ollama: A lightweight, extensible framework for building and running language models

    Ollama. Ollama: A lightweight, extensible framework for building and running language models. https://github.com/ollama/ollama, 2025

    2025

  39. [47]

    Bongard- logo: A new benchmark for human-level concept learning and reasoning

    Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. Bongard- logo: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33:16468–16480, 2020

    2020

  40. [48]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  41. [49]

    Rapid natural scene catego- rization in the near absence of attention

    Fei Fei Li, Rufin VanRullen, Christof Koch, and Pietro Perona. Rapid natural scene catego- rization in the near absence of attention. Proceedings of the National Academy of Sciences , 99(14):9596–9601, June 2002. 12

    2002

  42. [50]

    Michael I. Posner. Orienting of attention. Quarterly Journal of Experimental Psychology , 32(1):3–25, February 1980

    1980

  43. [51]

    Thinking, fast and slow

    Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, 2011

    2011

  44. [52]

    Neural machine translation by jointly learning to align and translate, 2016

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016

    2016

  45. [53]

    Working memory: Theories, models, and controversies

    Alan Baddeley. Working memory: Theories, models, and controversies. Annual Review of Psychology, 63(1):1–29, January 2012

    2012

  46. [54]

    Larry R. Squire. Memory and the hippocampus: A synthesis from findings with rats, monkeys, and humans. Psychological Review, 99(2):195–231, 1992

    1992

  47. [55]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural ...

    2017

  48. [56]

    GAMR: A guided attention model for (visual) reasoning

    Mohit Vaishnav and Thomas Serre. GAMR: A guided attention model for (visual) reasoning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  49. [57]

    visual attention

    Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. Semantic answer similarity for evaluating question answering models. In Adam Fisch, Alon Talmor, Danqi Chen, Eunsol Choi, Minjoon Seo, Patrick Lewis, Robin Jia, and Sewon Min, editors, Proceedings of the 3rd Workshop...

    2021

  50. [58]

    We use an indicator function I[·] which is 1 if the condition inside is true, and 0 otherwise

    Text Score (fCA): This measures if the correct caption is selected for each image description. We use an indicator function I[·] which is 1 if the condition inside is true, and 0 otherwise. fCA(i) =I hChoice C(D0,i, {C0,i, C1,i}) =C0,i and Choice C(D1,i, {C0,i, C1,i}) =C1,i i ...

  51. [59]

    Image Score (gCA): This measures if the correct image description is selected for each caption. gCA(i) =I hChoice D(C0,i, {D0,i, D1,i}) =D0,i and Choice D(C1,i, {D0,i, D1,i}) =D1,i i (2) This score is 1 only if the model correctly identifies the description for both caption C0...

  52. [60]

    "" Generates a visual analysis prompt . Args : m ( int ) : Number of positive samples . n ( int ) : Number of negative samples . Returns : str : The formatted prompt string

    Group Score (hCA): This requires all associations within the sample to be correct. hCA(i) =fCA(i) ∧ gCA(i) (3) Equivalently, hCA(i) = 1if and only if fCA(i) = 1and gCA(i) = 1. A.5 Model Prompts A.5.1 Direct Visual Rule Learning The prompt used for the Direct Visual Rule Learni...

  53. [61]

    Determine the rule or criterion that distinguishes the ‘ cat_2 ‘ samples from the ‘ cat_1 ‘ ones

  54. [62]

    Analyse the ‘ query image ‘ ( last image )

  55. [63]

    Invalid input : m and n must be non - negative . Received m ={ m } , n ={ n }

    Provide your conclusion for the ‘ query image ‘ if it can be categorized as either ‘ cat_1 ‘ or ‘ cat_2 ‘ based on the analysis and the rule . Ensure that the output is clear , well - formatted , and free of unnecessary explanations . Omit the ‘‘‘ tags at the beginning and end...

  56. [64]

    ** Scene / Environment **: Description of the overall setting depicted , such as urban , natural , indoor , or outdoor scenes

  57. [65]

    - ** Living Beings **: Animals , humans , or other biological entities

    ** Objects **: Define distinct items or entities present in the scene . - ** Living Beings **: Animals , humans , or other biological entities . - Species or classification ( e . g . , dog , bird , human ) . - Characteristics ( e . g . , color , posture , movement ) . - ** Ina...

  58. [66]

    - Specific descriptions of actions ( e

    ** Activities **: Observable actions or interactions involving any objects or beings . - Specific descriptions of actions ( e . g . , walking , flying ) . - Participants involved in these actions

  59. [67]

    - Detailed characteristics ( e

    ** Contextual Elements **: Environmental conditions and time markers , such as time of day or weather . - Detailed characteristics ( e . g . , cloudy , night , winter )

  60. [68]

    ** Visual Patterns **: Prominent colors , textures , and patterns that are visually significant

  61. [69]

    ** Emotional Undertones **: Any emotional presence or expressions evident in the image

  62. [70]

    ** Textual Information **: Any visible text within the image , including what it says and its visual style

  63. [71]

    Scene ": {

    ** Summary **: A concise narrative summarizing the overall content and context of the image . Ensure that every aspect from the image is represented under these categories . The information should be presented in the following JSON format : { " Scene ": { " Description ": "......

  64. [72]

    Derive the rule that defines the cat_2 samples

  65. [73]

    person with instrument

    Apply this rule to categorize the test image . Here are the image descriptions : ### Positive Samples ( cat_2 ) : { all_image_specs [: m ]} ### Negative Samples ( cat_1 ) : { all_image_specs [ m : m + n ]} ### Test Image : { all_image_specs [ -1]} Provide your output in the fo...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.