Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A five-stage RAVEN benchmark shows MLLMs master perception but flounder at rule extraction.

desk verdict The benchmark is worth having, but the new metric is not ready as specified. read the letter →

arxiv 2505.21850 v2 pith:AOLPNEOF submitted 2025-05-28 cs.CV cs.AI

Yanbei Jiang , Yihao Ding , Chao Lei , Jiayang Ao , Jey Han Lau , Krista A. Ehinger This is my paper
classification cs.CVcs.AI
keywords abstractvisualreasoningmultimodallargelanguagemodelsmulti-stagebenchmarkRAVENMSEvallogicalchainruledeductionevaluationmetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing abstract visual reasoning benchmarks grade only the final answer, so a model that guesses the right panel by chance and a model that reasons correctly until the last step receive the same score. This paper introduces MultiStAR, a RAVEN-based benchmark that splits the reasoning process into five dependent stages—single-panel perception, two-panel comparison, one-row rule deduction, two-row rule deduction, and the full puzzle—and proposes MSEval, a metric that scores each stage using the model's probability of the correct answer at all dependent stages, weighted by each stage's measured influence on the next. On 17 MLLMs the benchmark shows that current models handle basic perception well (up to 88% accuracy on one-panel questions) but fall to near-random levels at rule-detection stages, and that providing ground-truth intermediate answers rescues intermediate stages but not the final puzzle. The paper's central claim is that separating perception from rule detection reveals where abstract visual reasoning breaks down, and that MSEval captures partial progress that accuracy misses.

What carries the argument

The load-bearing object is the Logical Chain, a handcrafted, corpus-level dependency graph with five stages that mimic a bottom-up human strategy: perceive a single panel, compare two panels, deduce the rule from one row, confirm it with two rows, then solve the original 3x3 puzzle. Each edge $(j \to t)$ means node $j$'s answer is reformatted and prepended as prior information $H_t$ to node $t$'s question. The metric carrying the argument is MSEval, which turns each node's logits into a per-node score $p^{(i)}_j / \varepsilon^{(i)}_j$ and then averages these scores across the dependency set with NCMI weights, computed by perturbing a node's answer set and measuring the entropy change of the current node's output. This is what lets the benchmark credit a model for correct intermediate reasoning even when the final answer is wrong, and penalize a model that reaches the right final answer without consistent intermediate steps.

What would settle it

Identify the subset of Logical-Chain instances where the authors' own rule templates imply the 1R or 2R stage is under-specified (e.g., rules involving three distinct values or permutations), then re-evaluate the same models with an additional sentence supplying the missing second-row values. If accuracy and MSEval on those instances jump from near-random to near-human levels, the paper's conclusion that MLLMs specifically fail at rule detection would need to be revised toward 'MLLMs fail at rule detection when the benchmark withholds necessary information.'

Watch

Extended reading notes

Core claim

MultiStAR decomposes RAVEN puzzles into a fixed logical chain of subproblems—1P (one-panel perception), 2P (two-panel comparison), 1R (one-row rule deduction), 2R (two-row rule deduction), and Final—so that each stage's question is answerable in principle from the image plus the answers of its dependent stages. MSEval scores stage $t$ as $\mathrm{MSEval}^{(i)}_t = \sum_{j \in D_t \cup \{t\}} w^{(i)}_j \cdot p^{(i)}_j / \varepsilon^{(i)}_j$, where $p^{(i)}_j$ is the softmax probability the model assigns to the ground-truth answer at node $j$, $\varepsilon^{(i)}_j = 1/|A^{(i)}_j|$ is the random-chance rate, and $w^{(i)}_j$ is the normalized conditional mutual information between node $j$'s choices and the model's output at the current node. The empirical claim is that among 17 evaluated MLLMs, one-panel perception accuracy is strong (GPT-4o 88.1%, Gemini-1.5-pro 83.2%), while accuracy on one-row rule deduction, two-row rule deduction, and the final RAVEN puzzle drops to roughly random levels (best models: 46.9%, 37.8%, 12.1%, respectively). When prior information is supplied, nearly all models improve on intermediate stages in both accuracy and MSEval, but the final-stage answer stays near random for most, showing that current MLLMs can consume stepwise hints yet fail to chain them into a full analogy solution.

Load-bearing premise

The logical chain is fixed at the corpus level, so the framework assumes each stage's question plus its prior answers always provides enough information for the next stage; the authors themselves note in the Limitation section that for some one-row rules, such as 'three different numbers,' the second-row information is also needed, meaning stage scores can reflect missing information rather than pure reasoning ability.

Editorial extensions

If this is right

  • MSEval-based scores give a diagnosis rather than a single number: high MSEval with a wrong final answer indicates partial reasoning, while high final accuracy with low MSEval (as observed for Qwen2-VL-72B) flags probable pattern memorization.
  • Providing ground-truth intermediate answers lifts 1R and 2R accuracy sharply (e.g., GPT-4o from 34.7% to 66.0% at 1R, and 28.9% to 93.8% at 2R), suggesting stepwise supervision is a concrete route to improving abstract reasoning in MLLMs.
  • The metric is defined for any chained multiple-choice task and requires only per-stage logits, so it can be applied to other multi-step reasoning domains such as mathematics and science, as the authors note in the conclusion.
  • Attribute-level results identify number and position as relatively easy for current models, while color and size rules remain hard, giving a focused target for future training data and model design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If MSEval became a standard reporting metric, leaderboards would reorder relative to accuracy-based rankings: models that guess the final answer would be downgraded, while models that reason faithfully but slip at the last step would be rewarded, changing what training optimizes for.
  • The paper's contamination analysis suggests a practical diagnostic: on RAVEN-derived benchmarks, a model showing high final-stage accuracy with near-random intermediate accuracy is likely memorizing puzzle patterns, so the logical chain itself could serve as a contamination detector.
  • Because the chain is corpus-level and fixed, a natural extension is instance-level chain construction driven by the ground-truth XML rules, which would remove under-specification and could also generate chains of varying length to measure reasoning depth.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MultiStAR, a multi-stage abstract visual reasoning benchmark derived from RAVEN, with two subtasks: Direct Answer, which isolates perception, comparison, and rule-deduction stages, and Logical Chain, which connects these stages into a fixed dependency structure ending at the original RAVEN puzzle. The authors also propose MSEval, a logit-based metric intended to credit correct intermediate stages alongside the final answer, and evaluate 17 multimodal LLMs plus human participants. The empirical findings are that models often perform well on single-panel perception questions but degrade sharply on rule deduction, and that providing prior-stage information helps models on intermediate stages more than on the final RAVEN puzzle.

Significance. If the metric is properly formalized and the weighting procedure made reproducible, MultiStAR is a useful diagnostic resource: it shifts evaluation from end-accuracy to the locus of failure in abstract visual reasoning, and it ships with external ground truth from RAVEN XML, human verification, human performance norms, and released code and data. The authors are also appropriately cautious about possible RAVEN contamination in two open models. The paper's main methodological contribution, MSEval, currently has formal and reproducibility gaps that prevent the reported numbers from being fully trusted, but these appear fixable within the manuscript's scope.

major comments (4)
  1. [§3.3, Eqs. (4)–(9)] The definition of MSEval is internally inconsistent. Equation (4) is labeled 'Joint Prob' and includes a 'Norm' operator, but its right-hand side is an unnormalized product of exp(p_j / ε_j), which is not a joint probability and can exceed 1. Equations (8)–(9) show that the quantity actually reported is the weighted sum Σ_j NCMI(i,j,t)·p_j/ε_j, i.e., a weighted average of per-node calibrated softmax values, not a joint probability; taking the logarithm of a weighted geometric mean does not turn it into a probability. The same 'joint probability' language appears in the Introduction and in Section 3.3. Because Table 3 and the qualitative analysis are built on this quantity, the formal definition must be corrected: either define MSEval straightforwardly as this weighted sum and remove the joint-probability claim, or provide a properly normalized joint probability and reconcile it with the equations actually used.
  2. [§3.3, Eq. (6) and Algorithm 1] The NCMI weighting scheme is not reproducible as written. The instruction to 'Alter A_j to generate perturbed outputs A_{j→t}' does not specify how answer options are altered, how many perturbed option sets are produced, whether the model is re-queried for each perturbation, or how the entropies H(·) in Eq. (6) are estimated; the conditioning notation D^{(i),−j}_t is also undefined. Because every MSEval score in Table 3 is multiplied by weights obtained from this procedure, an unreproducible weighting scheme leaves the main metric unsupported. In addition, since CMI/NCMI is computed from the same model being scored, the weights are model-relative: the same chain can receive different weights for different models, so cross-model MSEval comparisons in Table 3 conflate reasoning accuracy with sensitivity to the perturbation procedure. Please specify the perturbation distribution and entropy estimators exactly, and provide code or a synthetic verification.
  3. [§8, Limitation and §3.1.2] The paper acknowledges that 'prior information is insufficient for the current stage' in some cases, giving the example of a one-row rule ('Three Different Numbers') that requires second-row information. This is not merely a design limitation; it means stage-wise accuracy and MSEval scores in Table 3 can conflate model reasoning with missing information, particularly at the 1R and 2R stages. Please quantify how often this occurs, filter or augment such instances, and state explicitly how this affects the claim that prior information helps models on intermediate stages.
  4. [Tables 2 and 3] All model accuracy and MSEval numbers are reported as single runs without confidence intervals, significance tests, or multiple decoding seeds. Given that the Logical Chain task has only 3.92K QA pairs across five stages (roughly 700–800 per stage per model), differences of a few points in Table 2 (e.g., GPT-4o vs. Gemini at 1R: 40.0 vs. 46.9) and the qualitative rankings in Table 3 may be within noise. Please add variance estimates (e.g., bootstrap CIs or multiple random seeds) or at least state the sampling procedure for the reported subsets.
minor comments (4)
  1. [Appendix A.7] The text says 'Tables 15 and 16 present the percentage increase,' but the referenced items are Figures 15 and 16; please fix the cross-reference or renumber the floats.
  2. [Appendix Table 17] The MSEval rows for Intern2-VL-8B under the Vanilla condition are identical to the Idefics2-8B values (2.02, 1.48, 1.51, 1.51, 1.44), which conflicts with Table 3 where Intern-VL2-8B under prior information has MSEval values (1.75, 1.38, 1.30, 1.18, 1.26). Please verify and correct the appendix entries.
  3. [§3.3, Eqs. (5)–(6)] The symbol A_j is overloaded: it denotes the set of answer options in Eq. (5) and a random variable in Eq. (6). Use A_j for the option set and a different symbol, such as O_j, for the random variable representing the output at node j.
  4. [Throughout] Minor typographical issues include 'close-source' for 'closed-source' in several places, 'Gemeni' for 'Gemini' in the Figure 5 caption, and 'V anilla' for 'Vanilla' in Table 6 and Appendix Table 17; these should be corrected in a final pass.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark's ground truth is external (RAVEN XML), the central empirical finding comes from accuracy, and MSEval, though model-relative in its weighting, is a metric rather than a prediction derived from its own inputs.

full rationale

The paper's derivation chain is self-contained with respect to the claims it makes. MultiStAR's ground truth is taken from RAVEN's XML files, which are external to the authors and to the models being evaluated; the dataset construction uses functional programs and templates independent of model outputs. The central empirical result—that MLLMs perform well on basic perception but decline on rule-deduction stages—is read directly from accuracy tables, not from a fitted or self-referential quantity. MSEval is the only potentially self-referential component: its NCMI weights are computed from the logits of the same model being scored, so the metric is model-relative. However, this is a property of the metric's design, not circularity: the score is still anchored to external ground-truth correctness, and the paper does not use MSEval to derive its central finding; it uses it to illustrate partial success. The Limitation section's acknowledgment that the logical chain can under-specify stages is a benchmark limitation, not a circular step. The only self-citations are illustrative and non-load-bearing, and the comparison to MARVEL cites a different research group. No equation in the paper reduces a predicted quantity to its input by construction, so no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark and metric introduce no fitted constants; all probabilities and weights are computed from model outputs and predefined RAVEN rules.

assumptions (3)
  • domain assumption RAVEN XML files provide complete and correct ground-truth object attributes and rules.
    All MultiStAR questions and answers are generated from these XML annotations, so any error in them propagates to the benchmark.
  • ad hoc to paper The handcrafted logical chain accurately represents the essential intermediate steps for solving RAVEN puzzles.
    The chain is fixed at corpus level and the authors note cases where prior information is insufficient (Section 8), making this a load-bearing design assumption.
  • domain assumption Softmax probabilities over answer logits are a meaningful and comparable confidence signal across stages.
    MSEval relies on these probabilities and on conditional mutual information estimates derived from perturbing answer sets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task." pith.science (2026). https://pith.science/paper/AOLPNEOF

@misc{pith2026250521850,
  author       = {Pith},
  title        = {Pith review of: Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AOLPNEOF}},
  note         = {Machine review of arXiv:2505.21850}
}
read the original abstract

Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail. To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark, based on RAVEN, designed to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiStAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.

Figures

Figures reproduced from arXiv: 2505.21850 by the authors.

Figure 1
Figure 1. Left: RAVEN puzzle. The correct answer is 1. Right: Direct Answer subtask, where questions are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Left: Direct Answer subtask, showcasing six configurations along with their corresponding examples. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our MultiStAR dataset generation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: This example is the MSEval score calculation [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Breakdown analysis of five attributes for [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The top two rows are dependent stages (All Correct), the bottom row is current stage (Incorrect). [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The left three panels (a), (b), and (c) present analyses of the Direct Answer task, while the right panel (d) [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The full logical chain: To arrive at the final [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The original RAVEN puzzle, includes seven [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Errors distribution for each model under the [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 12
Figure 12. Figure 12: Sample Question from the Questionnaire for [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 11
Figure 11. Figure 11: A detailed guide provided to participants at [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 14
Figure 14. Figure 14: Sample Question from the One-Panel Basic [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 13
Figure 13. Figure 13: A detailed guide provided to participants at [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 15
Figure 15. Figure 15: Accuracy percentage increase after incorporating prior info. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: MSEval percentage increase after incorporating prior info. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: The average accuracy trend for the Direct [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 20
Figure 20. Figure 20: The Two-Panels Comparison accuracy trend [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]
Figure 21
Figure 21. Figure 21: The One-Row Deduction accuracy trend for the Direct Answer task as model sizes gradually in￾crease. The trend line is derived using Gaussian smooth￾ing [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: The Two-Rows Deduction accuracy trend for the Direct Answer task as model sizes gradually in￾crease. The trend line is derived using Gaussian smooth￾ing [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 23
Figure 23. Figure 23: The Final accuracy trend for the Direct An [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: The Radar Chart for all six models in Logical Chain subtask. [PITH_FULL_IMAGE:figures/full_fig_p021_24.png]
Figure 25
Figure 25. Figure 25: Document Structure for Handling Long Prompt [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 20 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Kian Ahrabian, Zhivar Sourati, Kexuan Sun, Jiarui Zhang, Yifan Jiang, Fred Morstatter, and Jay Pujara. 2024. The curious case of nonverbal abstract reasoning with multi-modal large language models. In Proceedings of Thirty Seventh Conference on Learning Theory

    work page 2024

  4. [4]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  5. [5]

    Xu Cao, Bolin Lai, Wenqian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Jianguo Cao, and James M Rehg. 2024. What is the visual cognition gap between humans and multimodal llms? arXiv preprint arXiv:2406.10424

    arXiv 2024

  6. [6]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    arXiv 2024

  7. [7]

    Fran c ois Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547

    arXiv 2019

  8. [8]

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402

    arXiv 2024

Show all 39 references
  1. [9]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    2023 arXiv

  2. [10]

    Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. 2021. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769--1779

    2021

  3. [11]

    Yihao Ding, Siwen Luo, Hyunsuk Chung, and Soyeon Caren Han. 2023. Vqa: A new dataset for real-world vqa on pdf documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 585--601. Springer

    2023

  4. [12]

    Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2022. Cric: A vqa dataset for compositional reasoning on vision and commonsense. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5561--5578

    2022

  5. [13]

    Gael Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2024. Large language models are not strong abstract reasoners yet. In ICLR 2024 Workshop: How Far Are We From AGI

    2024

  6. [14]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904--6913

    2017

  7. [15]

    Ruozhen He, Paola Cascante-Bonilla, Ziyan Yang, Alexander C Berg, and Vicente Ordonez. 2024. Improved visual grounding through self-consistent explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13095--13105

    2024

  8. [16]

    Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. 2021. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55:661--688

    2021

  9. [17]

    Yanbei Jiang, Krista A Ehinger, and Jey Han Lau. 2024 a . Kale: An artwork image captioning system augmented with heterogeneous graph. arXiv preprint arXiv:2409.10921

    2024 arXiv

  10. [18]

    Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, and Jay Pujara. 2024 b . Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. arXiv preprint arXiv:2404.13591

    2024 arXiv

  11. [19]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Reco...

    2017

  12. [20]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159--174

    1977

  13. [21]

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? arXiv preprint arXiv:2405.02246

    2024 arXiv

  14. [22]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  15. [23]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  16. [24]

    Jacek Ma \'n dziuk and Adam \.Z ychowski. 2019. Deepiq: A human-inspired ai system for solving iq test problems. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE

    2019

  17. [25]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

    2019

  18. [26]

    Arsenii Kirillovich Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. Transactions on machine learning research

    2023

  19. [27]

    Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020. Bongard-logo: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33:16468--16480

    2020

  20. [28]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/

    2024

  21. [29]

    Jean Raven. 2003. Raven progressive matrices. In Handbook of nonverbal assessment, pages 223--237. Springer

    2003

  22. [30]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289--301

    2022

  23. [31]

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. 2023. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19...

    2023

  24. [32]

    Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. 2018. Measuring abstract reasoning in neural networks. In International conference on machine learning, pages 4477--4486

    2018

  25. [33]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  26. [34]

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156--3164

    2015

  27. [35]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  28. [36]

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. 2024. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872

    2024

  29. [37]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conferenc...

    2024

  30. [38]

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5317--5327

    2019

  31. [39]

    Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2024. How far are we from intelligent visual deductive reasoning? arXiv preprint arXiv:2403.04732

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.