Pith. sign in

REVIEW 5 major objections 4 minor 34 references

IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new 500-question visual IQ benchmark finds top vision-language models fail 3D spatial and anagram reasoning.

desk verdict A useful 500-question visual IQ dataset, but the reasoning-score claim is undermined by a flawed judge validation and internal annotation errors. read the letter →

arxiv 2505.12000 v1 pith:QK7HD5MC submitted 2025-05-17 cs.CV

Tan-Hanh Pham , Phu-Vinh Nguyen , Dang The Hung , Bui Trong Duong , Vu Nguyen Thanh , Chris Ngo , Tri Quang Truong , Truong-Son Hy This is my paper · ORCID
classification cs.CV
keywords IQBenchvision-languagemodelsvisualreasoningbenchmarkfluidintelligenceLLM-as-a-judgeevaluation3Dspatialanagram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IQBench is a 500-question, visually centered IQ benchmark built to test whether vision-language models (VLMs) can reason from images rather than fall back on learned text. The paper reports that the best model, o4-mini, reaches 0.615 accuracy and a 0.696 reasoning score, while gemini-2.5-flash and claude-3.7-sonnet follow at 0.578 and 0.548 accuracy. Across all seven models, 3D spatial reasoning and anagram tasks remain near the bottom, and reasoning scores often exceed accuracy, meaning a model can justify a wrong answer coherently. The authors conclude that answer-only evaluation overstates VLM intelligence and that reasoning quality must be scored alongside correctness.

What carries the argument

The load-bearing mechanism is the dual-metric evaluation framework. Accuracy is exact match between the model's final answer and the ground truth; reasoning is scored by gpt-4o-mini, an LLM judge that receives the question text, the ground-truth reasoning pattern, the ground-truth answer, the VLM's explanation, and the VLM's final answer, and returns 1 or 0 depending on whether the explanation is logically sound and consistent with the pattern. The other central object is the dataset itself: 500 manually collected and annotated, vision-centric IQ questions, 50 per topic and 390 open-ended, designed to minimize textual priors and prevent data leakage. The judge converts explanations into a scalable numerical score, and that score is what carries all the paper's claims about reasoning quality and its divergence from accuracy.

What would settle it

Have independent human experts score reasoning on all 500 IQBench questions and compare with gpt-4o-mini task by task; if agreement on 3D spatial and anagram tasks is near chance, as the 100-sample comparison hints (experts 0.23 vs judge 0.82 on 3D SPRT), then the reasoning rankings and the claimed reasoning-versus-accuracy disconnect would not hold.

Watch

Extended reading notes

Core claim

The central claim is that IQBench exposes three things: leading VLMs are strong on number-series and deductive-figure tasks but weak on 3D spatial and anagram reasoning; the reasoning score, produced by an LLM-as-a-judge (gpt-4o-mini) comparing each model's explanation with the annotated reasoning pattern, frequently disagrees with answer accuracy, so models can reach correct answers through flawed reasoning or wrong answers through coherent reasoning; and human experts, while close to the judge on average (0.68 vs 0.696), diverge sharply on specific tasks such as 3D spatial reasoning (0.23 vs 0.82). The paper presents this divergence not as a failure of the benchmark but as the reason reasoning must be measured separately from accuracy.

Load-bearing premise

The reasoning scores all rest on one judge, gpt-4o-mini, agreeing with human experts about what counts as correct reasoning; if that judge is biased on a task, the headline comparison between reasoning quality and answer correctness weakens.

Editorial extensions

If this is right

  • Answer-accuracy benchmarks will keep overstating VLM intelligence, because on IQBench coherent reasoning and correct answers do not move together.
  • Evaluation of future models should report a reasoning score alongside accuracy, crediting models that justify answers correctly and withholding credit from lucky guesses.
  • 3D spatial understanding and anagram manipulation are the two task families that separate current models, making them useful focused probes for VLM architecture improvement.
  • Because the benchmark is vision-centric and manually curated, a high IQBench score would indicate image-based reasoning rather than memorized textual knowledge.
  • The reported human-judge agreement supports scaling reasoning evaluation with an LLM judge, provided per-task agreement is checked rather than only the overall average.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the anagram failures may come from the model's tokenization rather than from a lack of reasoning, since scrambled-letter rearrangement is exactly where subword tokenizers behave irregularly.
  • The paper's own human-evaluation table warns that the reasoning-score rankings for 3D spatial tasks are the least trustworthy: the judge gives o4-mini 0.82 while the three experts average 0.23, so any claim about spatial reasoning quality should be read with that caveat.
  • A direct testable extension is to re-score all 500 questions with human raters or task-specific rubrics; if per-task agreement with gpt-4o-mini stays high, the reasoning-score story strengthens, and if not, reasoning scores should be reported task by task instead of as one average.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. IQBench introduces a manually curated benchmark of 500 vision-centric IQ-test questions across 10 reasoning categories, with 50 items per category. Seven vision-language models are evaluated zero-shot under two metrics: exact-match answer accuracy and a reasoning score produced by an LLM-as-judge pipeline using gpt-4o-mini. The accuracy results place o4-mini, gemini-2.5-flash, and claude-3.7-sonnet at the top, and uniformly show low accuracy on 3D spatial and anagram tasks. The reasoning scores are used to argue that explanation quality often diverges from final-answer correctness, with the highest divergence on 3D SPRT and anagram tasks. A human evaluation of 100 o4-mini predictions is reported as validating the LLM judge.

Significance. If the accuracy tables are taken at face value, IQBench provides a useful vision-centric stress test with substantial manual curation, and the observation that all tested VLMs fail at 3D spatial rotation and anagram tasks is a falsifiable, reproducible result. The attempt to measure reasoning quality separately from answer correctness is also valuable and would be a real contribution if the measurement were valid. However, the paper's headline claim about reasoning divergence rests on a judge-validation procedure that the manuscript's own data contradict. With only 50 items per task and no uncertainty reporting, several of the ranking claims are also stronger than the evidence supports. The public release of code and data is a concrete strength, as is the use of exact-match accuracy for the accuracy scores.

major comments (5)
  1. [Section 3.3, Table 4] The human evaluation cannot validate the LLM-as-judge because the two score sets are computed over different item sets. The LLM-as-judge row reproduces the full-dataset reasoning scores from Table 3 (e.g., 0.92 on MDRT corresponds to 46/50), while the three human experts scored only 100 randomly sampled predictions, approximately 10 per task. Comparing the resulting averages (0.696 vs. 0.68) is not a per-item agreement test and does not establish that gpt-4o-mini is a reliable proxy for human judgment.
  2. [Table 4, 3D SPRT row] Even accepting Table 4 at face value, the judge is 0.59 points above the human mean on 3D SPRT (0.82 vs. 0.23; human range 0.20-0.30). This is exactly the task named in the paper's headline failure claim and the location of the largest accuracy-reasoning divergence in Table 3 (gpt-4o: 0.20 accuracy vs. 0.56 reasoning). A judge that accepts spatial explanations that human experts reject would manufacture the reported divergence, so the abstract's and Section 3.2's claim that reasoning quality is inconsistent with final-answer correctness is not supported.
  3. [Table 4, Ana5 row] The human reasoning scores are unstable on the same task: on Ana5, Expert 1 scores 0.90 while Experts 2 and 3 score 0.00 and 0.10. With roughly 10 predictions per task in the human sample, task-level human scores carry enormous binomial error. The manuscript needs item-level agreement statistics (e.g., per-item judge-human agreement, Cohen's kappa, or confusion matrices) and confidence intervals before it can claim that the judge and humans are closely aligned.
  4. [Table 2 and Table 3] No uncertainty quantification is provided for any accuracy or reasoning score. With 50 items per task, the standard error of a 0.50 score is roughly 0.07, and even the overall average difference between o4-mini (0.615) and gemini-2.5-flash (0.578) over 500 items is only about 1.7 standard errors. The paper should report binomial confidence intervals or bootstrap intervals and avoid claiming fine-grained rankings that are within sampling noise.
  5. [Figure 1, first example] A showcased sample contains an internal contradiction: the written pattern says "Therefore option (D) is correct option," but the displayed answer line says "Answer: A." Since IQBench's accuracy scores depend entirely on ground-truth labels, this inconsistency should be resolved and the quality-control process for answer annotations should be described.
minor comments (4)
  1. [Section 4, Conclusion] The conclusion names "Gemini 1.5 Flash" as a tested model, but the experiments use gemini-2.5-flash and gemini-2.0-flash; the conclusion also lists "Claude 3.5 Sonnet" while the tables use claude-3.5-sonnet. Please correct the model names and align them with the experimental section.
  2. [Tables 2 and 3, gpt-o3 row] The gpt-o3 row is only populated for MDRT, DRTF, and Ana5, with no average score; the paper should either report the missing entries or clearly state that gpt-o3 was evaluated on a subset and exclude it from cross-model rankings.
  3. [Appendix C, judge prompt] The judge prompt asks whether the VLM reasoning "leads to the correct final answer," but Figure 4a presents a case where the reasoning is considered correct although the final answer is wrong. The scoring criterion for this situation should be stated explicitly, otherwise different judges may apply different thresholds.
  4. [Section 2.2 and Figure 4] The figure caption for Figure 4a says "Correct reasoning with incorrect prediction," while the text says the VLM "misinterprets the visual options." Clarify whether the reasoning score is meant to judge the logical chain only or also the mapping from the chain to the selected option, since this affects the interpretation of every reasoning score in Table 3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IQBench's accuracy results rest on exact-match scoring of a new manually curated dataset, and the reasoning scores are an explicitly stated LLM-judge methodology rather than a prediction fitted to the benchmark's own outputs.

full rationale

I walked the derivation chain in IQBench. The benchmark contribution is a manually curated 500-question visual IQ dataset with exact-match accuracy scoring, so the accuracy rankings (o4-mini 0.615, etc.) are externally grounded in human-authored answers and are not derived from the benchmark's own assumptions. The reasoning-score framework is introduced openly in Section 2.3 as an LLM-as-judge procedure using gpt-4o-mini, and the paper's Limitation section explicitly acknowledges that this pipeline is a limitation. Nothing in the reasoning-score definition is fitted to the accuracy results or to the human evaluation; it is a separate measurement. The paper's claim that reasoning and accuracy diverge is a property of those two independently defined metrics, not a reduction of a predicted quantity to an input. The main evidential weakness is Section 3.3: Table 4's LLM-as-judge row is identical to the o4-mini row of Table 3, while the three human experts evaluated only 100 randomly sampled predictions, so the reported 'close alignment' (0.696 vs 0.68) does not statistically validate the judge, and the judge over-scores 3D SPRT (0.82 vs 0.23) on precisely a task central to the paper's failure claim. This is a validity problem for the reasoning-score conclusions, but it is not circularity: no parameter is fitted to the human labels, no uniqueness theorem or self-citation is used to forbid alternatives, and no quantity is defined in terms of the quantity it is used to predict. I therefore find no step that reduces by construction to its own input, and no load-bearing self-citation chain.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numerical parameters are fitted; the ledger records the unverified premises the benchmark's conclusions depend on, chiefly the validity of the LLM judge, the novelty of the questions, and the correctness of the annotations.

assumptions (5)
  • domain assumption LLM-as-judge (gpt-4o-mini) reasoning scores are a valid proxy for human reasoning quality.
    Invoked in Sections 2.3 and 3.3; contradicted by Table 4 on 3D SPRT and anagram tasks.
  • domain assumption The 500 curated questions are novel enough to avoid training data leakage.
    Stated in Sections 2.1 and 2.2; only lexical overlap checks are described, with no image-hash or memorization test.
  • domain assumption The ground-truth answers and reasoning patterns are correct.
    Figure 1 contains samples where the listed answer contradicts the printed pattern, such as a missing-number question whose pattern ends with option D but whose answer is listed as A.
  • domain assumption Standard IQ test items measure fluid intelligence in VLMs.
    Assumed in the introduction; no convergent validity evidence is provided.
  • domain assumption Exact match on open-ended answers is an appropriate correctness metric.
    Section 2.3; 390 of 500 questions are open-ended, and response formatting differences such as '49 N' versus '49' can count as wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests." pith.science (2026). https://pith.science/paper/QK7HD5MC

@misc{pith2026250512000,
  author       = {Pith},
  title        = {Pith review of: IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ Tests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QK7HD5MC}},
  note         = {Machine review of arXiv:2505.12000}
}
read the original abstract

Although large Vision-Language Models (VLMs) have demonstrated remarkable performance in a wide range of multimodal tasks, their true reasoning capabilities on human IQ tests remain underexplored. To advance research on the fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the reasoning capabilities of VLMs, which we argue are more important than the accuracy of the final prediction. **Our benchmark is visually centric, minimizing the dependence on unnecessary textual content**, thus encouraging models to derive answers primarily from image-based information rather than learned textual knowledge. To this end, we manually collected and annotated 500 visual IQ questions to **prevent unintentional data leakage during training**. Unlike prior work that focuses primarily on the accuracy of the final answer, we evaluate the reasoning ability of the models by assessing their explanations and the patterns used to solve each problem, along with the accuracy of the final prediction and human evaluation. Our experiments show that there are substantial performance disparities between tasks, with models such as `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest average accuracies of 0.615, 0.578, and 0.548, respectively. However, all models struggle with 3D spatial and anagram reasoning tasks, highlighting significant limitations in current VLMs' general reasoning abilities. In terms of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieved top averages of 0.696, 0.586, and 0.516, respectively. These results highlight inconsistencies between the reasoning processes of the models and their final answers, emphasizing the importance of evaluating the accuracy of the reasoning in addition to the final predictions.

Figures

Figures reproduced from arXiv: 2505.12000 by the authors.

Figure 1
Figure 1. Representative IQ test covering logic, pattern recognition, and spatial reasoning for VLM [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Accuracy evaluation of the advanced multimodal models on IQBench. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Reasoning evaluation of the advanced multimodal models on IQBench. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of VLM reasoning errors compared to human annotations. (a) Correct reasoning [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 14 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  2. [2]

    Silvar-med: A speech-driven visual language model for explainable abnormality detection in medical imaging

    Tan-Hanh Pham, Trong-Duong Bui, Minh Luu Quang, Tan-Huong Pham, Chris Ngo, and Truong-Son Hy. Silvar-med: A speech-driven visual language model for explainable abnormality detection in medical imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2025

    work page 2025

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  4. [4]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  5. [5]

    Piergiovanni, Piotr Padlewski, Daniel Salz, et al

    Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony J. Piergiovanni, Piotr Padlewski, Daniel Salz, et al. PaLI: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  6. [6]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716–23736, 2022

    2022

  7. [7]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  8. [8]

    Artificial general intelligence: Emergence and definition

    Ben Goertzel. Artificial general intelligence: Emergence and definition. In Artificial General Intelligence, pages 1–20. Springer, 2007

    work page 2007

Show all 34 references
  1. [9]

    Intelligence: Its structure, growth and action, volume 35

    Raymond Bernard Cattell. Intelligence: Its structure, growth and action, volume 35. Elsevier, 1987. 9

    1987

  2. [10]

    On the measure of intelligence

    François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019

    1911 arXiv

  3. [11]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  4. [12]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    2022 arXiv

  5. [13]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  6. [14]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  7. [15]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing System...

    2022

  8. [16]

    From recognition to cognition: Visual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019

    2019

  9. [17]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  10. [18]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The 3rd Workshop on Mathematical Reasoning and A...

    2023

  11. [19]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, May 2022

    2022

  12. [20]

    Image2struct: Benchmarking structure extraction for vision-language models

    Josselin S Roberts, Tony Lee, Chi H Wong, Michihiro Yasunaga, Yifan Mai, and Percy Liang. Image2struct: Benchmarking structure extraction for vision-language models. Advances in Neural Information Processing Systems, 37:115058–115097, 2024

    2024

  13. [21]

    Know what you don’t know: Unanswerable ques- tions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 784–789, 2018

    2018

  14. [22]

    Don’t just assume; look and answer: Overcoming priors for visual question answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4971–4980, 2018

    2018

  15. [23]

    Shortcut learning in deep neural networks

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, et al. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  16. [24]

    Evaluation of openai o1: Opportunities and challenges of agi

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai o1: Opportunities and challenges of agi. CoRR, 2024

    2024

  17. [25]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 10

    2024 arXiv

  18. [26]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  19. [27]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  20. [28]

    Start building with gemini 2.5 flash, 2025

    Google. Start building with gemini 2.5 flash, 2025. URLhttps://developers.googleblog. com/en/start-building-with-gemini-25-flash/

    2025

  21. [29]

    Grok 3 beta — the age of reasoning agents, 2025

    xAI. Grok 3 beta — the age of reasoning agents, 2025. URL https://x.ai/news/grok-3

    2025

  22. [30]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  23. [31]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  24. [32]

    Generalized planning for the abstraction and reasoning corpus

    Chao Lei, Nir Lipovetzky, and Krista A Ehinger. Generalized planning for the abstraction and reasoning corpus. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20168–20175, 2024

    2024

  25. [33]

    Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity

    Jing Bi, Junjia Guo, Susan Liang, Guangyu Sun, Luchuan Song, Yunlong Tang, Jinxi He, Jiarui Wu, Ali V osoughi, Chen Chen, et al. Verify: A benchmark of visual explanation and reasoning for investigating multimodal reasoning fidelity. arXiv preprint arXiv:2503.11557, 2025

    2025 arXiv

  26. [34]

    Mm-iq: Benchmarking human-like abstraction and reasoning in multimodal models

    Huanqia Cai, Yijun Yang, and Winston Hu. Mm-iq: Benchmarking human-like abstraction and reasoning in multimodal models. arXiv preprint arXiv:2502.00698, 2025. 11 A Prompts for General VLMs Given the image, answer the following question: {question} Your answer must include your...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.