Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CSVQA, a new benchmark of 1,378 Chinese high-school STEM questions, finds that even the best vision-language model tested answers fewer than half correctly.

desk verdict New Chinese STEM VQA benchmark with useful data and a plausible headline result, but the multimodal-specific claim needs a text-only baseline and the LLM judge needs human validation. read the letter →

arxiv 2505.24120 v2 pith:GWO5ISPB submitted 2025-05-30 cs.CV cs.AI

Ai Jian , Weijie Qiu , Xiaokun Wang , Peiyu Wang , Yunzhuo Hao , Jiangbo Pei , Yichen Wei , Yi Peng
show 1 more author
Xuchen Song
This is my paper
classification cs.CVcs.AI
keywords vision-languagemodelsmultimodalbenchmarkscientificreasoningSTEMChinesevisualquestionansweringexplanation-drivenevaluationchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current vision-language models fail at genuine scientific reasoning, not just at image description. It introduces CSVQA, a Chinese-language benchmark of 1,378 question-answer pairs drawn from high-school textbooks and exams across physics, chemistry, biology, and math, each tied to one of 14 visual formats. On this test the strongest evaluated model reaches only 49.6% overall accuracy, while the best open-weight model reaches 38.5%. A second, explanation-alignment evaluation shows that even some correct answers are not backed by valid reasoning steps. If the benchmark is sound, it gives the community a sharper instrument for measuring progress toward scientific reasoning in multimodal models.

What carries the argument

The load-bearing instrument is the benchmark itself: 1,378 human-curated Chinese STEM question-answer pairs spanning 14 visual modalities, with 81.1% of items carrying step-by-step explanations. Those curated explanations make possible the paper's process-tracing evaluation, in which GPT-4o judges whether a model's correct answer is backed by reasoning that aligns with the human solution. The CSVQA-Hard subset is built by intersecting difficulty ratings with samples that text-only models consistently get wrong, isolating questions that truly require visual evidence.

What would settle it

Perform a contamination audit that matches CSVQA questions and images against the pretraining corpora of the evaluated models; if a substantial fraction of questions appear verbatim or near-verbatim, scores would need to be recomputed on a paraphrased held-out version to see whether the 49.6% ceiling reflects reasoning or recall.

Watch

Extended reading notes

Core claim

The paper claims that CSVQA is a valid diagnostic test for scientific visual reasoning and that current vision-language models, both proprietary and open-weight, perform far below competence on it. The dataset contains 1,378 human-validated questions, with 81.1% accompanied by step-by-step explanations, and spans 14 image types across biology, chemistry, math, and physics. In the headline evaluation, the top-ranked proprietary model scores 49.6%, the best open-weight model scores 38.5%, and lightweight open models fall to roughly 10% or below. A process-tracing evaluation, which checks whether correct answers align with curated human solution paths, finds that even the strongest model aligns only 95.4% of its correct answers, while the best open-weight model aligns 67.6%, indicating that correct final answers often arise from pattern matching rather than valid reasoning. Error analysis of one strong model attributes 38.7% of failures to faulty reasoning, 23.0% to perception errors, and 22.6% to missing domain knowledge.

Load-bearing premise

Because the questions come from publicly available textbooks and exams, the headline numbers assume models have not memorized the exact test items during pretraining; the paper reports no contamination check.

Editorial extensions

If this is right

  • No current vision-language model can reliably solve high-school-level multimodal STEM problems, so the practical ceiling for automated scientific tutoring or exam assistance remains below competence.
  • Open-weight models are approaching but still clearly behind proprietary models, with the best gap in overall accuracy at 11.1 percentage points.
  • Physics and math questions, which depend on abstract and symbolic visual representations, are the weakest areas across almost all models.
  • Open-ended questions are substantially harder than multiple-choice questions for most models, suggesting that constrained choice formats mask part of the reasoning deficit.
  • Chain-of-thought prompting helps lightweight models but often hurts stronger models, so reasoning gains depend heavily on model architecture and prompt design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the benchmark's accuracy figures measure reasoning only if the models have not memorized these textbook and exam questions during pretraining; a contamination audit against web-scale corpora would make the scores interpretable.
  • Because CSVQA includes a bilingual translated version, the dataset could be used to test whether reasoning failures are language-specific or shared across Chinese and English prompts.
  • The error taxonomy suggests that improving visual encoders alone will not close the gap; the next bottleneck is multi-step inference over symbolic images, which points toward training on explanation-aligned reasoning traces.
  • The explanation-alignment protocol could be adapted as a training objective: models could be fine-tuned to produce explanations that match curated reasoning paths, not merely the final answer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CSVQA, a Chinese multimodal benchmark for evaluating STEM reasoning in vision-language models (VLMs). The dataset contains 1,378 question-answer pairs sourced from public Chinese high-school textbooks and exam papers, covering physics, chemistry, biology, and mathematics, with images in 1,341 questions and 81.1% of items accompanied by expert-annotated explanations. The authors evaluate 15 (main text count) or 16 (Table 3 count) open-source and closed-source VLMs under multiple-choice and open-ended settings, with rule-based and GPT-4o-based answer extraction. They report that the best-performing model, o1, achieves only 49.6% overall accuracy, that models perform worse on physics and open-ended questions, and that chain-of-thought prompting has mixed effects. They also construct a CSVQA-Hard subset and an explanation-alignment evaluation using GPT-4o to judge whether correct answers are supported by valid reasoning. The paper argues that current VLMs still struggle with rigorous scientific reasoning that requires integrating visual evidence with domain knowledge.

Significance. If the benchmark is uncontaminated and the measurements are trustworthy, CSVQA would be a useful addition to the multimodal scientific reasoning evaluation landscape, complementing ScienceQA, MMMU, and MathVista by providing a Chinese, high-school-level, STEM-focused test set with dense human explanations. The dataset release on HuggingFace, the detailed quality-control pipeline, and the inclusion of both multiple-choice and open-ended formats are concrete contributions. The explanation-alignment evaluation is a promising idea for distinguishing correct answers from lucky guesses. However, the significance of the claimed empirical findings—that even top models fail to reason multimodally—depends critically on three unaddressed issues: the absence of a text-only baseline, the lack of contamination analysis for publicly sourced exam questions, and the absence of human validation of the GPT-4o judge used for open-ended scoring and reasoning-alignment assessment. These issues currently leave the main conclusions under-supported.

major comments (4)
  1. [Section 3.1, Table 1 and Section 4] The central claim that CSVQA measures multimodal visual-evidence reasoning is not supported because the paper never reports a text-only baseline. Although Table 1 shows that 1,341 of 1,378 questions contain an image, the authors do not provide accuracy when models are given only the Chinese text and options without the image. This matters because many questions drawn from textbooks may be answerable from the text alone, with the image serving as a redundant illustration. The construction of CSVQA-Hard in Section 3.1 even relies on feeding only textual components to several models, so the authors already have the machinery to compute text-only performance. Without this ablation, the headline 49.6% accuracy cannot be attributed to visual reasoning; it may largely reflect text-based scientific reasoning. I request a text-only evaluation of the full benchmark and an analysis of the fraction of items where the image is strictly necessary.
  2. [Section 3.2] The dataset is collected from 'publicly available Chinese high school textbooks and examination papers,' which are exactly the kind of materials likely to appear in web-scale pretraining corpora. The paper contains no contamination or overlap analysis with common training data (e.g., n-gram overlap with pretraining corpora or canary checks). Without this, the reported accuracy numbers (e.g., 49.6% for o1) may conflate genuine reasoning with memorization of exact or near-identical questions. This is load-bearing because the paper's conclusion that VLMs 'still struggle with rigorous scientific reasoning' assumes that the evaluation measures reasoning rather than recall. I request an overlap analysis with popular training corpora and a discussion of how the authors handle known benchmark leakage risks.
  3. [Section 5.2 and Appendix B.1] The explanation-driven evaluation and the open-ended question scoring both rely on GPT-4o as the judge, but the paper provides no validation of this judge against human raters. Table 4 reports large differences in reasoning alignment across models (e.g., o1 at 95.4% vs. Qwen2.5VL-72B at 80.8%), and these numbers are presented as evidence about whether models arrive at answers through genuine understanding. However, if GPT-4o's judgments are biased—for example, by preferring long or confident explanations—the differences may be artifacts. Similarly, open-ended answers are scored only by GPT-4o without any human agreement study (Section 4.1). I request a human-annotation study on a sample, with agreement statistics such as Cohen's kappa, to establish the validity of these GPT-4o-based evaluations.
  4. [Section 3.1 and Figure 6] The construction of CSVQA-Hard is circular with respect to the paper's later argument that models perform worse on hard questions, which is used to 'validate our difficulty classification methodology.' CSVQA-Hard is defined as the intersection of questions that (a) were classified as hard by GPT-4o and (b) were consistently answered incorrectly by four models when given only text. The paper then reports that models perform worse on CSVQA-Hard than on the full benchmark (Figure 7) and interprets this as evidence of difficulty and visual dependency. This reasoning is circular: the subset was selected partly because models already failed on it, so the performance drop is unsurprising and does not independently validate the difficulty labels. An independent human difficulty rating, or a held-out construction method that does not use the evaluated models' failures as a selection criterion, is needed.
minor comments (5)
  1. [Abstract and Section 4.2, Table 3] The paper states that 15 VLMs were evaluated, but Table 3 lists 16 model rows plus a Random Choice baseline, and Appendix B.3 evaluates many additional models. Please clarify the exact number of models used for the main headline result and reconcile the counts across the main text and appendix.
  2. [Table 3 and Section 4.2] There are naming inconsistencies: the table lists 'Qwen2.5VL-72B' while the text refers to 'Qwen2.5-78B-Instruct' and 'Qwen2.5VL-78B'; 'Internvl2-5-78B' and 'InternVL3-78b' appear with inconsistent capitalization. Please standardize model names throughout.
  3. [Section 5.2, Table 4] The sentence 'most models tend to achieve higher accuracy on open-ended formats' is not supported by Table 4: Qwen2.5VL-72B (open 78.6 vs. overall 80.8) and Claude3.7-sonnet (open 87.5 vs. overall 90.7) have lower open-ended alignment than their overall alignment. Please revise the claim or explain the discrepancy.
  4. [Section 3.2] There are typographical errors such as 'stablishes' and 'filtering out unsuitable question types' (should be 'filtering out' or 'removing'). Please proofread for language issues.
  5. [Appendix B.3] The appendix presents three evaluation settings (rule-based CoT, direct answer, GPT-scored CoT) but the main text does not clearly explain which setting is used for the headline Table 3 results. Please state explicitly which evaluation protocol underlies the main results and why the alternative protocols are secondary.

Circularity Check

1 steps flagged · score 4.0 of 10

CSVQA-Hard is selected by model failures and then cited as evidence of model failure on hard visual questions; the headline full-benchmark accuracy is a direct measurement and is not circular.

  1. self definitional [Section 3.1 (CSVQA-Hard construction) and Section 5.1 (difficulty analysis)]
    "To specifically identify visually challenging questions, we implemented an additional filtering approach. We feed only the textual components to Gemini2.0-flash, Claude3.7-sonnet, Qwen2.5VL-78B-Instruct and DeepSeekR1, retaining those consistently answered incorrectly. The intersection of these visually dependent samples with previously classified hard questions formed our CSVQA-Hard subset. ... models suffer a noticeable performance drop on the hard subset ... Furthermore, these results validate our difficulty classification methodology."

    CSVQA-Hard is constructed by retaining questions that four text-only models (including Gemini2.0-flash and Claude3.7-sonnet, two of the evaluated VLMs) consistently answered incorrectly, intersected with GPT-4o hard labels. The paper then reports that models 'suffer a noticeable performance drop on the hard subset' and claims this 'validates our difficulty classification methodology.' Because membership in CSVQA-Hard was partly defined by model failure, the subsequent low accuracy on that subset is to a substantial degree guaranteed by the selection rule; the observation does not independently confirm that the subset isolates visual dependency or reasoning difficulty. The headline 49.6% on the full benchmark is a direct measurement and is not affected by this circularity.

full rationale

The paper's central accuracy claim (o1 at 49.6% on the full CSVQA benchmark) is a straightforward external measurement of model performance on a fixed set of human-collected questions; there is no fitted parameter or self-referential derivation in that result. The main circularity concern is localized to CSVQA-Hard: the subset is defined by selecting questions that text-only versions of several evaluated models answered incorrectly, and the same subset is then used to demonstrate that models perform poorly on 'visually dependent' hard questions and to validate the difficulty classification. That is a selection artifact rather than an independent finding, so it raises the score to a moderate level. The absence of a text-only baseline on the full benchmark is a threat to the multimodal-specificity interpretation of the headline number, but it is a validity gap, not a circular step. The paper cites EMMA, co-authored by one of the CSVQA authors, but that citation is contextual and not load-bearing. No contamination analysis appears, but contamination risk is also an external validity concern rather than a circularity of the derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on dataset representativeness, absence of contamination, and judge reliability rather than on free parameters or invented entities. The paper presents no derivation with fitted constants, and the paper introduces no new physical or mathematical entities.

assumptions (4)
  • domain assumption Public Chinese high school textbooks and exam questions are a valid proxy for scientific reasoning.
    The dataset is built entirely from these materials (Section 3.2), so CSVQA scores are interpreted as scientific reasoning measures.
  • domain assumption The evaluated models have not memorized the benchmark questions during pretraining.
    No contamination analysis is provided, despite collection from publicly available exam papers (Section 3.2). This assumption is load-bearing for interpreting accuracy as reasoning.
  • domain assumption GPT-4o is a reliable judge of open-ended answers and explanation alignment.
    GPT-4o scores all open-ended questions (Section 4.1) and judges reasoning alignment (Section 5.2), with no human validation or inter-annotator agreement reported.
  • domain assumption LLM-based difficulty and visual-dependency ratings identify genuinely harder questions.
    GPT-4o and four filtering models define difficulty levels and CSVQA-Hard (Section 3.1, Figure 6), and these labels are then validated by showing accuracy drops on the subset they define.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs." pith.science (2026). https://pith.science/paper/GWO5ISPB

@misc{pith2026250524120,
  author       = {Pith},
  title        = {Pith review of: CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWO5ISPB}},
  note         = {Machine review of arXiv:2505.24120}
}
read the original abstract

Vision-Language Models (VLMs) have demonstrated remarkable progress in multimodal understanding, yet their capabilities for scientific reasoning remain inadequately assessed. Current multimodal benchmarks predominantly evaluate generic image comprehension or text-driven reasoning, lacking authentic scientific contexts that require domain-specific knowledge integration with visual evidence analysis. To fill this gap, we present CSVQA, a diagnostic multimodal benchmark specifically designed for evaluating scientific reasoning through domain-grounded visual question answering. Our benchmark features 1,378 carefully constructed question-answer pairs spanning diverse STEM disciplines, each demanding domain knowledge, integration of visual evidence, and higher-order reasoning. Compared to prior multimodal benchmarks, CSVQA places greater emphasis on real-world scientific content and complex reasoning. We additionally propose a rigorous evaluation protocol to systematically assess whether model predictions are substantiated by valid intermediate reasoning steps based on curated explanations. Our comprehensive evaluation of 15 VLMs on this benchmark reveals notable performance disparities, as even the top-ranked proprietary model attains only 49.6% accuracy. This empirical evidence underscores the pressing need for advancing scientific reasoning capabilities in VLMs. Our CSVQA is released at https://huggingface.co/datasets/Skywork/CSVQA

Figures

Figures reproduced from arXiv: 2505.24120 by the authors.

Figure 1
Figure 1. Challenges in the CSVQA Benchmark. (a) Multidisciplinary STEM Focus: Questions cover STEM subjects, ensuring comprehensive evaluation across disciplines; (b) Diverse Visual Modalities: The benchmark includes 14 types of images; (c) Integration of Domain-Specific Knowledge and Real-World Scenarios. crafted. CSVQA is designed to evaluate the capability of VLMs to comprehend and reason about scientific content in image… view at source ↗
Figure 2
Figure 2. An example of a math problem from the CSVQA benchmark. Multi-step reasoning is [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. demonstrates that CSVQA features a more uniform, extended, and balanced length distribution compared to existing benchmarks. This characteristic enables more robust evaluation of reason￾ing across varied input conditions while establishing a more challenging assessment framework. Moreover, the dataset maintains an equitable distribution of both question types and difficulty levels, effectively mitigating sampling bi… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Subject accuracy comparison across models.Model names are abbreviated in the figures of Section 5 for visual clarity [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Pipeline for constructing the CSVQA￾Hard via the intersection of difficulty and visual dependency filtering [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Error distribution [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Examples of different image types used in our benchmark. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Examples of data format for open-ended (top) and multiple-choice (bottom) questions. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Example of visual perception failure, where the model misinterprets image content. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Failure due to insufficient domain knowledge required to answer the question. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Incorrect logical reasoning leading to a wrong conclusion. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Output truncation resulting in incomplete or ambiguous answers. [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Successful reasoning example of Claude3.7-sonnet. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Successful reasoning example of Gemini2.0-flash. [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Successful reasoning example of InternVL3-78B. [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Successful reasoning example of o1. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Successful reasoning example of QVQ-72B. [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: Successful reasoning example of Qwen2.5-72B. [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]
Figure 21
Figure 21. Figure 21: Incorrect reasoning example of Claude3.7-sonnet. [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Incorrect reasoning example of Gemini2.0-flash. [PITH_FULL_IMAGE:figures/full_fig_p032_22.png]
Figure 23
Figure 23. Figure 23: Incorrect reasoning example of InternVL3-78B. [PITH_FULL_IMAGE:figures/full_fig_p033_23.png]
Figure 24
Figure 24. Figure 24: Incorrect reasoning example of o1. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_24.png]
Figure 25
Figure 25. Figure 25: Incorrect reasoning of of QVQ-72B. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_25.png]
Figure 26
Figure 26. Figure 26: Incorrect reasoning example of Qwen2.5-72B. [PITH_FULL_IMAGE:figures/full_fig_p036_26.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Skywork-R1V3 Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.

Reference graph

Works this paper leans on

45 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1

    2023

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    work page 2023

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI . Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. 1, 4

    work page 2025

  4. [4]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,

    arXiv

  5. [5]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023. 1

    arXiv 2023

  6. [6]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms, 2025. 1

    work page 2025

  7. [7]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team. Gemini: A family of highly capable multimodal models, 2024. 1

    work page 2024

  8. [8]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 1, 6

    work page 2024

Show all 45 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi...

    2025

  2. [10]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021. 1, 3

    2021

  3. [11]

    V Jawahar

    Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021. 1, 3

    2021

  4. [12]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. 1, 3

    2024

  5. [13]

    Lxmert: Learning cross-modality encoder representations from transformers, 2019

    Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers, 2019. 3

    2019

  6. [14]

    Uniter: Universal image-text representation learning, 2020

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, 2020. 3 10

    2020

  7. [15]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 3

    2021

  8. [16]

    Le, Yunhsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. 3

    2021

  9. [17]

    Evev2: Improved baselines for encoder-free vision-language models, 2025

    Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models, 2025. 3

    2025

  10. [18]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training, 2025

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training, 2025. 3, 5, 6

    2025

  11. [19]

    Introducing Gemini 2.0: Our New AI Model for the Agentic Era

    Google DeepMind. Introducing Gemini 2.0: Our New AI Model for the Agentic Era. https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/, 2024. 3, 4, 6

    2024

  12. [20]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 4, 6

    2024

  13. [21]

    A diagram is worth a dozen images, 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, 2016. 3, 5

    2016

  14. [22]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019. 3

    2019

  15. [23]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. 3

    2019

  16. [24]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. 3, 5

    2022

  17. [25]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  18. [26]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.arXiv preprint arXiv:2501.05444, 2025. 3, 5

    2025 arXiv

  19. [27]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. 3, 5

    2024

  20. [28]

    Measuring multimodal mathematical reasoning with math-vision dataset, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024. 3

    2024

  21. [29]

    Claude-3.7, 2025

    Anthropic. Claude-3.7, 2025. 4, 6

    2025

  22. [30]

    Qwen2.5 technical report, 2025

    Qwen Team. Qwen2.5 technical report, 2025. 4, 6

    2025

  23. [31]

    Mineru: An open-source solution for precise document content extraction, 2024

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. Mineru: An open-source solution for precise document content extraction, 2024. 4

    2024

  24. [32]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. 4 11

  25. [33]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. 5

    2023

  26. [34]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Ta¸ sırlar. Introducing our multimodal models, 2023. 5, 6

    2023

  27. [35]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras

    Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. 5, 6

  28. [36]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024

  29. [37]

    Gemma Team. Gemma 3. 2025. 5, 6

    2025

  30. [38]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li,...

    2025

  31. [39]

    Building and better understanding vision-language models: insights and future directions, 2024

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions, 2024. 6

    2024

  32. [40]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 6

    2024

  33. [41]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume La...

    2024

  34. [42]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024. 6

    2024

  35. [43]

    So the final answer is \boxed

    OpenAI. Introducing chatgpt pro. https://openai.com/index/ introducing-chatgpt-pro/. 2, 3, 6 12 A Overview of Appendix A.1 Image Type Examples To provide a clear overview of the visual diversity represented in our benchmark, Fig. 9 presents representative examples of each imag...

  36. [44]

    Is the model’s solution process logically rigorous and coherent, indi- cating a true understanding of the problem? (Yes/No)

  37. [45]

    No," please identify the main unreasonable aspects or obvious flaws in the solution; if

    If your answer is "No," please identify the main unreasonable aspects or obvious flaws in the solution; if "Yes," please briefly explain its strengths or reasonable aspects. Table 6: Prompt templates for different question types and reasoning strategies. 16 B.2 Evaluated Model...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.