Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

K12Vista: Exploring the Boundaries of MLLMs in K-12 Education

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read K12Vista claims to be the most comprehensive Chinese K-12 multimodal benchmark to date, and pairs it with a step-by-step process evaluator, K12-PEM, that grades models on how they reason, not just what they answer.

desk verdict Solid benchmark, shaky process evaluation: K12Vista is a useful resource, but the step-by-step error analysis depends on an evaluator with low per-category accuracy. read the letter →

arxiv 2506.01676 v1 pith:IMCUR3FL submitted 2025-06-02 cs.AI

Chong Li , Chenglin Zhu , Tao Zhang , Mingan Lin , Zenan Zhou , Jian Xie This is my paper
classification cs.AI
keywords multimodallargelanguagemodelsK12VistaChineseK12educationbenchmarkchain-of-thoughtreasoningstep-by-stepevaluationprocessmodelerrortaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to close a gap in how multimodal large language models (MLLMs) are tested on school knowledge: existing K-12 benchmarks cover single subjects, are small, lean on multiple-choice questions, and grade only final answers. It introduces K12Vista, a Chinese-language benchmark of 33,660 science exam questions spanning mathematics, physics, chemistry, biology, and geography from primary to high school, with three question types and fine-grained metadata for each item. Its second contribution is a method rather than just data: step-by-step evaluation, which decomposes a model's chain of thought into steps, tags each step as correct or as one of eight error types, and reports a Step-by-Step Score. A trained 7-billion-parameter evaluator, K12-PEM, runs this method using K12-PEM-800K, a corpus of 840,175 step-level judgements, and a human-annotated test set, K12-PEBench, checks how well the evaluator agrees with experts. If the claims hold, the field gains both the largest Chinese K-12 multimodal evaluation resource to date and a reusable way to measure reasoning processes instead of answer accuracy alone.

What carries the argument

Two machinery pieces carry the argument. The first is K12Vista itself: 33,660 questions with structured reference solutions (averaging 5.15 steps each), drawn from non-public school exams to mitigate contamination, with per-question metadata covering grade, subject, question type, knowledge point, and difficulty. The second is the step-evaluation pipeline built on the Step-by-Step Score, defined as $N/M$: a response is split into reasoning steps and each step $s_i$ is tagged with a judgment $j_i$ and explanation $r_i$, forming the triple-tag list $[[s_i, j_i, r_i]]_{i=1}^M$, and the score counts the fraction of steps judged correct. K12-PEM, the model that applies this scoring, is Qwen2.5-VL-7B-Instruct fine-tuned on K12-PEM-800K, a dataset of 840,175 annotations produced by decomposing the chain-of-thought outputs of 40 models with GPT-4o and judging each step by majority vote among GPT-4o, Gemini2-Thinking, Qwen2.5-VL-72B, and InternVL2.5-78B-MPO. The human-annotated K12-PEBench (3,033 samples) measures how accurately K12-PEM and other models perform this process-evaluation task, on which K12-PEM reports 69.38 percent overall step-label accuracy.

What would settle it

Have trained human annotators re-label the steps of a random sample of CoT solutions from K12-PEBench and compute per-step agreement with K12-PEM. If the kappa coefficient falls far below the reported 0.75, or if the error-type distinctions (for instance, knowledge application versus logical reasoning) collapse to chance level, then the step-by-step scores and the error analyses — the paper's main new results — would not survive human verification.

Watch

Extended reading notes

Core claim

On the authors' own terms, the central claim is that a benchmark like K12Vista — large enough, broad enough, and multimodal enough — can systematically expose how MLLMs reason about K-12 science, and that reasoning quality can and should be evaluated step by step rather than by final answer alone. The paper reports that every model it tests scores below 60 percent overall on K12Vista, that the strongest model (Gemini2-thinking) reaches 55.47 percent under direct inference and 57.36 percent under step-by-step evaluation, and that accuracy falls steadily as grade level rises. The step error analysis shows that the dominant failure modes are knowledge application errors and logical reasoning errors, not image perception failures, and that fill-in-blank questions and free-response questions separate models more sharply than multiple choice does. Because most models score lower under step-by-step evaluation than under direct inference, while reasoning-enhanced models score higher, the paper concludes that answer-only evaluation systematically misrepresents MLLM reasoning ability and that process evaluation provides a truer and more actionable measure.

Load-bearing premise

A majority vote of four large language models produced the 840,175 step-level labels that train K12-PEM, with no full human verification reported for that training corpus, so any systematic bias in what those judges count as a correct step would propagate into every step-by-step score in the paper's results.

Editorial extensions

If this is right

  • Researchers get a contamination-resistant Chinese K-12 science testbed: 33,660 questions from non-public school exams, with metadata enabling analysis by grade, subject, question type, and knowledge point.
  • Step-by-step evaluation becomes an affordable alternative to closed-API grading: the fine-tuned 7B K12-PEM reaches 0.75 kappa with expert step judgements and outperforms GPT-4o's overall accuracy on K12-PEBench.
  • The error analysis redirects improvement efforts: since knowledge application and logical reasoning dominate the error distribution, those skills are the main lever for raising MLLM performance on K-12 questions.
  • Answer-only evaluation can misrank models: most models' step-by-step scores fall below their direct-inference scores, and the gap widens at higher grades, so process scoring reveals reasoning weaknesses that final-answer accuracy hides.
  • Reasoning-enhanced models are distinguishable: Gemini2-thinking and O3-mini are the only models whose step-by-step scores exceed their direct scores, suggesting the benchmark can detect genuine process-level gains in future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's pipeline — decompose, tag with a nine-label taxonomy, majority-vote panel, fine-tune a small evaluator — could be transplanted to other languages and subjects; whether the taxonomy survives the transfer is a testable claim the authors do not make.
  • Because the Step-by-Step Score weights every step equally ($N/M$), a model could earn a high score by getting many easy steps right while missing the decisive step; a variant that weights steps by importance is a natural modification the paper does not explore and could change model rankings.
  • The monotone accuracy drop from primary to high school, combined with the wide spread on free-response questions, suggests K12Vista could serve as a difficulty-calibrated tool for building grade-targeted evaluations — an application the authors report data for but do not develop.
  • A control experiment the paper leaves implicit: fine-tuning the same 7B architecture on the human-annotated K12-PEBench alone would show whether the LLM-panel labels carry as much signal as human labels, directly testing the weakest point of the automated pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper presents K12Vista, a Chinese multimodal K-12 benchmark containing 33,660 questions across five subjects (mathematics, physics, chemistry, biology, and geography), three question types, and grades 1-12, with metadata such as knowledge points, difficulty, and reference solutions. The authors also construct K12-PEM-800K, an automatically annotated step-level process evaluation corpus, fine-tune a process evaluation model K12-PEM, and introduce a human-annotated process evaluation benchmark K12-PEBench. Using these resources, the paper evaluates a range of MLLMs and text-only LLMs under direct-inference and step-by-step evaluation, reporting accuracy, step-by-step scores, and step-wise error-type distributions.

Significance. The benchmark resource is potentially valuable: K12Vista is substantially larger than existing Chinese K12 multimodal benchmarks, spans multiple subjects and question types, includes human-refined reference solutions, and is planned for public release. K12-PEBench is a useful human-annotated asset for studying process evaluation. The paper also deserves credit for providing fine-grained metadata and for explicitly discussing dataset construction and quality checks. However, the paper's second contribution depends on K12-PEM, whose agreement with human step-level labels is only 69.38% overall and much lower for several error categories that drive the analysis. Therefore, while the benchmark and direct-inference results are likely usable, the process-evaluation and error-distribution conclusions currently outrun the evidence.

major comments (4)
  1. [Section 5.2; Table 7; Figure 6; Table 9] The Step-by-Step Score and the error distributions in Figure 6 are computed from K12-PEM judgments. On the human-annotated K12-PEBench, K12-PEM achieves 69.38% overall accuracy, with per-class accuracies of 24.87% for Lack of Relevant Knowledge, 25.32% for Hallucination Error, 30.14% for Logical Reasoning Error, and 34.51% for Question Misunderstanding. Since the evaluator disagrees with human annotators on a substantial fraction of step labels, the step-by-step scores in Table 7 and the error-type profiles in Figure 6 are not established as reliable. Please report step-by-step metrics on a human-validated subset with uncertainty estimates, provide a confusion matrix, or substantially temper the process-level conclusions.
  2. [Section 4.1; Appendix C.3] The K12-PEM-800K training labels (840,175 step judgments) come from majority voting among GPT-4o, Gemini2-Thinking, Qwen2.5-VL-72B, and InternVL2.5-78B-MPO, with no human verification of the training set. K12-PEM is then fine-tuned on these labels, so systematic panel biases propagate into all step-by-step results. K12-PEBench breaks the circularity for validating K12-PEM itself, but its small size (3,033 samples) and the low per-category accuracies mean it does not currently protect the main results. I recommend reporting panel-human agreement per panel member on K12-PEBench and auditing a random sample of K12-PEM-800K with human annotators.
  3. [Section 3.2; Section 3.3; Table 2] The sampling description is internally inconsistent. The text states 'a minimum sample size of 1,000 questions for each discipline-grade-question type combination' and later '1,000 questions per type in each subject', but Table 2 reports 33,660 total questions. The first statement would require at least 180,000 questions for 5 subjects × 12 grades × 3 types, and the second at least 15,000 for 5 subjects × 3 types. Please clarify the actual stratified sampling plan and report per-cell counts so that the benchmark's coverage claims can be verified.
  4. [Appendix C.1; Table 9] The eight error categories overlap substantially; for example, Hallucination Error, Logical Reasoning Error, and Knowledge Application Error are distinguished by subtle criteria. The low per-class accuracies on K12-PEBench for these categories suggest that the taxonomy is not being applied consistently, either by human annotators or by K12-PEM. Consider a hierarchical labeling scheme or merging ambiguous categories before using error-type distributions as evidence about model weaknesses.
minor comments (7)
  1. [Throughout] There are numerous typos and spelling errors (e.g., 'Deekseek-R1', 'fianl', 'soluation', 'perfermance', 'surfer', 'understading', 'Visscience'); a careful proofread is needed.
  2. [Table 4; Table 7] Table 4 in Section 4 and Table 7 in Appendix C report the same main results; consolidate them to avoid duplication and confusion about which table is the primary result.
  3. [Figure 6] Figure 6 omits Incomplete Answer Error (IAE) from the legend, although it is one of the eight defined error types; state whether IAE was absent in the analyzed outputs or intentionally excluded.
  4. [Section 4.1] The model name 'Qwen2.5-V1-72B' appears to be a typo for 'Qwen2.5-VL-72B'; check all model names against Appendix C.2.
  5. [Table 1] In Table 1, the K12Vista row does not clearly show which checkmarks correspond to Reference Solutions versus Process Evaluation; use explicit column entries.
  6. [Appendix C.5] Appendix C.5 refers to 'Table 6' for additional MLLM results, but the additional results appear in Table 7; fix the cross-reference.
  7. [Section 3.4; Section 4.2] The quality evaluations report high-quality rates but not inter-annotator agreement; adding kappa or agreement percentages would strengthen the quality claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and process-evaluation model are validated against independent human annotations; the LLM-trained step evaluator is a measurement-validity limitation, not a derivational circularity.

full rationale

K12Vista's construction is independent of the claims it supports: questions are sourced from non-public school exams, filtered by difficulty and text-solvability, and manually validated (99% question quality, 96% answer quality, 94% reference-solution quality on a 1000-sample audit). The direct-inference evaluation uses a separate answer-extraction model and reference answers, so it is not fitted to the reported scores. The process-evaluation contribution is a trained evaluator, not an analytically derived prediction. K12-PEM-800K labels are produced by an LLM majority-vote panel, and K12-PEM is fine-tuned on them; however, the paper evaluates K12-PEM on K12-PEBench, a separately human-annotated benchmark excluded from training (90% step-label quality), and reports overall accuracy 69.38% and an expert kappa of 0.75 for step-by-step evaluation. The Step-by-Step Score is therefore not definitionally equal to the training signal; it is an imperfect empirical estimator. The concern that K12-PEM may inherit panel biases when scoring K12Vista outputs is a measurement-validity limitation, not a circular derivation, because no claim reduces to the training labels by construction. No self-citation chains, uniqueness arguments, or ansatz-by-citation steps are load-bearing. Thus no specific circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper does not derive a mathematical result; its claims rest on dataset-construction choices and on the accuracy of the annotating models and manual quality checks.

free parameters (2)
  • Low-challenge exclusion threshold
    Questions answered correctly by InternVL2-8B, Qwen2-VL-7B, and MiniCPM-V-2.6 were removed; this hand-chosen rule defines benchmark difficulty and shapes the negative results.
  • Text-solvable exclusion threshold
    Questions solvable by Qwen2.5-VL-72B with text-only input were removed to enforce multimodal dependency; this choice directly affects conclusions about visual reasoning failures.
assumptions (4)
  • domain assumption The LLM panel's majority-vote step judgements are correct enough to serve as training labels for K12-PEM.
    Invoked in Section 4.1 for K12-PEM-800K construction; the panel replaces human annotation for 840K samples.
  • domain assumption The 1000-sample manual quality check generalizes to the full K12Vista dataset.
    Section 3.4 and Appendix A.2 report 99% question, 96% answer, and 94% reference-solution quality on a random subset.
  • domain assumption Non-public offline school exams are absent from MLLM pretraining corpora.
    Section 3.1 motivates the data source as contamination-resistant, but no contamination test is provided.
  • domain assumption Mathpix OCR preserves formulas and images without introducing systematic errors.
    Section 3.2 Data Collection relies on Mathpix output as the basis for all question text and metadata.

how reviews work

0 comments
Cite this review

Pith. "Pith review of K12Vista: Exploring the Boundaries of MLLMs in K-12 Education." pith.science (2026). https://pith.science/paper/IMCUR3FL

@misc{pith2026250601676,
  author       = {Pith},
  title        = {Pith review of: K12Vista: Exploring the Boundaries of MLLMs in K-12 Education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMCUR3FL}},
  note         = {Machine review of arXiv:2506.01676}
}
read the original abstract

Multimodal large language models have demonstrated remarkable reasoning capabilities in various visual tasks. However, their abilities in K12 scenarios are still systematically underexplored. Previous studies suffer from various limitations including narrow subject coverage, insufficient data scale, lack of diversity in question types, and naive answer-centric evaluation method, resulting in insufficient exploration of model capabilities. To address these gaps, we propose K12Vista, the most comprehensive multimodal benchmark for Chinese K12 subject knowledge understanding and reasoning to date, featuring 33,000 questions across five core subjects from primary to high school and three question types. Moreover, beyond the final outcome, we are also concerned with the correctness of MLLMs' reasoning processes. For this purpose, we meticulously compiles errors from MLLMs' reasoning processes and leverage an automated data pipeline to construct K12-PEM-800K, the largest process evaluation dataset offering detailed step-by-step judgement annotations for MLLMs' reasoning. Subsequently, we developed K12-PEM, an advanced process evaluation model that integrates an overall assessment of both the reasoning process and answer correctness. Moreover, we also introduce K12-PEBench, the first high-quality, human-annotated benchmark specifically designed for evaluating abilities of reasoning process evaluation.Extensive experiments reveal that current MLLMs exhibit significant flaws when reasoning within K12Vista, providing critical insights for the development of more capable MLLMs.We open our resources at https://github.com/lichongod/K12Vista.

Figures

Figures reproduced from arXiv: 2506.01676 by the authors.

Figure 1
Figure 1. Some examples in K12Vista. Each question in K12Vista features high-quality text and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of K12Vista dataset construcion process. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of K12-PEM-800K data construcion process. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: The result of K12-PEM on K12- PEBench. 4.2 K12-PEBench Construction To construct K12-PEBench, we carefully selected from our previously collected CoT evaluation sam￾ples. After excluding the K12-PEM-800K dataset, we chose approximately 3,000 samples with rich rea￾sonin…
Figure 5
Figure 5. Figure 5: Accuracy of MLLMs across different subjects and question types. We demonstrate the results across five subjects (mathematics, physics, chemistry, biology, and geography) and three question types (multiple-choice, fill-in-blank, free response). 5.2 CoT reasoning step-by…
Figure 6
Figure 6. Figure 6: Distribution of Step-Wise Error Types We present the results across various subjects, question types, and grade levels, derived from Gemini2-thinking’s result on K12Vista. (ICE: Image Cognition Error; QM: Question Misunderstanding; LRK: Lack of Relevant Knowledge; KAE:…
Figure 7
Figure 7. Figure 7: Prompt for question item reviewing. We mainly checked issues with images and question [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompt for image caption. To facilitate LLM inference, we transformed image content into [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt for MLLMs inference. We designed six distinct prompts for MLLM inference, [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Prompt for direct inference evaluation. We require the MLLM to first extract reference [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Prompt for step-by-step evaluation. We require K12-PEM to first decompose students’ [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Prompt for split solution to steps. We have the GPT-4o decompose complete student [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Prompt for step-wise judgement. We defined nine Step-wise labels, enabling the MLLM [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: A sample error case of Math from response of GPT-4o. We provide their corresponding [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: A sample correct case of Chemistry from response of GPT-4o. We provide their corre [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: A sample correct case of Phtsics from response of GPT-4o. We provide their corresponding [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EduFlow: Advancing MLLMs' Problem-Solving Proficiency through Multi-Stage, Multi-Perspective Critique

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A multi-stage framework (data filtering, MCTS-guided trajectory construction, PRM-based reranking) improves Qwen MLLMs' accuracy on K-12 multimodal science benchmarks.

Reference graph

Works this paper leans on

55 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Mm-phyqa: Multimodal physics question-answering with multi-image cot prompting

    Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, and Rajiv Shah. Mm-phyqa: Multimodal physics question-answering with multi-image cot prompting. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 53–64. Springer, 2024

    work page 2024

  3. [3]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    arXiv 2024

  4. [4]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning

    Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning. arXiv preprint arXiv:2401.14011, 2024

    arXiv 2024

  7. [7]

    Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37:19209–19253, 2024. 10

    work page 2024

  8. [8]

    Visscience: An extensive benchmark for evaluating k12 educational multi-modal scientific reasoning

    Zhihuan Jiang, Zhen Yang, Jinhao Chen, Zhengxiao Du, Weihan Wang, Bin Xu, and Jie Tang. Visscience: An extensive benchmark for evaluating k12 educational multi-modal scientific reasoning. arXiv preprint arXiv:2409.13730, 2024

    arXiv 2024

Show all 55 references
  1. [9]

    21st century skills

    Teresa J Kennedy and Cheryl W Sundberg. 21st century skills. Science education in theory and practice: An introductory guide to learning theory, pages 479–496, 2020

    2020

  2. [10]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  3. [11]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  4. [12]

    Cmm-math: A chinese multimodal math dataset to evaluate and enhance the mathematics reasoning of large multimodal models

    Wentao Liu, Qianjun Pan, Yi Zhang, Zhuo Liu, Ji Wu, Jie Zhou, Aimin Zhou, Qin Chen, Bo Jiang, and Liang He. Cmm-math: A chinese multimodal math dataset to evaluate and enhance the mathematics reasoning of large multimodal models. arXiv preprint arXiv:2409.02834, 2024

    2024 arXiv

  5. [13]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  6. [14]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv e-prints, pages arXiv–2305, 2023

    2023

  7. [15]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  8. [16]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  9. [17]

    Improve mathematical reasoning in language models by automated process supervision

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024

    2024 arXiv

  10. [18]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  11. [19]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2024

    2024

  12. [20]

    Gpt-o1-mini: A multimodal language model

    OpenAI. Gpt-o1-mini: A multimodal language model. 2024

    2024

  13. [21]

    Gpt-o3-mini: A cost-effective reasoning model

    OpenAI. Gpt-o3-mini: A cost-effective reasoning model. 2024

    2024

  14. [22]

    Qvq: To see the world with wisdom

    qwen Team. Qvq: To see the world with wisdom. 2024

    2024

  15. [23]

    Measuring vision-language stem skills of neural models

    Jianhao Shen, Ye Yuan, Srbuhi Mirzoyan, Ming Zhang, and Chenguang Wang. Measuring vision-language stem skills of neural models. arXiv preprint arXiv:2402.17205, 2024

    2024 arXiv

  16. [24]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  17. [25]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  18. [26]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  19. [27]

    Visualprm: An effective process reward model for multimodal reasoning

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503.10291, 2025

    2025 arXiv

  20. [28]

    Grok-1.5 vision preview

    x.ai. Grok-1.5 vision preview. https://x.ai/news/grok-1.5v, 2024. 11

    2024

  21. [30]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  22. [31]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  23. [32]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  24. [33]

    Gaokao-mm: A chinese human-level benchmark for multimodal models evaluation

    Yi Zong and Xipeng Qiu. Gaokao-mm: A chinese human-level benchmark for multimodal models evaluation. arXiv preprint arXiv:2402.15745, 2024. 12 K12Vista: Exploring the Boundaries of MLLMs in K-12 Education Supplementary Materials A K12Vista Detail A.1 Data Construction Detail P...

    2024 arXiv

  25. [34]

    System Prompt You are a professional educational item developer with a deep expertise in evaluating question correctness, logical coherence, and alignment with educational objectives

  26. [35]

    0" or "1

    User Content Task: You are a professional multimodal exam item reviewer. review the following exam item including a textual question with corresponding image,reference solution and answer, and determine if the item is qualified. Directly output "0" or "1" first (0 = unqualifie...

  27. [36]

    Image Criteria: The item is unqualified if any of the following image related standards are violated:

  28. [37]

    Garbled content, unidentifiable subjects, or nonsensical elements are prohibited

    Image content must be clear and relevant. Garbled content, unidentifiable subjects, or nonsensical elements are prohibited

  29. [38]

    Images cannot consist solely of textual information (e.g., mathematical symbols, letters, or plain text)

  30. [39]

    Images must be clearly visible; blurry or unrecognizable content is not allowed

  31. [40]

    Textual Question Criteria :

  32. [41]

    unclear question focus or undefined conditions

    The question stem must contain a complete problem description, free of grammatical errors, semantic ambiguity, or missing critical information e.g. unclear question focus or undefined conditions

  33. [42]

    The question and its answer must be logically consistent based on the reference solution; any logical inconsistency will result in the item being deemed unqualified

  34. [43]

    0" (unqualified) or

    Disciplinary terminology, concepts, and formulas in the stem must align with definitions from authoritative textbooks, prohibiting factual errors e.g. formula derivation errors in mathematics. Input Format: * Input question item:{question} * Input reference solution:{reference...

  35. [44]

    Based on the input text question and combined with the input image, generate a description of the image. The description should detail the image's content, such as the spatial and geometric relationships of objects, quantities, numerical values, or other information, ensuring ...

  36. [45]

    Input text question : {question}

  37. [46]

    perceive

    Output format: Your output should only be the description of the image, with no other content. Figure 8: Prompt for image caption. To facilitate LLM inference, we transformed image content into textual representations. We provide their corresponding English translations. To mo...

  38. [47]

    Question Type: Multiple Choice

    Direct Infer Prompt: MCQ: Question: {question}. Question Type: Multiple Choice. Please read the question carefully and answer it. Your final answer should be the letter of the correct option. Do not output anything other than the final answer. FBQ: Question: {question}. Questi...

  39. [48]

    Question Type: Multiple Choice

    Step-by-Step Infer Prompt: MCQ: Question: {question}. Question Type: Multiple Choice. Please read the question carefully, think step-by-step, and answer it. Provide a detailed thought process and the final answer. Your final answer should be the letter of the correct option. P...

  40. [50]

    Question Type: Fill-in-Blank

    Final Answer: FBQ: Question: {question}. Question Type: Fill-in-Blank. Please read the question content. There is one or more '___' sections that require filling in. Provide the most appropriate answer based on the question's requirements and the provided information. If there...

  41. [52]

    Question Type: Free Response

    Final Answer: FRQ: Question: {question}. Question Type: Free Response. Please read the question content and the problem. Provide the most appropriate answer based on the question's requirements and the provided information. Read the question carefully, think step-by-step, and ...

  42. [53]

    Thought and Reasoning Process:

  43. [54]

    We designed six distinct prompts for MLLM inference, tailored to different reasoning modes and question types

    Final Answer: Figure 9: Prompt for MLLMs inference. We designed six distinct prompts for MLLM inference, tailored to different reasoning modes and question types. We provide their corresponding English translations. 19 PROMPTPROMPT User Content You are an answer evaluation ass...

  44. [55]

    The text question contains one or more parts that require the student's answer, which should be one or more words or phrases

    Definition of Question and Standard Answer: The question includes a text question and relevant images. The text question contains one or more parts that require the student's answer, which should be one or more words or phrases. A single question-and-answer item may have multi...

  45. [56]

    Extract Reference Answer List: Based on the relative positions of sub-questions within the question or other numbering schemes (e.g., letters, numbers), and the semantic meaning of the question, sequentially extract the corresponding reference answers for the blank positions f...

  46. [57]

    If a student's answer for a specific blank position is not found, record it as 'empty'

    Extract Student Answer List: Based on the relative positions of blank areas within the question or other numbering schemes (e.g., letters, numbers), and the semantic meaning of the question, sequentially extract the corresponding answers for the blank positions from each stude...

  47. [58]

    Reference Answer 1

    Evaluate Student's Answer: Based on the extracted reference answer list and student answer list, sequentially judge the correctness of each corresponding student answer. For student answers and reference answers, if both are phrases or sentences, their narrative content must b...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.