Pith. sign in

REVIEW 5 major objections 6 minor 121 references

Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that attaching human accuracy rates, common wrong answers, and solution strategies to 9,794 bilingual reasoning questions reveals that current multimodal models do not reason like humans, and that much of their apparent…

desk verdict Genuinely useful new benchmark, but the load-bearing human-performance metadata is unvalidated and the 'fake reasoning' claim overreaches against its own Table 4. read the letter →

arxiv 2505.11141 v2 pith:VLX66F36 submitted 2025-05-16 cs.CV cs.AI

Yansheng Qiu , Li Xiao , Zhaopan Xu , Pengfei Zhou , Zheng Wang , Kaipeng Zhang This is my paper · ORCID
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsreasoningbenchmarkhumanalignmentvisualfakecivilserviceexaminationbilingualevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Human-Aligned Bench, a set of 9,794 bilingual reasoning questions taken from Chinese civil service examinations, where every question comes with the human success rate, the option humans most often pick when wrong, and a written summary of how human experts approach that question type. The authors use this resource to test whether multimodal large language models reason the way humans do, rather than merely whether they answer correctly. They find that current models trail human accuracy overall, that their accuracy fails to track human-defined difficulty on image-based reasoning, and that injecting either human or self-generated solution strategies usually does not help, and often hurts, performance. The central conclusion is that much of what looks like reasoning in these models may be fake reasoning: it does not follow from the model's own understanding of the problem type.

What carries the argument

The central object is the Human-Aligned Bench dataset: 9,794 bilingual questions in four categories, visual reasoning, definition judgment, analogical reasoning, and logical judgment, each carrying a human correctness rate, a human error-prone option, and a per-category summary of human solution strategies. The dataset is assembled from Chinese civil service examination papers, whose questions are designed to require contextual reasoning with no outside knowledge, and it is translated into English to give bilingual coverage. The annotation layer is what carries the argument: it lets the authors measure accuracy in human-defined difficulty buckets, compute the consistency of model responses and model errors with human responses and errors, and test whether feeding models human or self-generated solution frameworks changes their accuracy. The solution frameworks themselves are the probe for fake reasoning.

What would settle it

Take a stratified random sample of about 600 questions, have a fresh panel of human participants answer them under exam-like conditions, and compare the resulting accuracy rates with the scraped online rates; if the rates diverge substantially, especially on hard questions, the benchmark's difficulty ladder and human-alignment conclusions would need recalibration. Separately, if supplying models with human- or self-generated solution strategies consistently improves accuracy across model families, the paper's fake-reasoning claim would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a reasoning benchmark can be made human-aligned by attaching three human measurements to every question: the rate at which human test-takers answer correctly, the distractor option that humans most often choose when they are wrong, and a written account of how human experts solve that question type. Using 9,794 questions from China's civil service examination, chosen because they test pure contextual reasoning rather than specialized knowledge, the authors compare eleven multimodal models with these human measurements. They report that text-based reasoning roughly tracks the human difficulty gradient, while visual reasoning does not: model accuracy stays flat or moves against the gradient as questions get harder for humans. On the alignment measures, models rarely choose the same wrong options that humans choose on easier questions, though their errors become more human-like on the hardest questions. The paper interprets the solution-injection experiments as evidence that most current models do not reason from their own understanding of a question type, describing the phenomenon as fake reasoning.

Load-bearing premise

The load-bearing premise is that the human accuracy rates scraped from online civil service exam preparation platforms accurately represent how real test-takers perform; if those rates are biased, then every difficulty label, every human-model alignment comparison, and the claim that the best model has approached average human performance would lose their foundation.

Editorial extensions

If this is right

  • Model accuracy can be reported as a function of human-defined difficulty, so a flat or inverted difficulty curve becomes an explicit diagnostic for visual reasoning rather than a hidden artifact.
  • The benchmark gives a decomposition of model skill by language and modality, isolating whether a failure is perceptual, textual, or reasoning-based.
  • The fake-reasoning result implies that prompt sensitivity is a measurable property of reasoning models and should be reported alongside average accuracy.
  • The human error-prone option makes it possible to track whether models are led astray by the same distractors that mislead people, turning error analysis into an alignment metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the online exam-prep accuracy statistics are not representative of the general test-taking population, the human baselines could overstate or understate true difficulty; a controlled replication on a fresh sample would settle this.
  • The fake-reasoning diagnosis could be sharpened by varying how strongly the injected solution is phrased, or by fine-tuning models on solution strategies to see whether the degradation disappears.
  • The same template could be applied to other standardized exams with published item-level statistics, turning any such exam into a human-aligned reasoning probe.
  • A stronger test of alignment would condition model errors on the solution strategy used; if models fail to make the same errors as humans even when given human strategies, the gap is in reasoning itself rather than in answer selection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Human-Aligned Bench, a 9,794-question bilingual benchmark drawn from Chinese civil service examinations across four reasoning categories (visual reasoning, definition judgment, analogical reasoning, logical judgment). Each question is accompanied by metadata consisting of a human correctness rate, the human error-prone option, and category-level human solution frameworks. The authors evaluate eleven proprietary and open MLLMs and report three main findings: current MLLMs are far from human performance on visual reasoning; MLLMs do not show human-like accuracy trends across human difficulty bins in visual reasoning; and adding human or self-generated solution summaries to prompts degrades most models, which the paper interprets as evidence of 'fake reasoning'. An additional claim is that Gemini-2.5-pro-exp-03-25 has approached the average human performance on the benchmark.

Significance. If the human metadata is validated, the benchmark would fill a real gap: most multimodal reasoning benchmarks lack fine-grained human performance data, error-prone distractors, and solution strategies, and the bilingual coverage is broader than that of MM-IQ, VisuLogic, and VISUALPUZZLES. The dataset and code are released, which supports reproducibility, and the evaluation covers a wide range of closed and open models. The error-consistency analyses in Figure 4 are a potentially distinctive contribution. However, the paper's central claims currently rest on an unvalidated source of human performance data and on an internally contradicted 'fake reasoning' analysis, so the contribution is not yet fully established.

major comments (5)
  1. [Section 3 (Data Collection) and Appendix A.1] The per-question human correctness rates and error-prone options are scraped from online civil-service exam-preparation platforms, but the paper reports no response counts per question, no platform identifiers, no inclusion/exclusion criteria, no inter-annotator agreement, and no independent validation against a held-out human sample. These rates are the ground truth for the difficulty bins in Table 3, for the accuracy-trend claims in Section 4.3, and for the 68.82% human baseline in Appendix A.1, so the headline that Gemini-2.5-pro-exp-03-25 has 'approached the average human performance' inherits any bias in this metadata. Moreover, 68.82% is a simple mean of question-level rates, which is a valid estimate of average human performance only if every question has the same number of respondents; the paper should report per-question response counts or use a response-weighted estimate.
  2. [Section 4.4 (Fake Reasoning Analysis) and Table 4] The claim that 'the performance of MLLM is degraded except for Gemini-2.5-pro-exp-03-25' is directly contradicted by Table 4, where GPT-4o-WHS improves by +3.38 overall and QvQ-72B-Preview-WHS improves by +0.43. The experimental design also has no control condition with a matched-length neutral passage, so the observed degradation, where it occurs, cannot be attributed specifically to human reasoning content rather than to prompt length, format, or distraction. The 'fake reasoning' conclusion should be reworded and supported with neutral controls, per-category results, and a statistical comparison.
  3. [Section 4.1 (Experimental Setup)] All results are reported from single runs at temperature 0.6, with no repeated sampling, no confidence intervals, and no significance tests. Consequently, fine-grained comparisons such as QvQ-Max outperforming o4-mini by 0.37% on Chinese questions (Section 4.2, Compare on Language Type) and the small differences among models in Table 3 cannot be distinguished from sampling noise. The authors should report multiple runs and variance estimates for the headline rankings and for the consistency analyses in Figure 4.
  4. [Table 3 and Section 4.3 (Visual Reasoning rows)] The visual-reasoning per-bin accuracies are reported without per-bin question counts, and the 0-20% and 20-40% columns contain small numbers of questions, so a difference of one or two percentage points can correspond to a handful of items. The claim in Section 4.3 that models exhibit uniform or inverse accuracy trends on visual reasoning across human difficulty levels is therefore not statistically supported as reported. The table should show per-bin N and human mean accuracy per bin, and the monotonicity claim should be tested statistically rather than by visual inspection.
  5. [Section 4 (Overall Results)] The benchmark questions are drawn from public civil service exams that are widely available online, and the paper reports no contamination check or discussion of the possibility that the evaluated models have seen these exact questions during training. Because the claims of approaching human performance and of 'fake reasoning' assume that models are solving rather than recalling, the authors should at least report exact-match or n-gram overlap analyses where feasible and discuss the residual contamination risk for closed models.
minor comments (6)
  1. [Table 1] The comparison row labels the benchmark 'Fake Reasoning' rather than 'Human-Aligned Bench'; this appears to be a typo from an earlier name and should be corrected.
  2. [Table 4] The table uses the abbreviation 'WHS' for both 'With human Solution' and 'With self Solution'; the second should be renamed to 'WSS' for clarity.
  3. [Section 4.2 and Figure 3] The text appears to swap the subfigures: Section 4.2 attributes the language-type comparison to Figure 3(a), while the figure caption assigns modalities to (a) and languages to (b).
  4. [Section 4.3 (Consistency on Response)] The expression 'approximately 75% (performance percentage / 60)' is unexplained, and no equations define the response-consistency or error-consistency metrics, so Figure 4 cannot be reproduced from the text.
  5. [Throughout] The paper uses 'human correctness rate', 'human accuracy rate', and 'human score rate' interchangeably; the authors should use one term consistently and define it as the percentage of respondents who chose the correct answer.
  6. [Abstract and Table 2] The abstract says the benchmark includes 'bilingual (Chinese and English) multimodal questions', but the English versions are GPT-4o translations of Chinese exam questions; this should be stated explicitly in the main text so that language comparisons are not misread as comparisons of original source languages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's human labels are externally sourced, and the model-versus-human analyses use those labels as fixed reference data rather than as fitted targets.

full rationale

The paper's central construction is a benchmark whose ground-truth answers, human accuracy rates, and error-prone options come from external civil service examination repositories and online exam-preparation platforms, not from the authors' own models or from any parameter fitted to model outputs. Section 4.3 compares MLLMs against these pre-existing human labels, so conclusions such as 'MLLMs failed to exhibit accuracy improvements corresponding to reduced task difficulty' are empirical observations against an external standard, not self-justifying definitions. The 'fake reasoning' analysis in Section 4.4 does not rest on a self-citation or on a fitted parameter; it is a behavioral interpretation of prompt-conditioned accuracy changes, and its main weakness is experimental control rather than circularity. No uniqueness theorem, ansatz, or load-bearing self-citation is invoked. The unverified provenance of the scraped human rates is an external-validity limitation, explicitly acknowledged in the data-collection description, but it does not make the derivation circular. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; the benchmark is an aggregation of existing exam questions and metadata. The hand-chosen temperature and difficulty bin boundaries are the only free parameters, and they affect the evaluation analysis rather than the benchmark's ground truth. The main unstated assumptions concern the reliability of human accuracy data and the purity of the reasoning tasks.

free parameters (2)
  • Inference temperature = 0.6
    Set for all models in Section 4.1; chosen by hand, affects response variability and reproducibility of all reported accuracies.
  • Human difficulty bins = 0-20, 20-40, 40-60, 60-80, 80-100 (human accuracy percent)
    Used to stratify all results (Tables 3 to 6, Figure 3); arbitrary boundaries chosen by the authors, not derived from the data or theory.
assumptions (3)
  • domain assumption Human accuracy rates from online exam prep platforms reliably measure human performance on each question.
    Every 'human-aligned' annotation (HCR, error-prone options) is sourced from these platforms, as described in Section 3 Data Collection. If the platforms' user populations are unrepresentative, the difficulty labels and human-model alignment conclusions are biased.
  • domain assumption Civil service exam reasoning questions isolate pure contextual reasoning and require no additional domain knowledge.
    Used in Sections 1 and 3 to justify the benchmark as a pure reasoning test. Some questions, e.g., the AAC flavonoid passage, may still draw on scientific background, which could confound the text reasoning comparisons.
  • domain assumption GPT-4o translation followed by human review preserves semantic content and correct answers.
    Data Processing section states all non-Chinese questions were translated via GPT-4o and reviewed. Any mistranslation would alter difficulty and could invalidate English-language comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans." pith.science (2026). https://pith.science/paper/VLX66F36

@misc{pith2026250511141,
  author       = {Pith},
  title        = {Pith review of: Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLX66F36}},
  note         = {Machine review of arXiv:2505.11141}
}
read the original abstract

The goal of achieving Artificial General Intelligence (AGI) is to imitate humans and surpass them. Models such as OpenAI's o1, o3, and DeepSeek's R1 have demonstrated that large language models (LLMs) with human-like reasoning capabilities exhibit exceptional performance and are being gradually integrated into multimodal large language models (MLLMs). However, whether these models possess capabilities comparable to humans in handling reasoning tasks remains unclear at present. In this paper, we propose Human-Aligned Bench, a benchmark for fine-grained alignment of multimodal reasoning with human performance. Specifically, we collected 9,794 multimodal questions that solely rely on contextual reasoning, including bilingual (Chinese and English) multimodal questions and pure text-based questions, encompassing four question types: visual reasoning, definition judgment, analogical reasoning, and logical judgment. More importantly, each question is accompanied by human success rates and options that humans are prone to choosing incorrectly. Extensive experiments on the Human-Aligned Bench reveal notable differences between the performance of current MLLMs in multimodal reasoning and human performance. The findings on our benchmark provide insights into the development of the next-generation models.

Figures

Figures reproduced from arXiv: 2505.11141 by the authors.

Figure 1
Figure 1. Overview of Human-Aligned Bench. Human-Aligned Bench contains 4 categories of questions, each of which has both Chinese and English versions. Each question contains human scoring rates and error-prone options. These questions require models’ abilities in visual logic and pure text reasoning. • Analysis of Fake Reasoning Abilities. We conduct joint analysis using human’s prior reasoning processes and the built-in rea… view at source ↗
Figure 2
Figure 2. Data curation and statistics of our Human-Aligned Bench. The data curation pipeline consists of four stages: data collection, screening, parsing, and processing. 2.0-flash-thinking [37]. Notwithstanding its current nascent stage, these pioneering contributions have illuminated new investigative pathways towards the realization of AGI. Multimodal Reasoning Benchmarks. With the rapid MLLMs, multimodal benchmark has ev… view at source ↗
Figure 3
Figure 3. Performance (%) on the Human-Aligned Bench across different modalities (text, image) and bilingual (Chinese and English ) in five human difficulty levels. (a) delineates the evaluation metrics for MLLMs, showcasing their mean accuracy across different modalities and tiers of difficulty. (b) presents the evaluation metrics for MLLMs, illustrating their mean accuracy across different languages and levels of complexity… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Consistency on Response and Error. (a), (b), and (c) represent the Consistency on Response between MLLMs and humans. (d), (e), and (f) denote the Consistency on Error between MLLMs and humans. In the heatmap, each position (x, y) represents the proportion of consistenc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

121 extracted references · 35 canonical work pages

  1. [1]

    Reasoning, problem solving, and intelligence.Handbook of human intelligence, pages 225–307, 1982

    Robert J Sternberg. Reasoning, problem solving, and intelligence.Handbook of human intelligence, pages 225–307, 1982

    1982

  2. [2]

    Intelligence and reasoning.The Cambridge handbook of intelligence, pages 419–441, 2011

    David F Lohman and Joni M Lakin. Intelligence and reasoning.The Cambridge handbook of intelligence, pages 419–441, 2011

    2011

  3. [3]

    Levels of agi: Operationalizing progress on the path to agi.arXiv preprint arXiv:2311.02462, 2023

    Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Levels of agi: Operationalizing progress on the path to agi.arXiv preprint arXiv:2311.02462, 2023

    arXiv 2023

  4. [4]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.arXiv preprint arXiv:2503.07536, 2025

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.arXiv preprint arXiv:2503.07536, 2025

    arXiv 2025

  5. [5]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  6. [6]

    Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline.arXiv preprint arXiv:2404.02893, 2024

    Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline.arXiv preprint arXiv:2404.02893, 2024

    arXiv 2024

  7. [7]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    arXiv 2025

  8. [8]

    Logicasker: Evaluating and improving the logical reasoning ability of large language models.arXiv preprint arXiv:2401.00757, 2024

    Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, and Michael R Lyu. Logicasker: Evaluating and improving the logical reasoning ability of large language models.arXiv preprint arXiv:2401.00757, 2024

    arXiv 2024

Show all 121 references
  1. [9]

    Symbol-llm: Towards foundational symbol-centric interface for large language models.arXiv preprint arXiv:2311.09278, 2023

    Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, and Jun Liu. Symbol-llm: Towards foundational symbol-centric interface for large language models.arXiv preprint arXiv:2311.09278, 2023

    2023 arXiv

  2. [10]

    Language models can be logical solvers.arXiv preprint arXiv:2311.06158, 2023

    Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. Language models can be logical solvers.arXiv preprint arXiv:2311.06158, 2023

    2023 arXiv

  3. [11]

    Logicot: Logical chain-of-thought instruction-tuning.arXiv preprint arXiv:2305.12147, 2023

    Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, and Yue Zhang. Logicot: Logical chain-of-thought instruction-tuning.arXiv preprint arXiv:2305.12147, 2023

    2023 arXiv

  4. [12]

    Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025

    Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jo- celyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025

    2025 arXiv

  5. [13]

    Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–30, 2024

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–30, 2024

    2024

  6. [14]

    Structured chain-of-thought prompting for code generation.ACM Transactions on Software Engineering and Methodology, 34(2):1–23, 2025

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting for code generation.ACM Transactions on Software Engineering and Methodology, 34(2):1–23, 2025

    2025

  7. [15]

    Codecot: Tackling code syntax errors in cot reasoning for code generation.arXiv preprint arXiv:2308.08784, 2023

    Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. Codecot: Tackling code syntax errors in cot reasoning for code generation.arXiv preprint arXiv:2308.08784, 2023

    2023 arXiv

  8. [16]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 10

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Springer, 2007

    Ben Goertzel and Cassio Pennachin.Artificial general intelligence, volume 2. Springer, 2007

    2007

  11. [19]

    Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning, 2024

    Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasonin...

    2024 arXiv

  12. [20]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  13. [21]

    R1-v: Reinforcing super generalization ability in vision-language models with less than $3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https://github.com/Deep-Agent/R1-V , 2025. Accessed: 2025-02-02

    2025

  14. [22]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  15. [23]

    Visualprm: An effective process reward model for multimodal reasoning.arXiv preprint arXiv:2503.10291, 2025

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning.arXiv preprint arXiv:2503.10291, 2025

    2025 arXiv

  16. [24]

    Othink-mr1: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning.arXiv preprint arXiv:2503.16081, 2025

    Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, and Jun Wang. Othink-mr1: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning.arXiv preprint arXiv:2503.16081, 2025

    2025 arXiv

  17. [25]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model. https://github.com/om-ai-lab/ VLM-R1, 2025. Accessed: 2025-02-15

    2025

  18. [26]

    Open-r1-video

    Xiaodong Wang and Peixi Peng. Open-r1-video. https://github.com/Wang-Xiaodong1899/ Open-R1-Video, 2025

    2025

  19. [27]

    Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    2025 arXiv

  20. [28]

    Mm-iq: Benchmarking human-like abstraction and reasoning in multimodal models.arXiv preprint arXiv:2502.00698, 2025

    Huanqia Cai, Yijun Yang, and Winston Hu. Mm-iq: Benchmarking human-like abstraction and reasoning in multimodal models.arXiv preprint arXiv:2502.00698, 2025

    2025 arXiv

  21. [29]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models.arXiv preprint arXiv:2504.15279, 2025

    2025 arXiv

  22. [30]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  24. [32]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–...

    2022

  25. [33]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  26. [34]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  27. [35]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. In12th International Conference on Learning Representations, ICLR 2024, 2024. 11

    2024

  28. [36]

    Gpt-4o: A multimodal language model, 2024

    OpenAI. Gpt-4o: A multimodal language model, 2024. URL https://openai.com/gpt-4o. Accessed: 2025-03-08

    2024

  29. [37]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  30. [38]

    Claude: A conversational ai assistant, 2024

    Anthropic. Claude: A conversational ai assistant, 2024. URL https://claude.ai. Accessed: 2025-03- 08

    2024

  31. [39]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  33. [41]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  34. [42]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  35. [43]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF Conference on Computer Visi...

    2024

  36. [44]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  37. [45]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  38. [46]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  39. [47]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  40. [48]

    Pangea: A fully open multilingual multimodal LLM for 39 languages

    Xiang Yue, Yueqi Song, Akari Asai, Simran Khanuja, Anjali Kantharuban, Seungone Kim, Jean de Dieu Nyandwi, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A fully open multilingual multimodal LLM for 39 languages. InThe Thirteenth International Confer...

    2025

  41. [49]

    Harnessing webpage uis for text-rich visual understanding.ArXiv, abs/2410.13824,

    Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding.ArXiv, abs/2410.13824,

    arXiv

  42. [50]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.ArXiv preprint, abs/2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.ArXiv preprint, abs/2406.16860, 2024. URL https://arx...

    2024 arXiv

  43. [51]

    The llama 3 herd of models.ArXiv preprint, abs/2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.ArXiv preprint, abs/2407.21783, 2024. URLhttps://arxiv.org/abs/2407.21783

    2024 arXiv

  44. [52]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See Kiong Ng, Lidong Bing, and Roy Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 4663–4680, 2024. 12

    2024

  45. [53]

    Multimath: Bridging visual and mathematical reasoning for large language models.arXiv preprint arXiv:2409.00147, 2024

    Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models.arXiv preprint arXiv:2409.00147, 2024

    2024 arXiv

  46. [54]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023

    2023

  47. [55]

    Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. Med-moe: Mixture of domain-specific experts for lightweight medical vision-language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 3843–3860, 2024

    2024

  48. [56]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  49. [57]

    Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922, 2024

    Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922, 2024

    2024 arXiv

  50. [58]

    Star: Self-taught reasoner bootstrapping reasoning with reasoning

    Eric Zelikman, YH Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. InProc. the 36th International Conference on Neural Information Processing Systems, volume 1126, 2024

    2024

  51. [59]

    Quiet- STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet- STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024

    2024 arXiv

  52. [60]

    Qvq: To see the world with wisdom, December 2024

    Qwen Team. Qvq: To see the world with wisdom, December 2024. URL https://qwenlm.github.io/ blog/qvq-72b-preview/

    2024

  53. [61]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):220102, 2024

    2024

  54. [62]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  55. [63]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  56. [64]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688, 2023

    2023 arXiv

  57. [65]

    Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios.arXiv preprint arXiv:2401.00741, 2024

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios.arXiv preprint arXiv:2401.00741, 2024

    2024 arXiv

  58. [66]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  59. [67]

    Egothink: Evaluating first-person perspective thinking capability of vision-language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first-person perspective thinking capability of vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14291...

    2024

  60. [68]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  61. [69]

    Ok-vqa: A visual question answering benchmark requiring external knowledge.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3190–3199, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3190–3199, 2019. URL https://api.semanticscholar....

    2019

  62. [70]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024. 13

    2024

  63. [71]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. InProceedings of the IEEE/CVF Conference on Co...

    2024

  64. [72]

    Humanity’s last exam.ArXiv, abs/2501.14249, 2025

    Humanity’s Last Exam’s Authors. Humanity’s last exam.ArXiv, abs/2501.14249, 2025. URL https: //api.semanticscholar.org/CorpusID:275906652

    2025 arXiv

  65. [73]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  66. [74]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  67. [75]

    Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models.arXiv preprint arXiv:2403.10378, 2024

    Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models.arXiv preprint arXiv:2403.10378, 2024

    2024 arXiv

  68. [76]

    Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries, 23 (3):289–301, 2022

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries, 23 (3):289–301, 2022

    2022

  69. [77]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    arXiv

  70. [78]

    Mmiu: Multimodal multi-image understanding for evaluating large vision- language models.arXiv preprint arXiv:2408.02718, 2024

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision- language models.arXiv preprint arXiv:2408.02718, 2024

    2024 arXiv

  71. [79]

    Introducing openai o3 and o4-mini, 2025

    OpenAI. Introducing openai o3 and o4-mini, 2025. URL https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: 2025-03-08

    2025

  72. [80]

    Gemini 2.5: Our most intelligent ai model, 2025

    Team Gemini. Gemini 2.5: Our most intelligent ai model, 2025. URL https://blog. google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ #gemini-2-5-thinking. Accessed: 2025-05-07

    2025

  73. [81]

    black + white = black

    Team Qwen. Qvq-max: Think with evidence, 2025. URL https://qwenlm.github.io/blog/ qvq-max-preview/. Accessed: 2025-05-07. 14 Contents 1 Introduction 1 2 Related Work 3 3 Human-Aligned Bench 4 4 Experiments and Results 5 4.1 Experimental Setup . . . . . . . . . . . . . . . . . ...

    2025

  74. [83]

    Subject: - The definition specifies who the doer of the action or the subject of the state is. - This may be individuals with specific identities (e.g., civil servants, minors), specific organizations (e.g., legal entities, government agencies), groups with certain characteris...

  75. [84]

    Object/Target: - The definition specifies the target of the action or the involved entity. - This may be concrete items (e.g., public property), abstract concepts (e.g., information, reputation), specific relationships (e.g., contractual relationships), or specific groups (e.g...

  76. [85]

    Action/State/Property: - The core content of the definition, describing what is specifically done, what state is occupied, or what properties are possessed. - This may include specific actions (e.g., theft, rescue), psychological activities (e.g., intent, negligence), processe...

  77. [86]

    during working hours,

    Conditions/Situations/Methods: - Definitions often specify the specific background, preconditions, or means by which the action occurs. - Examples: "during working hours," "without permission," "by violent means," "for public interest." - Key: Determine whether the situations,...

  78. [87]

    for profit,

    Purpose/Cause/Result: - Definitions sometimes specify the purpose of the action, the triggering cause, or the necessary outcome. - Examples: "for profit," "due to force majeure," "leading to serious consequences," "aimed at improving efficiency." - Key: Determine whether the m...

  79. [88]

    must," "main,

    Qualifiers/Keywords: - Definitions often include words that play a critical limiting role, such as "must," "main," "only," "or," "and," "excluding," "at least," "intentional," "negligent," etc. - Key: Accurately understand the logical meanings and scopes of these words, as the...

  80. [89]

    deconstruct

    Read the Definition Carefully and Deconstruct Core Elements: - Step 1: Read the definition thoroughly to grasp its overall meaning. - Step 2: Read slowly and "deconstruct" the definition to identify core constituent ele- ments such as [Subject], [Object], [Action/State], [Cond...

  81. [90]

    - Step 5: Strictly and systematically compare the option’s information with the defini- tion’s core elements

    Analyze Options One by One and Compare with Definition Elements: - Step 4: Read the first option and extract its key information (also decomposable by elements such as subject, action, conditions, etc.). - Step 5: Strictly and systematically compare the option’s information wi...

  82. [91]

    belongs to

    Filter, Judge, Eliminate, and Select: - Step 6: - For "belongs to" questions: If an option fully meets all elements of the definition, it is likely the correct answer; if any necessary element does not match, eliminate it directly. - For "does not belong to" questions: Look fo...

  83. [92]

    Choose the one that best matches or mismatches the definition

    Compare and Choose the Best (for Ambiguous Options): - Step 8: If multiple options seem to fit or not fit (rare), return to the definition, read the keywords and implicit logic carefully, and compare which option is closer to or further from the definition’s core characteristi...

  84. [93]

    Always take the definition as the sole criterion

    Subjective Assumptions, Deviating from the Definition: The most common mistake is judging based on life experience or prior knowledge instead of strictly adhering to the specific definition given in the question. Always take the definition as the sole criterion

  85. [94]

    must," "main,

    Overlooking Keywords: Failing to notice qualifiers like "must," "main," or "intentional," leading to misinterpretations of the definition’s scope. 3. Missing Elements: Selecting an option that satisfies some but not all necessary elements of the definition. Ensure all hard rul...

  86. [95]

    justifiable defense

    Conceptual Confusion: The option describes a situation similar to the definition’s concept but essentially different (e.g., "justifiable defense" vs. "excessive defense")

  87. [96]

    Or" vs

    Pay Attention to "Or" vs. "And": Clarify whether the definition uses "or" (satisfying one condition is enough) or "and" (all conditions must be met simultaneously)

  88. [97]

    belongs to

    Positive/Negative Question Formats: Check carefully whether the question asks "belongs to" or "does not belong to" the definition to avoid choosing the opposite

  89. [98]

    Word-Picking

    "Word-Picking" Technique: Definition judgment is essentially about information matching and logical judgment. Sometimes, it requires meticulous comparison of subtle wording differences between options and the definition

  90. [99]

    one-sentence summary

    Core Simplification Method: For complex definitions, paraphrase the core meaning in your own words ("one-sentence summary") to grasp the essence before evaluating options

  91. [100]

    在工作时间”、“未经许可

    Element Checklist Method: Mentally or on paper list the definition’s key elements and check each option against them with ticks or crosses for clarity. Please keep in mind the knowledge points and ways to slove problems for knowledge definition that have been given, when answe...

  92. [101]

    必须”、“主要”、“故意

    主观臆断，脱离定义：最常见的错误是凭生活经验或已有知识进行判断，而不 是严格依据【题目给出的特定定义】。务必以定义为唯一标准。 2.忽略关键词：未能注意到 “必须”、“主要”、“故意”等限定词，导致对定义的范 围理解错误。

  93. [102]

    要素缺失或不全：选项满足了定义的部分要素，但未能满足全部【必要】要 素，被误选。要确保所有【硬性规定】都满足。

  94. [103]

    正当防 卫”与“防卫过当

    概念混淆：选项描述的情况与定义涉及的概念相似，但实质不同（如 “正当防 卫”与“防卫过当”）。

  95. [104]

    或”与“且”：看清定义 中是用 “或

    注意 “或”与“且”：看清定义 中是用 “或”连接条件（满足其一即可）还是 用“且”/“并”（必须同时满足）。

  96. [105]

    属于”还是“不属于

    肯定/否定提问方式：看清楚题目问的是 “属于”还是“不属于”该定义，避免选 反。

  97. [106]

    抠字眼”技巧：定义判断本质上是信息匹配和逻辑判断，有时需要细致地 “抠字 眼

    “抠字眼”技巧：定义判断本质上是信息匹配和逻辑判断，有时需要细致地 “抠字 眼”，对比选项和定义在表述上的细微差别。 8.简化核心法：对于复杂的定义，尝试用自己的话转述其核心意思（ “一句话概 括”），抓住本质，再去看选项会更清晰。

  98. [107]

    AppleFruit

    要素核对表法：心里或纸上列出定义的几个关键要素，逐个核对选项是否满 足，打勾或打叉，一目了然。 请参考总结的定义判断的核心知识点以及做题方法回答接下来的问题： 26 B.7 Human Solutions for Analogical Reasoning in English Here is a short account of the key knowledge points and ways to slove problems in analogical reasoning. ### I. Core Knowledge Points: Foun...

  99. [108]

    if...then

    Translational Reasoning - Question Type Judgment: The question stem or options contain typical logical connectives such as "if...then...", "only if...". - Answering Techniques: Translate first, then reason. Translate sentences with logical connectives in the question stem into...

  100. [109]

    - Problem-Solving Methods: - Use the substitution method when option information is sufficient

    Naive Logic - Question Type Characteristics: The question stem provides conditions that require reasoning to derive a conclusion. - Problem-Solving Methods: - Use the substitution method when option information is sufficient. - In more difficult questions, the answer is often ...

  101. [110]

    - Weakening the Evidence: Point out flaws or inadequacies in the evidence

    Weakening Arguments - Weakening the Thesis: Directly challenge the thesis by proposing an opposite view or counterexample. - Weakening the Evidence: Point out flaws or inadequacies in the evidence. - Breaking the Link: Disrupt the logical connection between the thesis and the ...

  102. [111]

    - Strengthening the Evidence: Provide more robust support for the thesis

    Strengthening Arguments - Strengthening the Thesis: Explicitly affirm the thesis or provide consistent informa- tion. - Strengthening the Evidence: Provide more robust support for the thesis. - Establishing a Link: Build a logical "bridge" between the thesis and the evidence. ...

  103. [112]

    Translational Reasoning: Prioritize translational reasoning when logical connectives are present

  104. [113]

    Naive Logic: Use naive logic when the question stem contains numerous complex conditions. 32

  105. [114]

    #### (2) Micro Analysis to Lock in Specific Rules

    Probabilistic Reasoning: Identify whether it is a weakening or strengthening question based on the presence of a thesis and evidence in the question stem. #### (2) Micro Analysis to Lock in Specific Rules

  106. [115]

    Translational Reasoning: Translate the question stem first, then analyze options using reasoning rules

  107. [116]

    Naive Logic: Reason step-by-step based on the given conditions; use the substitution method when necessary

  108. [117]

    - Strengthening Arguments: Prioritize supplementing evidence or establishing logical links

    Probabilistic Reasoning: - Weakening Arguments: Prioritize direct negation of the conclusion or causal inver- sion. - Strengthening Arguments: Prioritize supplementing evidence or establishing logical links. ### III. Common Pitfalls and Tips

  109. [118]

    denying the antecedent

    Translational Reasoning: Remember that "denying the antecedent" and "affirming the consequent" cannot yield definite conclusions

  110. [119]

    - In strengthening, supplementing evidence and eliminating alternative causes are strongly supportive

    Probabilistic Reasoning: - In weakening, direct negation of the conclusion and causal inversion are highly effective. - In strengthening, supplementing evidence and eliminating alternative causes are strongly supportive

  111. [120]

    Control Experiments: Weakening typically involves alternative causes; strengthening typically involves eliminating alternative causes

  112. [121]

    如果...那么...，只有...才

    Premise Assumptions: Options addressing the "jump" between premises and conclusions in the argument are generally the answer. Please keep in mind the knowledge points and ways to slove problems for logical judgment that have been given, when answering questions after this: 33 ...

  113. [2024]

    URLhttps://api.semanticscholar.org/CorpusID:273403951

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.