Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source MLLMs average below 50% on multi-image reasoning

desk verdict MMRB fills a real gap in multi-image reasoning evaluation, but the process-score table is not trustworthy until the Qwen3 scorer is validated and its metric semantics are made explicit. read the letter →

arxiv 2506.04280 v1 pith:ZY7HIXH7 submitted 2025-06-04 cs.CV cs.AI

Ziming Cheng , Binrui Xu , Lisheng Gong , Zuhe Song , Tianshuo Zhou , Shiqi Zhong , Siyu Ren , Mingxiang Chen
show 9 more authors
Xiangchao Meng Yuxin Zhang Yanlin Li Lei Ren Wei Chen Zhiyuan Huang Mingjie Zhan Xiaojie Wang Fangxiang Feng
This is my paper
classification cs.CVcs.AI
keywords multi-imagereasoningmultimodalbenchmarkchain-of-thoughtevaluationprocessscorerewardmodelsvisualsentence-levelmatchingMLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MMRB, a benchmark built to test whether multimodal large language models can reason across several images at once, with 92 sub-tasks spanning spatial, temporal, and semantic reasoning and 4,750 samples annotated with 68,882 chain-of-thought reasoning steps. Its central claim is that existing evaluations miss this capability: single-image reasoning benchmarks never require cross-image integration, and multi-image benchmarks only score final answers. On MMRB the paper finds that open-source models average below 50% on both outcome and process scores, commercial models average about 65% outcome and 83% process accuracy, and eight multimodal reward models are nearly unable to rank multi-image answers reliably. The authors argue this makes multi-image reasoning a distinct, still-immature capability worth measuring separately.

What carries the argument

The load-bearing evaluation device is a sentence-level matching framework built on the open-source Qwen3-32B model. It compares a model's chain-of-thought response, sentence by sentence, against the gold reasoning steps and classifies each step as correct only if some sentence matches it lexically or paraphrastically and carries the same reasoning type; the process score for a sample is the best precision across the three annotated solution paths, $\max_{R_i \in R} \text{Precision}(R_i)$. The reward subset is built by pairing incorrect AI-generated answers with human-corrected versions and running two test sets with reversed order of accepted and rejected answers, which exposes order sensitivity. A preliminary sample-capacity experiment sets 50 samples per sub-task based on a 95% confidence interval requiring a standard error of the mean below 0.0102.

What would settle it

Take a random sample of 200 MMRB responses, have independent human annotators label each reasoning step as correct or incorrect, and compare their labels to Qwen3-32B's sentence-matching verdicts; if agreement is low or the disagreement tracks model family, size, or format compliance, the paper's process-score comparisons and the open-source gap do not stand as reported.

Watch

Extended reading notes

Core claim

MMRB is claimed to be the first benchmark that combines multi-image input with annotated, multi-solution chain-of-thought reasoning. Each task carries up to three human-refined reasoning paths, each step typed into one of six cognitive operations (task understanding, information grounding, commonsense seeking, logical reasoning, arithmetic calculating, draw conclusion), and 1,198 samples had at least one reasoning step corrected by human annotators while 355 incorrect final answers from source datasets were fixed. The benchmark's headline result is a measured capability gap: across 34 evaluated MLLMs, open-source models stay below 50% average on outcome and process scores, while commercial models reach roughly 65% outcome and 83% process accuracy. A derivative 2,313-sample reward-model subset shows that simply reversing the order of accepted and rejected answers can swing accuracy drastically, with one model falling from 85.69% to 10.20% on the two test sets.

Load-bearing premise

The load-bearing premise is that the automatic Qwen3-32B matcher correctly decides whether a model's chain-of-thought sentence matches a gold reasoning step, with no reported human validation or inter-annotator agreement; if the matcher is biased or noisy, every process score in the paper's main table becomes unreliable.

Editorial extensions

If this is right

  • If MMRB's scores hold, multi-image reasoning becomes a separate axis of MLLM evaluation, distinct from single-image visual reasoning and from multi-image understanding that ignores intermediate steps.
  • The gap between open-source and commercial models suggests that open-source training recipes lack the data or test-time procedures needed for reliable cross-image grounding, not just final-answer accuracy.
  • CoT prompting is predicted to add roughly 6.4% outcome accuracy for non-reasoning commercial models and about 1.2% for reasoning-specialized ones, so evaluation should always report both direct and CoT conditions.
  • Reward models trained on single-image preference data cannot be assumed to rank multi-image answers; MMRB's order-swapped test sets provide a concrete way to measure that instability.
  • The three-path multi-solution annotation means a correct answer can be reached by genuinely different reasoning, so process scores should use the best match rather than requiring a single gold trajectory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report human agreement for the Qwen3-32B matcher; if that matcher is biased toward verbose or format-compliant responses, the process-score ranking of open-source models could be understated, not only noisy.
  • A natural extension is to treat reward-model order sensitivity as a robustness metric for any multi-image preference model, e.g., reporting the absolute difference between the two order-randomized test sets as a stability score.
  • Since MMRB's samples average 6.17 images, the same annotation pipeline could be applied to video frames, turning short video clips into multi-image reasoning tasks with step-level scoring.
  • The finding that answer order flips rankings suggests that benchmark protocols for multi-image rewards should randomize or counterbalance answer order in every evaluation run, not just in a single derivative subset.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MMRB, a benchmark of 92 multi-image reasoning sub-tasks with 4,750 samples and 68,882 human-corrected chain-of-thought reasoning steps, plus a derivative reward-model subset. The authors evaluate 40 MLLMs and 8 multimodal reward models, reporting outcome, process, and efficacy scores. The main findings are that open-source MLLMs average below 50% outcome and process accuracy while commercial models average about 65% outcome and 83% process accuracy, and that multimodal reward models are unstable when the order of accepted/rejected answers is reversed. The benchmark is positioned as the first to combine multi-image input with annotated intermediate reasoning steps.

Significance. The benchmark addresses a real gap, and the outcome-score evaluation is largely rule-based and supports the commercial/open-source gap. The human-correction of 355 ground truths and 1,198 reasoning paths is a substantial curation effort. If the process-score methodology is validated, the benchmark could become a useful resource for multi-image reasoning evaluation. However, the process-score results are central to the paper's novelty and currently depend on an unvalidated LLM matcher with an ambiguous metric definition, so the significance of the process-score claims is conditional on that validation.

major comments (4)
  1. [Appendix B / Appendix H.2 / Section 5.2] The process-score metric is not well defined and the evaluator is not validated. Appendix B defines Precision(R_i) as the fraction of gold reasoning steps for which some sentence in the model's CoT matches (a recall-style measure), while the prompt in Appendix H.2 asks the Qwen3-32B evaluator to 'For each reasoning step in the CoT answer, determine if it is correct or incorrect according to the matched sentences.' These two semantics can disagree substantially: the Appendix B formula never penalizes extra or fabricated steps, whereas the prompt-based evaluation can. The manuscript provides no human validation of the Qwen3-32B matcher, no inter-annotator agreement, and no scoring code, so the reader cannot determine which definition produced the process scores in Table 3. Consequently, the claim that open-source models average below 50% process accuracy and the fine-grained step-by-step reasoning findings are not yet supported. Please release the exact scoring implementation, validate the matcher against human judgments, and rename the metric (it is recall of gold steps, not precision).
  2. [Section 6.2 / Appendix C / Table 3] Table 3's caption indicates that some baselines are evaluated on one-fifth of the data, but the text does not say which models this affects, and no confidence intervals or standard errors are reported. The pilot in Appendix A estimates a minimum of 18 samples per sub-task for stable outcome scores; one-fifth of the 4,750 samples corresponds to roughly 10 samples per sub-task on average, below that threshold. If the affected models are the expensive commercial reasoning models, the headline comparison between commercial and open-source models may be confounded by sampling noise. Please identify the subsampled models and report error bars or confidence intervals for all scores.
  3. [Section 4.2.2 / Appendix H.1] The gold reasoning steps are generated by GPT-4o (Section 4.2.2), and GPT-4o's own CoT outputs are then scored against them. Because the gold steps may share stylistic and lexical patterns with GPT-4o's answers, GPT-4o (and possibly other API models with similar output style) may receive inflated process scores. The manuscript reports no control for this bias. Please add a control, for example by scoring models against gold steps written by a different generator or by having humans assess whether the Qwen3 matcher systematically favors particular model families.
  4. [Section 7.2 / Table 4] The reward-model evaluation in Section 7.2 and Table 4 reports large order effects (e.g., GPT-4o-mini drops from 85.69% to 10.20% when the order of accepted and rejected answers is reversed) but does not report the number of evaluation samples per test set, chance-level accuracy, or any measure of statistical uncertainty. The conclusion that multimodal reward models are 'nearly incapable' should be supported by confidence intervals and a statistical test of whether the average accuracy differs from chance. Please add these details.
minor comments (6)
  1. [Appendix D] Appendix D contains numerous typos ('ans', 'orrors', 'accetable', 'stubbore') and should be proofread.
  2. [Title page / footer] The line 'Received 31 May 2025; revised 12 March 2009; accepted 5 June 2009' is a template artifact with incorrect dates and should be corrected.
  3. [Appendix F / Table 5] Appendix F says the benchmark is sourced from 22 datasets, but Table 5 lists only about 10; please either list all sources or clarify that the table is a summary.
  4. [Table 5] Table 5 contains 'MMlU-Benchmark' (lowercase l) and 'MUIRBENCH'; use consistent naming (MMIU and MuirBench) throughout.
  5. [References / Table 5] The MIRB dataset appears in Table 5 but is not included in the reference list; please add the citation.
  6. [Section 5.1] Section 5.1 reports that the rule-based extractor correctly identifies 96% of answers; please clarify how the remaining 4% are scored (e.g., marked incorrect or excluded).

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the benchmark's central claims rest on rule-based outcome scoring, human-corrected gold annotations, and fixed accepted/rejected pairs; matcher-validity concerns are not circularity.

full rationale

MMRB's central claims are self-contained. The benchmark is constructed from existing multi-image task pools, with GPT-4o-generated CoT annotations refined by expert human annotators (Sections 4.2.2-4.2.3); the outcome scores use a rule-based answer extractor with roughly 96% extraction accuracy (Section 5.1), and the reward-model subset uses fixed accepted/rejected pairs derived from human corrections, with order-randomized test sets (Section 4.3). No fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem forces the design. The process score is computed by Qwen3-32B against human-refined gold reasoning steps, so it is not defined in terms of the evaluated model's own outputs; the fact that GPT-4o drafted the initial gold solutions and is also among the evaluated models is a possible self-comparison bias, not a circular reduction. The lack of reported matcher validation and the inconsistency between the Appendix B recall-style formula and the Appendix H per-sentence prompt are measurement-validity concerns that affect interpretability of the process-score columns, but they do not make any result equivalent to its input by construction. Appendix D's admission that roughly 18% of rule-judged errors could be considered acceptable answers is a scoring accuracy limitation, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on the assumption that LLM-generated, human-corrected reasoning paths are a valid gold standard, that the source datasets represent multi-image reasoning, and that the open-source sentence matcher approximates human judgment. No free parameters or invented entities appear.

assumptions (4)
  • domain assumption GPT-4o-generated reasoning trajectories, after human correction, constitute a valid gold standard for reasoning steps.
    Used to construct all CoT annotations (Sections 4.2.2 and 4.2.3). If these paths are not aligned with human reasoning, process scores lose meaning.
  • domain assumption Human correction of 25 percent of samples is sufficient to remove systematic bias from GPT-4o annotations.
    No inter-annotator agreement metrics are reported in Section 4.2.3.
  • domain assumption The Qwen3-32B sentence-level matcher approximates human judgment on step correctness.
    No validation against human ratings is provided in Section 5.2 or Appendix B.
  • domain assumption The 22 source datasets and the GPT-4o-assisted selection of 92 sub-tasks are representative of multi-image reasoning.
    Selection uses a GPT-4o CoT test and removal of hard-math questions, which may introduce bias (Section 4.2.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark." pith.science (2026). https://pith.science/paper/ZY7HIXH7

@misc{pith2026250604280,
  author       = {Pith},
  title        = {Pith review of: Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZY7HIXH7}},
  note         = {Machine review of arXiv:2506.04280}
}
abstract

With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the $\textbf{Multimodal Multi-image Reasoning Benchmark (MMRB)}$, the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises $\textbf{92 sub-tasks}$ covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on $\textbf{40 MLLMs}$, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.

Figures

Figures reproduced from arXiv: 2506.04280 by the authors.

Figure 1
Figure 1. Overview of the MMRB benchmark, which eval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Perceptually driven VS. Semantically driven tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Data annotation pipeline of MMRB benchmark and the evaluation process. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Linear regression on non-thinking models’ efficacy. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Standard Error of Mean (SEM) VS Sample Size (n) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Standard Error of Mean (SEM) VS Sample Size (n) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Error analysis of Open AI o1’s model response. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Examples of MMRB benchmark [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Examples of MMRB benchmark [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Prompt Template for Reasoning Step Generation by GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Prompt Template for Reasoning Steps Evaluation by Qwen3-32B. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Prompt Template for Reward Model Evaluation – Gemini 2. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Prompt Template for Reward Model Evaluation – GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Prompt Template for Reward Model Evaluation – GPT-4o-mini. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Prompt Template for Reward Model Evaluation – llava-critic. [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: Prompt Template for Reward Model Evaluation – R1-reward. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A multi-view vision-language model trained on 20,000 fetal ultrasound reports generates clinical text and diagnoses, reportedly outperforming general and medical baselines.

Reference graph

Works this paper leans on

67 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Rohan Anil et al. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312.11805(2023)

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    arXiv 2025

  4. [4]

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che

  5. [5]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al . 2024. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling.arXiv preprint arXiv:2412.05271(2024)

    arXiv 2024

  6. [6]

    Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. 2025. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 23678–23686

    work page 2025

  7. [7]

    OpenCompass Contributors. 2023. OpenCompass: A Universal Evaluation Plat- form for Foundation Models. https://github.com/open-compass/opencompass

    2023

  8. [8]

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy

Show all 67 references
  1. [9]

    Eli Friedman and Fred Fontaine. 2018. Generalizing across multi-objective reward functions in deep reinforcement learning.arXiv preprint arXiv:1809.06364(2018)

    2018 arXiv

  2. [10]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision. Springer, 148–166

    2024

  3. [11]

    Minghe Gao, Xuqi Liu, Zhongqi Yue, Yang Wu, Shuang Chen, Juncheng Li, Siliang Tang, Fei Wu, Tat-Seng Chua, and Yueting Zhuang. 2025. Benchmark- ing multimodal cot reward model stepwise by visual program.arXiv preprint arXiv:2504.06606(2025)

    2025 arXiv

  4. [12]

    Michael H Goldstein and Jennifer A Schwade. 2009. From birds to words: Percep- tion of structure in social interactions guides vocal development and language learning. (2009)

    2009

  5. [13]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720(2024)

    2024 arXiv

  6. [14]

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483(2024)

    2024 arXiv

  7. [15]

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. 2025. MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robust- ness, and Efficiency.arXiv preprint arXiv:2502.09...

    2025 arXiv

  8. [16]

    Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Sreenivas Gollapudi, Dee Guo, et al

  9. [17]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326(2024)

    2024 arXiv

  10. [18]

    Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat- Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. 2023. Fine- tuning multimodal llms to follow zero-shot demonstrative instructions.arXiv preprint arXiv:2308.04152(2023)

    2023 arXiv

  11. [19]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22195–22206

    2024

  12. [20]

    Remi: A dataset for reasoning with multiple images.Advances in Neural Information Processing Systems37 (2024), 60088–60109

    2024

  13. [21]

    Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al . 2024. MIBench: Evaluating Multi- modal Large Language Models over Multiple Images. InProceedings of the 2024 Conference on Empirical Methods in Natural Lang...

    2024

  14. [22]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255(2023)

    2023 arXiv

  15. [23]

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. 2024. Mmiu: Multimodal multi- image understanding for evaluating large vision-language models.arXiv preprint arXiv:2408.02718(2024)

    2024 arXiv

  16. [24]

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. 2024. VLFeedback: A Large- Scale AI Feedback Dataset for Large Vision-Language Models Alignment. In Proceedings of the 2024 Conference on Empirical Methods...

    2024

  17. [25]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles.International Journal on Digital Libraries23, 3 (2022), 289–301

    2022

  18. [26]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  19. [27]

    Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. Milebench: Benchmarking mllms in long context.arXiv preprint arXiv:2404.18532(2024)

    2024 arXiv

  20. [28]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems36 (2023), 53728–53741

    2023

  21. [29]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2024. Aligning Large Multimodal Models with Factually Augmented RLHF. InFindings of the Association for Computational Linguistics ACL 2024...

    2024

  22. [30]

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. 2025. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186(2025)

    2025 arXiv

  23. [31]

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. 2024. Muirbench: A comprehensive benchmark for robust multi-image understanding.arXiv preprint arXiv:2406.09411(2024)

    2024 arXiv

  24. [32]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems37 (2024), 95095–95169

    2024

  25. [33]

    Alane Suhr and Yoav Artzi. 2019. Nlvr2 visual bias analysis.arXiv preprint arXiv:1909.10411(2019)

    2019 arXiv

  26. [34]

    Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. 2025. Visualprm: An effective process reward model for multimodal reasoning.arXiv preprint arXiv:2503.10291(2025)

    2025 arXiv

  27. [35]

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Fuxiao Liu, Gedas Bertasius, et al. 2024. Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences. InProceedings of the 62nd Annual...

    2024

  28. [36]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  29. [37]

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. 2023. Q-bench: A benchmark for general-purpose foundation models on low-level vision.arXiv preprint arXiv:2309.14181(2023)

    2023 arXiv

  30. [38]

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. 2024. Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization.arXiv preprint arXiv:2411.10442(2024)

    2024 arXiv

  31. [39]

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. 2024. Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440 (2024)

    2024 arXiv

  32. [40]

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al . 2025. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279(2025)

    2025 arXiv

  33. [41]

    Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  34. [42]

    Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. 2025. Multi- modal rewardbench: Holistic evaluation of reward models for vision language models.arXiv preprint arXiv:2502.14191(2025)

    2025 arXiv

  35. [43]

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. 2024. Llava-critic: Learning to evaluate multi- modal models.arXiv preprint arXiv:2410.02712(2024)

    2024 arXiv

  36. [44]

    Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al . 2025. InternLM- XComposer2. 5-Reward: A Simple Yet Effective Multi-Modal Reward Model.arXiv preprint arXiv:2501.12368(2025)

    2025 arXiv

  37. [45]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240(2024)

    2024 arXiv

  38. [46]

    Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christopher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. 2024. Worldqa: Multimodal world knowledge in videos through long-chain reasoning.arXiv preprint arXiv:2405.03272(2024)

    2024 arXiv

  39. [47]

    Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, et al. 2025. R1-Reward: Training Mul- timodal Reward Model Through Stable Reinforcement Learning.arXiv preprint arXiv:2505.02835(2025)

    2025 arXiv

  40. [48]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference...

    2024

  41. [49]

    Xiongtao Zhou, Jie He, Lanyu Chen, Jingyu Li, Haojing Chen, Víctor Gutiérrez- Basulto, Jeff Z Pan, and Hanjie Chen. 2024. MiCEval: Unveiling Multimodal Chain of Thought’s Quality via Image Description and Reasoning Steps.arXiv preprint arXiv:2410.14668(2024)

    2024 arXiv

  42. [50]

    Do not in- clude any explanation. Only output your final answer in the ex- act format: Answer[<letter> or <your_answer_here>]

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint arXiv:2504.10479(2025). Evaluating M...

    2025 arXiv

  43. [53]

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal Chain-of-Thought Reasoning in Language Models. arXiv preprint arXiv:2302.00923(2023)

    2023 arXiv

  44. [56]

    A human annotated step-by-step solution to solving this task

  45. [57]

    reasoning step

    A Chain-of-Thought (CoT) answer generated by an LLM to be evaluated. Your task: - Compare the CoT answer to the step-by-step solution. - Match the related sentences from the CoT answer to the corresponding steps in the solution. - For each reasoning step in the CoT answer, det...

    2025

  46. [61]

    equally good

    Usefulness (informativeness/helpfulness) Please strictly choose the better response. You must choose a better answer, you can’t judge them as "equally good". Try your best to select a better one. Output in the following format: The better response: [1]. Because... or The bette...

    2025

  47. [65]

    equally good

    Usefulness (informativeness/helpfulness) Please strictly choose the better response. You must choose a better answer, you can’t judge them as "equally good". Try your best to select a better one. Output in the following format: The better response: [1]. Because... or The bette...

    2025

  48. [66]

    Accuracy of target description

  49. [67]

    Accuracy of relationship description

  50. [68]

    Accuracy of attribute description

  51. [69]

    equally good

    Usefulness (informativeness/helpfulness) Please strictly choose the better response. You must choose a better answer, you can’t judge them as "equally good". Try your best to select a better one. Output in the following format: The better response: [1]. Because... or The bette...

    2025

  52. [70]

    Responses should minimize the mention of objects not present in the ground truth answer, and inaccuracies in the description of existing objects

    Accuracy in Object Description: Evaluate the accuracy of the descriptions concerning the objects mentioned in the ground truth answer. Responses should minimize the mention of objects not present in the ground truth answer, and inaccuracies in the description of existing objects

  53. [71]

    Rank higher the responses that least misrepresent these relationships

    Accuracy in Depicting Relationships: Consider how accurately the relationships between objects are described compared to the ground truth answer. Rank higher the responses that least misrepresent these relationships

  54. [72]

    Responses should avoid inaccuracies in describing the characteristics of the objects present

    Accuracy in Describing Attributes: Assess the accuracy in the depiction of objects’ attributes compared to the ground truth answer. Responses should avoid inaccuracies in describing the characteristics of the objects present

  55. [73]

    Equally good

    Helpfulness: Consider whether the generated text provides valuable insights, additional context, or relevant information that contributes positively to the user’s comprehension of the image. Assess whether the language model accurately follows any specific instructions or guid...

    2025

  56. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  57. [2022]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems35 (2022), 23716–23736

    2022

  58. [2023]

    InProceedings of the IEEE/CVF international conference on computer vision

    MeViS: A large-scale benchmark for video segmentation with motion expressions. InProceedings of the IEEE/CVF international conference on computer vision. 2694–2703

  59. [2024]

    M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought.arXiv preprint arXiv:2405.16473(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.