Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

This paper argues that judging multimodal answers should be tested and trained as three distinct capabilities—comparing chains of thought, ignoring response length, and catching process errors—and that MCTS-generated preference pairs delive

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:51 UTC pith:P5BISNES

load-bearing objection M-JudgeBench is a genuinely useful capability benchmark and the MCTS recipe shows real gains, but the SOTA claim in §4.2.2 is undercut by the paper's own Table 2. the 5 major comments →

arxiv 2603.00546 v2 pith:P5BISNES submitted 2026-02-28 cs.AI cs.CV

Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation

Zeyu Chen , Huanjin Yao , Ziwang Zhao , Min Yang This is my paper
classification cs.AI cs.CV
keywords MLLM-as-a-judgecapability-oriented benchmarkchain-of-thought comparisonlength biasprocess error detectionMonte Carlo Tree Searchpreference data generationmultimodal evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish two linked claims. First, the right way to test an MLLM-as-a-judge is by the core judgment capabilities it needs—comparing two chains of thought, resisting response length, and spotting errors inside a reasoning process—rather than by task type; the authors build M-JudgeBench around these three capabilities and show that current judge models fail them, including strong proprietary ones. Second, training a judge on a small amount of MCTS-generated pairwise reasoning trajectories that vary in correctness and length substantially improves judge reliability, and the resulting M-Judger models outperform both open-source judge baselines and the original base models on M-JudgeBench and two existing judge benchmarks. If true, the field gains a diagnostic tool for finding why a judge is unreliable and a cheap data recipe for fixing it.

Core claim

A reliable judge requires two independently testable skills: deciding result correctness across reasoning styles and lengths, and detecting process errors even when the final answer is correct. The paper's central discovery is that most existing judge models, including proprietary ones, are poor at these skills—especially at resisting length bias and detecting subtle process flaws—and that structured Monte Carlo Tree Search rollouts produce exactly the contrastive data needed to train those skills. Adding about 13k such samples at either the supervised fine-tuning stage or the reinforcement-learning stage improves judge accuracy on new and existing benchmarks without degrading performance el

What carries the argument

M-JudgeBench is a benchmark of 3,712 curated pairwise instances organized into ten subtasks under three tasks: pairwise chain-of-thought comparison, length-bias avoidance, and process-error detection. Judge-MCTS is the data-generation framework that pairs with it: it runs Monte Carlo Tree Search over reasoning steps to produce four response classes—short-correct, short-error, long-correct, long-error—and then forms chosen–rejected pairs that differ in both correctness and length. The machinery works because the resulting supervision separates the dimensions a judge must attend to (correctness, process quality) from the dimensions it must ignore (length, style, wording).

Load-bearing premise

The benchmark labels are trustworthy: final answers are extracted from chains of thought and checked against ground truth by an LLM with exact-match filtering, and no human validation of the extracted labels is reported.

What would settle it

Take a random sample of M-JudgeBench pairs and have independent human annotators label which response is better without seeing the automated labels; if human agreement with the benchmark labels falls well below the level needed for reliable model ranking, then both the reported weaknesses of current judges and the reported improvements of M-Judger rest on noisy ground truth.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Judge evaluation should be organized around judgment capabilities rather than task categories; a benchmark built this way reveals failure modes that task-type benchmarks miss.
  • Length bias and process-error blindness are widespread, even in the strongest proprietary judge systems, so they are not solved by scale alone.
  • Adding a small amount of MCTS-generated reasoning pairs improves judge models across multiple base architectures, in both supervised and reinforcement-learning training settings.
  • The MCTS data recipe is compatible with existing judge benchmarks: it maintains or improves accuracy on the older benchmarks while lifting performance on M-JudgeBench.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same four-class pairing idea (short/long × correct/error) would likely transfer to text-only judge training, where process-error trajectories could be generated by the same tree-search mechanism without multimodal perception.
  • Because the reported gains come from only 13k extra samples, the bottleneck in judge training may be the contrastive structure of the data rather than its volume; scaling MCTS rollouts further or adding process-error noise directly into the tree could yield additional gains.
  • The benchmark's label-construction pipeline could be made stronger by replacing exact-match answer filtering with symbolic or verifier-based checking, which would make the reported diagnosis of judge weaknesses independent of a single extraction model.
  • The very low accuracy on length-bias subtasks suggests that judge models are poorly calibrated about confidence, so reporting calibration alongside pairwise accuracy on M-JudgeBench would sharpen the diagnosis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces M-JudgeBench, a capability-oriented benchmark for multimodal LLM judges, organized into ten subtasks across pairwise Chain-of-Thought comparison, length-bias avoidance, and process-error detection. Benchmark pairs are built by rolling out several MLLMs over seed questions, extracting answers with GPT-4.1, and filtering by exact match to ground truth; process-error pairs are produced by prompting models to inject subtle errors into correct reasoning chains. The paper also proposes Judge-MCTS, an OmegaPRM-style MCTS rollout scheme that yields short/long correct/error trajectories, and trains M-Judger models by SFT on 142k open-source pairs plus 13k MCTS pairs, optionally followed by DAPO RL. Main experiments report accuracy on M-JudgeBench, VL-RewardBench, and Multimodal RewardBench, with the claim in Sec. 4.2.2 that Qwen3-VL-8B enhanced by Judge-MCTS achieves state-of-the-art performance across all three benchmarks.

Significance. If the results held, the capability-oriented decomposition of judge reliability would be a useful step forward, and the MCTS-based data-generation recipe would offer a cost-efficient way to improve judge models with only 13k additional samples. The paper has concrete strengths: it evaluates a broad set of closed and open models, releases benchmark data and evaluation scripts, and reports gains on an existing benchmark (VL-RewardBench) that are independent of the authors' benchmark. Those positives make the contribution potentially valuable. However, the evidence is currently undermined by an internal contradiction in the stated SOTA claim, inconsistent numbers between Table 2 and Table 6, an unvalidated label pipeline, and a nonstandard benchmark subset. The significance is therefore conditional on correcting these issues.

major comments (5)
  1. [Sec. 4.2.2, Table 2] The claim that "Qwen3-VL-8B-Instruct enhanced by Judge-MCTS achieves state-of-the-art performance across all three benchmarks" is contradicted by the same table. On Multimodal RewardBench, R1-Reward scores 80.10, while M-Judger-RL-Qwen8B scores 65.52 and M-Judger-SFT-Qwen8B 65.67. The RL variant is also lower than its own base model (65.52 vs 65.74), which contradicts the statement in Sec. 4.2.2 that MCTS-augmented training "maintains or improves" existing-benchmark accuracy. The SOTA claim must be corrected, or the evaluation protocol re-run on a consistent subset with all baselines.
  2. [Table 2 vs. Table 6] The reported numbers for the same model are not internally consistent. M-Judger-RL-Qwen8B has overall M-JudgeBench accuracy 62.42 in Table 2 but 62.93 in Table 6. The task-level aggregates in Table 2 (Pairwise CoT 67.45, Length bias 44.53, Process error 92.14) do not match the subtask rows in Table 6; simple averaging of the four Pairwise CoT rows gives about 68.2, and the Length bias rows average about 53.4, not 44.53. If the task aggregates are sample-size weighted, the weighting should be stated; otherwise the two tables must be reconciled. This inconsistency undermines confidence in all reported experimental results.
  3. [Sec. 2.2] Benchmark label reliability is not established. Final answers are extracted by GPT-4.1 and matched to seed ground truth via exact-match filtering, but no human validation or inter-annotator agreement is reported. For the process-error subtasks, the correctness of the injected error and the preservation of the final answer are not verified beyond an LLM prompt. Since every conclusion about model weaknesses and improvements in Secs. 4.2.1-4.2.2 inherits these labels, a human audit of at least a random sample (and ideally the full benchmark) is necessary before M-JudgeBench can support claims about judge capabilities.
  4. [Sec. 4.2.2, Multimodal RewardBench] Excluding safety tasks from Multimodal RewardBench is a protocol change that makes comparison with published baseline numbers nontrivial. The 80.10 reported for R1-Reward may come from the full benchmark, so comparing it against M-Judger on a filtered subset can be misleading. Please either report scores on the full benchmark or explicitly define the subset and re-evaluate all baselines on exactly that subset.
  5. [Sec. 4.1.1] The prompt protocols are not uniform across model classes: specialized judge models (Unified Reward, UnifiedReward-Think, R1-Reward) use their original prompts with an appended instruction, while M-Judger and general-purpose models use a custom direct-output prompt. This confounds model capability with prompt adaptation, especially for the pairwise comparison task where output format matters. A prompt-sensitivity analysis or a unified protocol for all models is needed to make the comparisons fair.
minor comments (6)
  1. [Sec. 4.2.1, Table 1] Typographical issues: "performace" should be "performance", and "LLaV A" should be "LLaVA" throughout Tables 1 and 6.
  2. [Sec. F.1] The GitHub URL contains a space ("M Judger") and appears malformed; the repository link should be corrected.
  3. [Tables 2, 3, 6] Naming inconsistency: "M-Judger-SFT-Qwen4B" and "M-Judger-RL-Qwen4B" do not identify the underlying Qwen3-VL-4B-Instruct model; make the base model name explicit in the table row labels.
  4. [Table 4 vs. Sec. 2.3] The seed data table sums to 3,276 questions, while the benchmark contains 3,712 instances. Clarify whether one seed question can yield multiple pairs and provide per-subtask pair counts so the arithmetic is transparent.
  5. [Sec. 3.2 vs. Sec. 4.1.2] The method is described as following the OmegaPRM framework, but the experimental setup says the MM-PRM framework was adopted. State the relationship between the two and describe the value function used in MCTS rollouts.
  6. [Sec. F.2] Training data and fine-tuned models are said to "be made publicly available soon". For reproducibility, these should be released at submission time, not promised for a future date.

Circularity Check

0 steps flagged

No circular derivation; minor self-citations not load-bearing.

full rationale

The derivation chain is self-contained. M-JudgeBench labels come from external seed benchmarks' ground-truth (MMMU/MMMU-Pro/MMStar/MMReason/M3CoT/MathVision/MathVerse); GPT-4.1 is used only to extract final answers from CoT rollouts, not to define the correctness criterion. The M-Judger training data are generated separately (Judge-MCTS from ThinkLite-VL/MM-K12 via OmegaPRM/MM-PRM plus an open-source preference mixture), with explicit filtering of overlap against evaluation benchmarks, and gains are corroborated on external VL-RewardBench and Multimodal RewardBench, so no fitted parameter is renamed as a prediction. The fact that the training pairs (SC/SE/LC/LE) target the same capabilities that M-JudgeBench measures is construct alignment, not definitional circularity, because the benchmark is not constructed from M-Judger outputs. The only self-citations are co-author Huanjin Yao's MMReason (one of seven seed sources) and Mulberry (a related MCTS reference; the actual algorithm is based on OmegaPRM/MM-PRM); neither is load-bearing. Separate correctness concerns, not circularity: Table 2's SOTA claim is contradicted by R1-Reward's 80.10 vs M-Judger-RL-Qwen8B's 65.52 on Multimodal RewardBench, and the same model's M-JudgeBench overall accuracy differs between Table 2 (62.42) and Table 6 (62.93).

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical entities are posited; M-JudgeBench, Judge-MCTS, and M-Judger are datasets/models, not independent scientific entities with falsifiable handles beyond the reported experiments.

free parameters (4)
  • MCTS rollout temperature = 1.0
    Chosen for Judge-MCTS rollouts; not optimized, but affects trajectory diversity and therefore all downstream M-Judger results (Sec. 4.1.2).
  • MCTS max search count = 50
    Hand-selected rollout budget; controls length/correctness mix of generated pairs (Sec. 4.1.2).
  • MCTS rollouts per prompt = 4
    Number of rollouts per prompt; determines the supply of SC/SE/LC/LE pairs (Sec. 4.1.2).
  • Data mixture sizes = 142k open-source + 13k MCTS; 16k open-source + 13k MCTS for RL
    Chosen proportions; paper argues small MCTS injection is effective but no sweep is shown (Sec. 4.1.2).
axioms (5)
  • domain assumption Ground-truth answers in seed benchmarks (MMMU, MMMU-Pro, MMStar, MMReason, M3CoT, MathVision, MathVerse) are correct.
    M-JudgeBench labels are defined against these answers (Sec. 2.2); if seeds are wrong, all correctness labels shift.
  • domain assumption GPT-4.1 extracts final answers from CoT rollouts accurately enough for exact-match filtering.
    Used to decide correct vs incorrect responses in M-JudgeBench (Sec. 2.2); no human verification reported.
  • ad hoc to paper Noise injection preserves the final answer and introduces exactly one process-level error.
    Process-error detection assumes clean vs perturbed differ only in the intended error type (Sec. 2.2.3, App. D); if the injected text changes the answer or adds artifacts, the task is confounded.
  • domain assumption Seed1.6-VL acting as value/judge model in MCTS gives reliable step-level correctness estimates.
    MCTS trajectory labels (SC/SE/LC/LE) are guided by Seed1.6-VL (Sec. 4.1.2); errors propagate into training pairs.
  • ad hoc to paper The ten subtasks adequately span 'core judgment capabilities' relevant to judge reliability.
    The benchmark design claims this decomposition; no external validation that these dimensions are necessary/sufficient (Sec. 2.1).

pith-pipeline@v1.3.0-alltime-deepseek · 22459 in / 14364 out tokens · 124318 ms · 2026-08-02T19:51:13.737784+00:00 · methodology

0 comments
read the original abstract

Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains. Evaluating the capability and reliability of MLLM-as-a-judge systems is therefore essential for ensuring trustworthy assessment. Existing judge benchmarks categorize samples by task types but fail to capture the fundamental judgment capabilities required for reliable evaluation. In this work, we introduce M-JudgeBench, a ten-dimensional capability-oriented benchmark designed to comprehensively assess the judgment abilities of MLLMs. Our benchmark decomposes evaluation into pairwise Chain-of-Thought (CoT) comparison, length bias avoidance, and process error detection tasks, jointly covering ten fine-grained subtasks. This design enables diagnosis of model reliability across reasoning styles, response lengths, and cross-model variations. Systematic evaluation uncovers the systematic weaknesses in existing MLLM-as-a-judge systems. To address this issue, we further propose Judge-MCTS, a data construction framework generating pairwise reasoning trajectories with various correctness and length. Using Judge-MCTS, we construct an MCTS-augmented dataset and train M-Judger, a series of strong judge models. Extensive experiments demonstrate the superiority of M-Judger on existing judge benchmarks as well as M-JudgeBench. Overall, our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework, paving the way for future research on judge model evaluation and capability-driven judge training.

Figures

Figures reproduced from arXiv: 2603.00546 by Huanjin Yao, Min Yang, Zeyu Chen, Ziwang Zhao.

Figure 1
Figure 1. Figure 1: Comparison between existing judge benchmarks and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of M-JudgeBench. The figure illustrates the data construction methods and resulting task types in M-JudgeBench. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: It provides comprehensive and balanced cover￾age of core judgment capabilities that existing MLLM judge benchmarks overlook [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    MUSE is a new benchmark and three-stage evaluation protocol for text-to-CAD generation that assesses functionality, manufacturability, and assemblability of B-Rep assemblies beyond geometric similarity.

Reference graph

Works this paper leans on

78 extracted references · 20 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    Pith/arXiv arXiv 2025

  2. [2]

    MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark

    Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. InForty- first International Conference on Machine Learning, 2024. 1

    2024

  3. [3]

    Are we on the right way for eval- uating large vision-language models? InAdvances in Neural Information Processing Systems, pages 27056–27087

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for eval- uating large vision-language models? InAdvances in Neural Information Processing Systems, pages 27056–27087. Cur- ran Associates, Inc., 2024. 2, 7

    2024

  4. [4]

    M 3CoT: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3CoT: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8199–8221, Bangkok, Thailand, 2024. Association for Com- putational Linguistics. 2, 7

    2024

  5. [5]

    Every Sample Matters: Leveraging Mixture-of-Experts and High- Quality Data for Efficient and Accurate Code LLM, 2025

    Codefuse, Ling Team, Wenting Cai, Yuchen Cao, Chaoyu Chen, Chen Chen, Siba Chen, Qing Cui, Peng Di, Jun- peng Fang, Zi Gong, Ting Guo, Zhengyu He, Yang Huang, Cong Li, Jianguo Li, Zheng Li, Shijie Lian, BingChang Liu, Songshan Luo, Shuo Mao, Min Shen, Jian Wu, Jiaolong Yang, Wenjie Yang, Tong Ye, Hang Yu, Wei Zhang, Zhen- duo Zhang, Hailin Zhao, Xunjin Zh...

    Pith/arXiv arXiv 2025

  6. [6]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 3

    Pith/arXiv arXiv 2025

  7. [7]

    Efficient selectivity and backup operators in monte-carlo tree search

    R ´emi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. InComputers and Games, pages 72–83, Berlin, Heidelberg, 2007. Springer Berlin Heidel- berg. 2

    2007

  8. [8]

    Mm-ifengine: Towards multimodal instruction following

    Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Mm-ifengine: Towards multimodal instruction following. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1099–1109, 2025. 4, 7

    2025

  9. [9]

    MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scal- able Step-Level Supervision, 2025

    Lingxiao Du, Fanqing Meng, Zongkai Liu, Zhixiang Zhou, Ping Luo, Qiaosheng Zhang, and Wenqi Shao. MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scal- able Step-Level Supervision, 2025. arXiv:2505.13427 [cs]. 5

    Pith/arXiv arXiv 2025

  10. [10]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025. 3

    Pith/arXiv arXiv 2025

  11. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 5

    Pith/arXiv arXiv 2024

  12. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InPro- ceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 12

    2023

  13. [13]

    Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs, 2024

    Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xian- gru Peng, and Jiaya Jia. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs, 2024. arXiv:2406.18629 [cs]. 4, 7

    Pith/arXiv arXiv 2024

  14. [14]

    From generation to judgment: Op- portunities and challenges of LLM-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Op- portunities and challenges of LLM-as-a-judge. InProceed- ings of the 2025 Conference on Empirical Methods in Natu- ral Language Processing, pages ...

    2025

  15. [15]

    Vl- rewardbench: A challenging benchmark for vision-language generative reward models

    Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yi- fan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, Lingpeng Kong, and Qi Liu. Vl- rewardbench: A challenging benchmark for vision-language generative reward models. InCVPR, 2025. 1

    2025

  16. [16]

    Skywork-reward: Bag of tricks for reward modeling in llms

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. CoRR, abs/2410.18451, 2024. 4, 7

    Pith/arXiv arXiv 2024

  17. [17]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 5

    2024

  18. [18]

    MIA-DPO: Multi-image augmented di- rect preference optimization for large vision-language mod- els

    Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. MIA-DPO: Multi-image augmented di- rect preference optimization for large vision-language mod- els. InThe Thirteenth International Conference on Learning Representations, 2025. 4, 7

    2025

  19. [19]

    Improve Mathemat- ical Reasoning in Language Models by Automated Process Supervision, 2024

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve Mathemat- ical Reasoning in Language Models by Automated Process Supervision, 2024. arXiv:2406.06592 [cs]. 5

    Pith/arXiv arXiv 2024

  20. [20]

    Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Ze- tong Zhou, Shuang Gong, Yi Gui, Yao Wan, and Philip S. Yu. Judge anything: Mllm as a judge across any modal- 9 ity. InProceedings of the 31st ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining V .2, page 5742–5753, New York, NY , USA, 2025....

    2025

  21. [21]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Process- ing Systems, pages 53728–53741. Curran Associates, Inc.,

  22. [22]

    GLM-V . Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali C...

    Pith/arXiv arXiv 2025

  23. [23]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. 2

    2025

  24. [24]

    MiMo-VL Technical Report, 2025

    Xiaomi LLM-Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhix- ian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Y...

    Pith/arXiv arXiv 2025

  25. [25]

    Mea- suring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. InAdvances in Neural Information Processing Sys- tems, pages 95095–95169. Curran Associates, Inc., 2024. 2, 7

    2024

  26. [26]

    Self- taught evaluators.arXiv preprint arXiv:2408.02666, 2024

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self- taught evaluators.arXiv preprint arXiv:2408.02666, 2024. 4

    Pith/arXiv arXiv 2024

  27. [27]

    Enhancing the Rea- soning Ability of Multimodal Large Language Models via Mixed Preference Optimization, 2025

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the Rea- soning Ability of Multimodal Large Language Models via Mixed Preference Optimization, 2025. arXiv:2411.10442 [cs]. 4, 7

    Pith/arXiv arXiv 2025

  28. [28]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Sheng- long Ye, Jie Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 2

    Pith/arXiv arXiv 2025

  29. [29]

    SoTA with less: MCTS-guided sample selection for data-efficient visual reasoning self-improvement

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Lin- jie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Li- juan Wang. SoTA with less: MCTS-guided sample selection for data-efficient visual reasoning self-improvement. InThe Thirty-ninth Annual Conference on Neural Information Pro- cessing Systems, 2025. 5

    2025

  30. [30]

    Unified multimodal chain-of-thought reward model through reinforcement fine- tuning

    Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Unified multimodal chain-of-thought reward model through reinforcement fine- tuning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 2

    2025

  31. [31]

    Unified Reward Model for Multimodal Understand- ing and Generation, 2025

    Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified Reward Model for Multimodal Understand- ing and Generation, 2025. arXiv:2503.05236 [cs]. 2, 4, 7

    Pith/arXiv arXiv 2025

  32. [32]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 1

    2022

  33. [33]

    J1: Incentiviz- ing thinking in llm-as-a-judge via reinforcement learning

    Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. J1: Incentiviz- ing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505.10320, 2025. 4

    arXiv 2025

  34. [34]

    Multimodal Preference Data Synthetic Alignment with Re- ward Model, 2024

    Robert Wijaya, Ngoc-Bao Nguyen, and Ngai-Man Cheung. Multimodal Preference Data Synthetic Alignment with Re- ward Model, 2024. arXiv:2412.17417 [cs]. 4, 7

    arXiv 2024

  35. [35]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Trans- formers: State-of-t...

    2020

  36. [36]

    Monte carlo tree search boosts reasoning via iterative prefer- ence learning

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative prefer- ence learning. InThe First Workshop on System-2 Reasoning at Scale, NeurIPS’24, 2024. 2 10

    2024

  37. [37]

    Llava- critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava- critic: Learning to evaluate multimodal models. InProceed- ings of the Computer Vision and Pattern Recognition Con- ference, pages 13618–13628, 2025. 1

    2025

  38. [38]

    Chen, Wenzheng Liu, Wei Zhang, Wenjie Zeng, Xikun Zhang, Jingyi Zhang, YuXin Song, Wenhao Wu, and Dacheng Tao

    Huanjin Yao, Jiaxing Huang, Yawen Qiu, Michael K. Chen, Wenzheng Liu, Wei Zhang, Wenjie Zeng, Xikun Zhang, Jingyi Zhang, YuXin Song, Wenhao Wu, and Dacheng Tao. Mmreason: An open-ended multi-modal multi-step reason- ing benchmark for mllms toward agi. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 273–283, 2025. 2, 7

    2025

  39. [39]

    Mulberry: Em- powering MLLM with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, YuXin Song, Haocheng Feng, Li Shen, and Dacheng Tao. Mulberry: Em- powering MLLM with o1-like reasoning and reflection via collective monte carlo tree search. InThe Thirty-ninth An- nual Conference on Neural Information Processing Systems,

  40. [40]

    Multimodal RewardBench: Holistic Evalua- tion of Reward Models for Vision Language Models, 2025

    Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal RewardBench: Holistic Evalua- tion of Reward Models for Vision Language Models, 2025. arXiv:2502.14191 [cs]. 1

    Pith/arXiv arXiv 2025

  41. [41]

    A survey on multimodal large language models.National Science Review, 11(12): nwae403, 2024

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models.National Science Review, 11(12): nwae403, 2024. 1

    2024

  42. [42]

    DAPO: An Open-Source LLM Reinforcement Learn- ing System at Scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xi- aochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xi- angpeng Wei, Hao Zhou, Jingjing Li...

    Pith/arXiv arXiv 2025

  43. [43]

    Rlaif-v: Open-source ai feed- back leads to super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat- Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feed- back leads to super gpt-4v trustworthiness. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)...

    2025

  44. [44]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for...

    2024

  45. [45]

    MMMU-pro: A more robust multi-discipline multi- modal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neu- big. MMMU-pro: A more robust multi-discipline multi- modal understanding benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pa...

    2025

  46. [46]

    InternLM-XComposer2.5-reward: A simple yet effective multi-modal reward model

    Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, and Jiaqi Wang. InternLM-XComposer2.5-reward: A simple yet effective multi-modal reward model. InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 6547– 6563, Vienna, Austria, 2025. Asso...

    2025

  47. [47]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InComputer Vision – ECCV 2024, pages 169–186, Cham, 2025

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InComputer Vision – ECCV 2024, pages 169–186, Cham, 2025. Springer Nature Switzerland. 2, 7

    2024

  48. [48]

    Gpt-4v (ision) as a general- ist evaluator for vision-language tasks.arXiv preprint arXiv:2311.01361, 2023

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) as a general- ist evaluator for vision-language tasks.arXiv preprint arXiv:2311.01361, 2023. 1

    Pith/arXiv arXiv 2023

  49. [49]

    R1-Reward: Train- ing Multimodal Reward Model Through Stable Reinforce- ment Learning, 2025

    Yi-Fan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, and Liang Wang. R1-Reward: Train- ing Multimodal Reward Model Through Stable Reinforce- ment Learning, 2025. arXiv:2505.02835 [cs]. 2

    Pith/arXiv arXiv 2025

  50. [50]

    Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els. InProceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Lin...

    2024

  51. [51]

    Easyr1: An efficient, scalable, multi-modality rl training framework, 2025

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework, 2025. 12

    2025

  52. [52]

    Aligning Modalities in Vision Large Language Models via Preference Fine-tuning, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning Modalities in Vision Large Language Models via Preference Fine-tuning, 2024. arXiv:2402.11411 [cs]. 4, 7 11 Supplementary Material Contents

    Pith/arXiv arXiv 2024

  53. [54]

    Capability-Oriented Evaluation Framework

    M-JudgeBench Construction 2 2.1. Capability-Oriented Evaluation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2. Data Generation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2.1 . Pairwise CoT Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . ...

  54. [55]

    Open-Source Training Data Collection

    M-Judger Training 4 3.1. Open-Source Training Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2. Judge-MCTS Data Construction Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.2.1 . Reasoning Rollout via MCTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...

  55. [56]

    Experimental Setup

    Experiments 5 4.1. Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.1.1 . Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.1.2 . Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

  56. [57]

    Benchmark Examples 3 A.1

    Conclusions 7 A . Benchmark Examples 3 A.1 . Example1. CoTs from the Same Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 A.2 . Example2. CoTs from Different Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 A.3 . Example3. Process Error Judgment . . . . . . . . . . . . . . . . . . . ....

    2023

  57. [58]

    **Gross operating surplus**: Net operating surplus + Con- sumption of fixed capital = 240,000 + 110,000 = 350,000 Rm; 3

    **Compensation of employees**: 420,000 Rm; 2. **Gross operating surplus**: Net operating surplus + Con- sumption of fixed capital = 240,000 + 110,000 = 350,000 Rm; 3. **Net taxes on production and imports**: (Taxes on production + Taxes on products) - (Subsidies on production + Subsidies on products) = (16,000 + 84,000) - (2,000 + 5,000) = 100,000 - 7,000...

  58. [59]

    **Net operating surplus**: 240,000 Rm; 3

    **Compensation of employees**: 420,000 Rm; 2. **Net operating surplus**: 240,000 Rm; 3. **Consumption of fixed capital** (depreciation): 110,000 Rm; 4. **Net taxes on production** (Taxes on production - Subsidies on production): (16,000 - 2,000 = 14,000) Rm. Adding these together: [420,000 + 240,000 + 110,000 + 14,000 = 784,000] Type:shortcot same model E...

  59. [60]

    **Group the numbers by shape.**: **Triangles:** 4, 11, 18; **Circles:** 5, 12, 16; **Squares:** 8, 19, ?

  60. [61]

    **Sum of numbers in triangles: ** 4 + 11 + 18 = 33; **Sum of numbers in circles: ** 5 + 12 + 16 = 33

    **Look for a relationship within each group.** A common trick in these puzzles is to check the sum of the numbers for each shape type. **Sum of numbers in triangles: ** 4 + 11 + 18 = 33; **Sum of numbers in circles: ** 5 + 12 + 16 = 33

  61. [62]

    We can apply this rule to the squares

    **Apply the pattern.** The pattern is that the sum of the numbers in each shape group is 33. We can apply this rule to the squares. **Sum of numbers in squares: ** 8 + 19 + ? = 33; * 27 + ? = 33; * ? = 33 - 27; * ? = 6 The number that should replace the question mark is **6**. Rejected:Let’s try to find the pattern based on the shapes and numbers given: S...

  62. [63]

    Key Observations

    Unaffected parents (1: XY , 2: XX) have an affected daughter (4: XX) and unaffected children (3: XY , 5: XX); 2. Unaffected parents (5: XX, 6: XY) have an affected son (8: XY) and an unaffected daughter (7: XX). **Analysis of Inheritance Patterns** 1. **Autosomal Dominant**: Unlikely. Affected individuals must inherit the allele from at least one affected...

  63. [64]

    Modification to the original response: Alter the image-related information in the original response to introduce subtle discrepancies from the actual image content, ensure modifications create erroneous visual interpretations while maintaining reasoning plausibility

  64. [65]

    Keep the final answer unchanged: Only modify the visual comprehension process in the original response and make sure the final answer is the same as the original one

  65. [66]

    Principle of minimal modification: Introduce only a small error in the intermediate steps to corrupt the sentence and keep changes minimally detectable to human observers. D.2. Logical Reasoning Fallacies You are an expert in semantic comprehension and modification. You need to review the instruction and original response, then modify the response to meet...

  66. [67]

    Modification to the original response: Introduce subtle logical flaws or semantic deviations in the reasoning steps while maintaining proximity to correct logic

  67. [68]

    Keep the final answer unchanged: Only modify the reasoning process in the original response and make sure the final answer is the same as the original one

  68. [69]

    Principle of minimal modification: Introduce only a small error in the intermediate steps to corrupt the sentence and keep changes minimally detectable to human observers. D.3. Incidental Mistakes You are an expert in semantic comprehension and modification. You need to review the instruction and original response, then modify the response to meet these R...

  69. [70]

    Error Types to Inject (choose only 1-2 per solution): 1.1 Spelling errors: Minor misspellings (e.g., ”solution”→”soultion”, ”calculate”→”calulate”) 1.2 Numerical errors: Small digit transpositions or value changes (e.g., ”12010”→”10210”, ”3.14”→”3.41”) 1.3 Content omissions: Skip a short phrase or half-sentence without disrupting overall flow

  70. [71]

    Preserve 95 percent of the original text

  71. [72]

    Keep the final answer unchanged: Only modify the process in the original response and make sure the final answer is the same as the original one

  72. [73]

    Modify the given correct solution text by introducing subtle, hard-to-detect errors

    Make errors appear accidental/natural and ensure the solution remains plausible and logically coherent. Modify the given correct solution text by introducing subtle, hard-to-detect errors. Make minimal changes directly to the original response without rewriting the entire solution. D.4. Output Format For the three types of noise injected in the above subs...

  73. [74]

    Prioritize factual correctness above all other factors

    **Answer Correctness**: Check whether each response provides the most accurate and relevant final answer to the given question. Prioritize factual correctness above all other factors

  74. [75]

    Identify any reasoning fallacies, invalid inferences, or irrelevant logic chains that might affect reliability

    **Reasoning Soundness**: Examine whether the reasoning steps logically lead to the final answer. Identify any reasoning fallacies, invalid inferences, or irrelevant logic chains that might affect reliability

  75. [76]

    **Perceptual Understanding**: Assess the correctness of visual grounding — whether objects, regions, or rela- tionships in the image are correctly interpreted and referenced during reasoning

  76. [77]

    The response should neither omit essential reasoning nor over-elaborate with redundant or misleading content

    **Conciseness and Coherence**: Evaluate whether the reasoning process is clear, coherent, and appropriately de- tailed. The response should neither omit essential reasoning nor over-elaborate with redundant or misleading content. Do not prefer longer reasoning by default

  77. [78]

    The decision should rely on reasoning quality, not stylistic fluency or writing pref- erence

    **Style and Robustness**: Consider whether the model maintains consistent judgment quality across different response styles or model sources. The decision should rely on reasoning quality, not stylistic fluency or writing pref- erence. Please prioritize selecting the response with the most accurate final answer as the chosen one, and then consider the tho...

  78. [2025]

    Association for Computational Linguistics. 1