{"id":"5da9eb92-2a3e-4743-b9c8-65469f826729","arxiv_id":"2603.00546","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A capability-oriented multimodal judge benchmark and MCTS-based preference-data generation improve judge models on some benchmarks, but the paper's SOTA claim is not supported by its own numbers.","lead":"This paper introduces M-JudgeBench, a 3,712-sample benchmark that breaks MLLM judge ability into ten subtasks (pairwise comparison, length-bias avoidance, process-error detection), plus Judge-MCTS, a Monte-Carlo search method that generates preference pairs for training judge models. The authors report that MCTS-augmented training improves judge accuracy, but their \"state-of-the-art across all three benchmarks\" claim is contradicted by their own Table 2.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts the stated SOTA: R1-Reward (80.10) beats M-Judger-RL-Qwen8B (65.52) on Multimodal RewardBench.","rationale":"The reader's verdict is REJECT, and my concern supports that outcome, so I recommend UNCHANGED. The reader's weakest_assumption focuses on GPT-4.1 label extraction noise, which is a valid and serious issue, but the more decisive and directly verifiable problem is the paper's own Table 2 contradicting the stated SOTA claim. This internal inconsistency alone invalidates the headline assertion regardless of label quality. It is concrete, checkable, and does not rely on external data or judgments. The label-noise concern remains important for the benchmark's validity, but the SOTA contradiction is sufficient to reject the paper as written. I partially agree with the reader because they mentioned the Multimodal RewardBench gap in their rationale but did not elevate it to the weakest assumption; I consider it the primary load-bearing flaw.","tokens_in":22694,"tokens_out":3944,"duration_ms":38824,"concrete_test":"Reproduce Table 2's Multimodal RewardBench column using the released evaluation code and the same safety-excluded subset. Compute scores for R1-Reward and M-Judger-RL-Qwen8B; if R1-Reward remains ~80.10 and M-Judger ~65.52, the SOTA claim is contradicted. Also run the released M-JudgeBench evaluation to check whether M-Judger-RL-Qwen8B overall accuracy is 62.42 or 62.93; a persistent discrepancy indicates unreliable reported numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §4.2.2 that 'Qwen3-VL-8B-Instruct enhanced by Judge-MCTS achieves state-of-the-art performance across all three benchmarks' is internally contradicted by the paper's own Table 2. On Multimodal RewardBench, R1-Reward scores 80.10, while M-Judger-RL-Qwen8B scores 65.52—a 14.6-point gap. Even the best M-Judger variant on that benchmark (M-Judger-SFT-Qwen8B at 65.67) trails R1-Reward by over 14 points. The paper does not present any other M-Judger model exceeding 80.10 on Multimodal RewardBench. Therefore, the SOTA claim is false for one of the three benchmarks, assuming the reported numbers are accurate. This is an internal inconsistency, not a matter of external consensus or contested interpretation. Additionally, the M-JudgeBench overall accuracy for M-Judger-RL-Qwen8B is 62.42 in Table 2 but 62.93 in the supplementary Table 6, indicating inconsistent reporting of the same model. The authors need to either correct the SOTA claim, re-evaluate with a consistent protocol, or clarify the subset of Multimodal RewardBench used. Without such resolution, the headline result is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces M-JudgeBench, a capability-oriented benchmark for multimodal LLM judges, organized into ten subtasks across pairwise Chain-of-Thought comparison, length-bias avoidance, and process-error detection. Benchmark pairs are built by rolling out several MLLMs over seed questions, extracting answers with GPT-4.1, and filtering by exact match to ground truth; process-error pairs are produced by prompting models to inject subtle errors into correct reasoning chains. The paper also proposes Judge-MCTS, an OmegaPRM-style MCTS rollout scheme that yields short/long correct/error trajectories, and trains M-Judger models by SFT on 142k open-source pairs plus 13k MCTS pairs, optionally followed by DAPO RL. Main experiments report accuracy on M-JudgeBench, VL-RewardBench, and Multimodal RewardBench, with the claim in Sec. 4.2.2 that Qwen3-VL-8B enhanced by Judge-MCTS achieves state-of-the-art performance across all three benchmarks.","tokens_in":23103,"tokens_out":9657,"duration_ms":97620,"significance":"If the results held, the capability-oriented decomposition of judge reliability would be a useful step forward, and the MCTS-based data-generation recipe would offer a cost-efficient way to improve judge models with only 13k additional samples. The paper has concrete strengths: it evaluates a broad set of closed and open models, releases benchmark data and evaluation scripts, and reports gains on an existing benchmark (VL-RewardBench) that are independent of the authors' benchmark. Those positives make the contribution potentially valuable. However, the evidence is currently undermined by an internal contradiction in the stated SOTA claim, inconsistent numbers between Table 2 and Table 6, an unvalidated label pipeline, and a nonstandard benchmark subset. The significance is therefore conditional on correcting these issues.","major_comments":[{"comment":"The claim that \"Qwen3-VL-8B-Instruct enhanced by Judge-MCTS achieves state-of-the-art performance across all three benchmarks\" is contradicted by the same table. On Multimodal RewardBench, R1-Reward scores 80.10, while M-Judger-RL-Qwen8B scores 65.52 and M-Judger-SFT-Qwen8B 65.67. The RL variant is also lower than its own base model (65.52 vs 65.74), which contradicts the statement in Sec. 4.2.2 that MCTS-augmented training \"maintains or improves\" existing-benchmark accuracy. The SOTA claim must be corrected, or the evaluation protocol re-run on a consistent subset with all baselines.","section":"Sec. 4.2.2, Table 2"},{"comment":"The reported numbers for the same model are not internally consistent. M-Judger-RL-Qwen8B has overall M-JudgeBench accuracy 62.42 in Table 2 but 62.93 in Table 6. The task-level aggregates in Table 2 (Pairwise CoT 67.45, Length bias 44.53, Process error 92.14) do not match the subtask rows in Table 6; simple averaging of the four Pairwise CoT rows gives about 68.2, and the Length bias rows average about 53.4, not 44.53. If the task aggregates are sample-size weighted, the weighting should be stated; otherwise the two tables must be reconciled. This inconsistency undermines confidence in all reported experimental results.","section":"Table 2 vs. Table 6"},{"comment":"Benchmark label reliability is not established. Final answers are extracted by GPT-4.1 and matched to seed ground truth via exact-match filtering, but no human validation or inter-annotator agreement is reported. For the process-error subtasks, the correctness of the injected error and the preservation of the final answer are not verified beyond an LLM prompt. Since every conclusion about model weaknesses and improvements in Secs. 4.2.1-4.2.2 inherits these labels, a human audit of at least a random sample (and ideally the full benchmark) is necessary before M-JudgeBench can support claims about judge capabilities.","section":"Sec. 2.2"},{"comment":"Excluding safety tasks from Multimodal RewardBench is a protocol change that makes comparison with published baseline numbers nontrivial. The 80.10 reported for R1-Reward may come from the full benchmark, so comparing it against M-Judger on a filtered subset can be misleading. Please either report scores on the full benchmark or explicitly define the subset and re-evaluate all baselines on exactly that subset.","section":"Sec. 4.2.2, Multimodal RewardBench"},{"comment":"The prompt protocols are not uniform across model classes: specialized judge models (Unified Reward, UnifiedReward-Think, R1-Reward) use their original prompts with an appended instruction, while M-Judger and general-purpose models use a custom direct-output prompt. This confounds model capability with prompt adaptation, especially for the pairwise comparison task where output format matters. A prompt-sensitivity analysis or a unified protocol for all models is needed to make the comparisons fair.","section":"Sec. 4.1.1"}],"minor_comments":[{"comment":"Typographical issues: \"performace\" should be \"performance\", and \"LLaV A\" should be \"LLaVA\" throughout Tables 1 and 6.","section":"Sec. 4.2.1, Table 1"},{"comment":"The GitHub URL contains a space (\"M Judger\") and appears malformed; the repository link should be corrected.","section":"Sec. F.1"},{"comment":"Naming inconsistency: \"M-Judger-SFT-Qwen4B\" and \"M-Judger-RL-Qwen4B\" do not identify the underlying Qwen3-VL-4B-Instruct model; make the base model name explicit in the table row labels.","section":"Tables 2, 3, 6"},{"comment":"The seed data table sums to 3,276 questions, while the benchmark contains 3,712 instances. Clarify whether one seed question can yield multiple pairs and provide per-subtask pair counts so the arithmetic is transparent.","section":"Table 4 vs. Sec. 2.3"},{"comment":"The method is described as following the OmegaPRM framework, but the experimental setup says the MM-PRM framework was adopted. State the relationship between the two and describe the value function used in MCTS rollouts.","section":"Sec. 3.2 vs. Sec. 4.1.2"},{"comment":"Training data and fine-tuned models are said to \"be made publicly available soon\". For reproducibility, these should be released at submission time, not promised for a future date.","section":"Sec. F.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is salvageable, but the false SOTA claim and the Table 2/Table 6 inconsistencies are serious. If the authors cannot reconcile the numbers after re-running, I would move to reject. The benchmark-label audit is also essential; without it, M-JudgeBench's validity is unsupported. I recommend major revision rather than outright rejection because the capability-oriented decomposition and the MCTS data-generation recipe are potentially useful contributions if the evidence is cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: there is one solid contribution here and one overclaim. The solid part is M-JudgeBench, which decomposes judge evaluation into pairwise CoT comparison, length-bias avoidance, and process-error detection across ten subtasks. The design surfaces real weaknesses in existing judge models — length bias is evident even in GPT-5, and process-error detection is hard for everyone, including specialized judge models. That is a genuinely useful evaluation instrument for the multimodal-judge community. The MCTS-based data generation is a reasonable adaptation of OmegaPRM/MM-PRM, and the gains from adding 13k MCTS samples are consistent across several backbones on M-JudgeBench itself. That is the best evidence in the paper.\n\nThe problem is the central claim in Section 4.2.2: Judge-MCTS achieves state-of-the-art performance on all three benchmarks. Table 2 says otherwise. On Multimodal RewardBench, R1-Reward scores 80.10 and the best M-Judger variant scores 65.67, a 14-point gap. The paper needs to say M-Judger wins on M-JudgeBench and VL-RewardBench, and loses on Multimodal RewardBench. This is not a nitpick; it is the abstract's main selling point. There is also a smaller internal inconsistency: overall accuracy for M-Judger-RL-Qwen8B is 62.42 in Table 2 and 62.93 in the supplementary Table 6. Same model, same protocol, different numbers.\n\nThe other soft spot is label quality. Section 2.2 relies on GPT-4.1 to extract final answers from CoT rollouts and exact-match filtering to assign labels. No human validation is reported. If the extraction is noisy, the benchmark's subtask accuracies — and the measured weaknesses — are only as good as that extraction. For a benchmark that is supposed to be a reusable instrument, I would want to see a sample audit, even a few hundred pairwise judgments. The GitHub link also appears to be a placeholder and the training artifacts are promised for later, so reproducibility is currently limited.\n\nWeighing the whole thing: the benchmark is worth building on, and the MCTS data recipe is plausible even if the SOTA story needs to be redrawn. I would send this to peer review, not desk reject. The reviewer should ask the authors to fix the SOTA claim, reconcile the two reported numbers for the same model, and provide human-validated label checks on the benchmark.","headline":"M-JudgeBench is a genuinely useful capability benchmark and the MCTS recipe shows real gains, but the SOTA claim in §4.2.2 is undercut by the paper's own Table 2.","tokens_in":749,"tokens_out":2014,"would_cite":false,"duration_ms":41673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that judging multimodal answers should be tested and trained as three distinct capabilities—comparing chains of thought, ignoring response length, and catching process errors—and that MCTS-generated preference pairs delive","keywords":["MLLM-as-a-judge","capability-oriented benchmark","chain-of-thought comparison","length bias","process error detection","Monte Carlo Tree Search","preference data generation","multimodal evaluation"],"falsifier":"Take a random sample of M-JudgeBench pairs and have independent human annotators label which response is better without seeing the automated labels; if human agreement with the benchmark labels falls well below the level needed for reliable model ranking, then both the reported weaknesses of current judges and the reported improvements of M-Judger rest on noisy ground truth.","tokens_in":22620,"feed_emoji":"⚖️","tokens_out":4236,"duration_ms":50499,"temperature":0.7,"pith_summary":"This paper tries to establish two linked claims. First, the right way to test an MLLM-as-a-judge is by the core judgment capabilities it needs—comparing two chains of thought, resisting response length, and spotting errors inside a reasoning process—rather than by task type; the authors build M-JudgeBench around these three capabilities and show that current judge models fail them, including strong proprietary ones. Second, training a judge on a small amount of MCTS-generated pairwise reasoning trajectories that vary in correctness and length substantially improves judge reliability, and the resulting M-Judger models outperform both open-source judge baselines and the original base models on M-JudgeBench and two existing judge benchmarks. If true, the field gains a diagnostic tool for finding why a judge is unreliable and a cheap data recipe for fixing it.","feed_headline":"13k MCTS pairs lift judge accuracy on all three benchmarks","feed_subtitle":"A capability-based benchmark exposes length bias and process blindness; a small, structured data recipe fixes them.","key_machinery":"M-JudgeBench is a benchmark of 3,712 curated pairwise instances organized into ten subtasks under three tasks: pairwise chain-of-thought comparison, length-bias avoidance, and process-error detection. Judge-MCTS is the data-generation framework that pairs with it: it runs Monte Carlo Tree Search over reasoning steps to produce four response classes—short-correct, short-error, long-correct, long-error—and then forms chosen–rejected pairs that differ in both correctness and length. The machinery works because the resulting supervision separates the dimensions a judge must attend to (correctness, process quality) from the dimensions it must ignore (length, style, wording).","core_discovery":"A reliable judge requires two independently testable skills: deciding result correctness across reasoning styles and lengths, and detecting process errors even when the final answer is correct. The paper's central discovery is that most existing judge models, including proprietary ones, are poor at these skills—especially at resisting length bias and detecting subtle process flaws—and that structured Monte Carlo Tree Search rollouts produce exactly the contrastive data needed to train those skills. Adding about 13k such samples at either the supervised fine-tuning stage or the reinforcement-learning stage improves judge accuracy on new and existing benchmarks without degrading performance el","pith_inferences":["The same four-class pairing idea (short/long × correct/error) would likely transfer to text-only judge training, where process-error trajectories could be generated by the same tree-search mechanism without multimodal perception.","Because the reported gains come from only 13k extra samples, the bottleneck in judge training may be the contrastive structure of the data rather than its volume; scaling MCTS rollouts further or adding process-error noise directly into the tree could yield additional gains.","The benchmark's label-construction pipeline could be made stronger by replacing exact-match answer filtering with symbolic or verifier-based checking, which would make the reported diagnosis of judge weaknesses independent of a single extraction model.","The very low accuracy on length-bias subtasks suggests that judge models are poorly calibrated about confidence, so reporting calibration alongside pairwise accuracy on M-JudgeBench would sharpen the diagnosis."],"forward_implications":["Judge evaluation should be organized around judgment capabilities rather than task categories; a benchmark built this way reveals failure modes that task-type benchmarks miss.","Length bias and process-error blindness are widespread, even in the strongest proprietary judge systems, so they are not solved by scale alone.","Adding a small amount of MCTS-generated reasoning pairs improves judge models across multiple base architectures, in both supervised and reinforcement-learning training settings.","The MCTS data recipe is compatible with existing judge benchmarks: it maintains or improves accuracy on the older benchmarks while lifting performance on M-JudgeBench."],"fun_headline_variants":["MCTS data fixes judge length bias and process blindness","Capability benchmark reveals judge flaws; MCTS data repairs them","13k MCTS samples sharpen multimodal judges on all benchmarks","Judge-MCTS: small data recipe beats length bias and process errors","New benchmark and MCTS training unmask judge blind spots"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark labels are trustworthy: final answers are extracted from chains of thought and checked against ground truth by an LLM with exact-match filtering, and no human validation of the extracted labels is reported.","fun_headline_variants_meta":{"raw":{"variants":["MCTS data fixes judge length bias and process blindness","Capability benchmark reveals judge flaws; MCTS data repairs them","13k MCTS samples sharpen multimodal judges on all benchmarks","Judge-MCTS: small data recipe beats length bias and process errors","New benchmark and MCTS training unmask judge blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2239,"prompt_tokens":776,"completion_tokens":1463,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":520,"tokens_out":1463,"duration_ms":11826,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:51:13.737784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of M-JudgeBench pairs and have independent human annotators label which response is better without seeing the automated labels; if human agreement with the benchmark labels falls well below the level needed for reliable model ranking, then both the reported weaknesses of current judges and the reported improvements of M-Judger rest on noisy ground truth.","supporting_citations":[],"review_version":1}