{"id":"45170083-ae36-4998-bcc9-cef5f7289f2d","arxiv_id":"2506.23563","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MMReason is an open-ended multimodal reasoning benchmark that filters out guessable and memorized questions and scores model answers both by final answer and by intermediate steps.","lead":"MMReason is a new benchmark that tests multimodal AI models on 1,384 open-ended, image-dependent questions requiring multi-step reasoning. Early results put the strongest model, GPT-4o, at 25.7% final-answer accuracy, suggesting current models are far from reliable visual reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decisive weakness is the unvalidated GPT-4o step grader: all intermediate-step scores in Table 3 rest on a judge with no human agreement study, no disclosed annotation set size, and no distribution or error analysis of the ternary labels.","rationale":"The filtering evidence is the paper's strongest independent support: after the text-only voting filter, GPT-4o's text-only accuracy falls from 13.4% to 0.78% (Table 5), which is a real, reproducible signal that removed items were text-answerable or memorized. The final-answer result (25.7% for GPT-4o) is therefore a credible empirical measurement conditional on the filtering procedure. I do not think the paper is fatally flawed; the missing validation is a condition, not a refutation. I therefore retain the reader's CONDITIONAL verdict rather than moving to REJECT or UNVERDICTED. For the claim to stand, the authors need to supply the human agreement study and judge-variance analysis described above; without it, the \"precise and comprehensive\" wording applies only to the curated question set, not to the intermediate-step scores. My agreement with the reader is complete on the core weakness, and I would additionally flag the segmentation sensitivity of Eq. 1 and the 0.5 reward for unverifiable steps as specific mechanisms by which judge bias could enter.","tokens_in":15592,"tokens_out":5252,"duration_ms":51862,"concrete_test":"Draw a random sample of 100 MMReason questions; run all 13 models to obtain 1,300 responses. Have two human annotators independently segment each response and score every step with the same ternary rubric plus the reference solutions. Report Cohen's kappa between GPT-4o and each human, and recompute the 13-model intermediate ranking using human scores. If kappa < 0.6, or if the top-5 ranking by S_inter changes, or if the mean |ΔS_inter| exceeds 5 points, the Table 3 intermediate column is not reliable. As a secondary check, re-score the same 1,300 responses with GPT-4o at temperature 0 twice and compare S_inter to quantify judge variance, and re-run with a different step-splitting prompt to test segmentation sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated novelty is \"a reference-based ternary scoring mechanism to reliably assess intermediate reasoning steps\" (Section 4.3). In practice, the mechanism is GPT-4o segmenting each response into N steps and assigning 1/0.5/0 (correct/unverifiable/incorrect), then averaging via Eq. 1. The reader's concern is correct: the paper provides no human agreement study, no second judge, no reported size of the annotated subset, and no breakdown of how often each label is used. The averaged score in Eq. 1 is also sensitive to how GPT-4o chooses to segment a response; two valid segmentations of the same solution can yield different S_inter, so the score is not defined independently of the judge's segmentation. The \"unverifiable\" category is scored 0.5, which can reward a model for writing an uncheckable step rather than a wrong one; the paper gives no evidence that this neutral category is applied consistently. Figure 3 itself shows a questionable case: the wrong response, whose Step 4 performs an incorrect substitution (using 115,200 instead of 58,500) and yields a wrong final answer, receives intermediate score 0.7, suggesting the grader may be lenient in ways that are not analyzed. Because Table 3's intermediate column is a headline result and the word \"precisely\" in the central claim depends on it, the missing validation is genuinely load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MMReason, a multimodal multi-step reasoning benchmark consisting of 1,384 open-ended questions across six disciplines, with questions reformulated from existing benchmarks and newly collected from the web. The construction pipeline converts multiple-choice questions into open-ended formats, filters out memorized or visually irrelevant items using a multi-model text-only voting procedure (K=4 models, T=2 rounds), and annotates questions with step-by-step reference solutions. The evaluation uses GPT-4o both to extract final answers and to segment responses into steps, assigning ternary labels (correct/unverifiable/incorrect) that are averaged into an intermediate-step score. The main empirical claim is that state-of-the-art MLLMs perform poorly on this benchmark, with the best model (GPT-4o) achieving only 25.7% final-answer accuracy.","tokens_in":15861,"tokens_out":5614,"duration_ms":56281,"significance":"If the construction and evaluation methods are valid, MMReason would be a valuable benchmark: the filtering evidence in Table 5 is the strongest part, showing that text-only accuracy drops to 0.65-0.78% while visual accuracy remains 11.7-25.7% and the multimodal relevance rate rises above 94%, which concretely demonstrates that the retained questions require visual input. The benchmark also covers a broader disciplinary and difficulty range than many existing multimodal reasoning benchmarks. However, the novelty claims regarding intermediate-step evaluation are not yet supported because the GPT-4o-based ternary scoring mechanism lacks validation; the final-answer accuracy results are plausible and interesting, but the paper's stated advantage over final-answer-only benchmarks depends on the unvalidated grader.","major_comments":[{"comment":"The paper does not specify how many of the 1,384 benchmark questions have reference step-by-step solutions. Section 4.1 states that 'a subset of questions from our newly collected data' is manually annotated, which implies that not all questions in MMReason have reference solutions. If the intermediate-step scores in Table 3 are computed only on that subset, the paper must report the subset size and confirm that all models are scored on exactly the same questions; otherwise the intermediate-score column is not comparable with the final-answer column and the model rankings may be inconsistent across metrics.","section":"Section 4.1 and 4.3"},{"comment":"The reference-based ternary scoring mechanism rests entirely on GPT-4o for both segmenting responses into steps and assigning correct/unverifiable/incorrect labels. The paper provides no human agreement study, no second judge, no disclosed size of any validation subset, and no distribution or error analysis of the ternary labels. This is load-bearing because the word 'precisely' in the central claim and the entire intermediate-step column of Table 3 depend on the reliability of this grader. The concern is compounded by the fact that GPT-4o is also the top-scoring model on the benchmark, so a grader bias toward GPT-4o's response style could inflate its intermediate score; a second judge or a human-annotated subset would be needed to rule this out.","section":"Section 4.3"},{"comment":"The intermediate score S_inter = (1/N) sum Score(s_n) is not independent of how GPT-4o chooses to segment a response. Two equally valid segmentations of the same solution can yield different values of N and different step-level labels, so the metric is not well-defined unless segmentation stability is reported. In addition, the 'unverifiable' category is always scored 0.5, which can reward a model for writing an uncheckable step rather than a correct one; the paper gives no evidence that this neutral category is applied consistently across responses or across models.","section":"Section 4.3, Eq. (1)"},{"comment":"The qualitative example in Figure 3 raises concerns about grader leniency: the wrong response, whose Step 4 performs an incorrect substitution (using 115,200 instead of 58,500) and which yields a wrong final answer, nevertheless receives an intermediate score of 0.7. The paper does not discuss how such partial-credit cases are calibrated, how often they occur, or whether the threshold for a 'passing' intermediate score is meaningful. An error analysis of the grader itself—not just of the evaluated models—is necessary to interpret the intermediate-step scores in Table 3.","section":"Figure 3"}],"minor_comments":[{"comment":"Section 1 reports GPT-4o's intermediate-step score as 'xx.x%', but Table 3 lists 42.1; this unresolved placeholder must be corrected.","section":"Section 1 and Table 3"},{"comment":"There are typos in model names: 'Calude-3.7V Soonet' should be 'Claude-3.7V Sonnet', and Table 3's 'LLaV A-OneVision' and 'LLaV A-CoT' should be 'LLaVA-OneVision' and 'LLaVA-CoT'.","section":"Section 5.2.1 and Table 3"},{"comment":"The error analysis in Figure 4 is based on only 50 randomly selected incorrect responses from GPT-4o; the paper should state the sampling procedure and the confidence interval or stability of the reported distribution, especially because 'Reasoning Error' and 'Perceptual Error' are close in proportion.","section":"Section 5.3 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's filtering experiment is valuable and the final-answer results are interesting, but the unvalidated GPT-4o grader is a serious obstacle to the paper's central claim about precise intermediate-step evaluation. The 'xx.x%' placeholder suggests the manuscript was submitted before final proofreading. I would want to see a human agreement study on a few hundred steps, a clear statement of annotation coverage across the 1,384 questions, and an analysis of segmentation stability before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: MMReason is a real attempt to fix three known problems in MLLM reasoning benchmarks, and the filtering part mostly works. The multi-model text-only voting filter is a genuine new ingredient, and the before/after numbers in Table 5 are informative: text-only accuracy drops to under 1% while visual accuracy stays around 11-25%, so the retained 1,384 questions look hard to memorize and do seem to require the image. The final-answer result, GPT-4o at 25.7%, is credible as a measurement conditional on that filter. That part deserves credit.\n\nThe soft spot is exactly where the stress-test note lands: the intermediate-step scoring is the paper's stated novelty and it is unvalidated. The entire intermediate column in Table 3 comes from GPT-4o segmenting each response and labeling each step correct/unverifiable/incorrect, scored 1/0.5/0 and averaged. There is no human agreement study, no reported size of the annotated subset, no distribution of the three labels, no second judge, and no analysis of how often 'unverifiable' is used. This matters for two concrete reasons. First, the averaged score is not defined independently of the judge's segmentation: the same solution can be cut into different step boundaries and yield different intermediate scores. Second, the 'unverifiable' category gets 0.5, so a model that writes an uncheckable step is rewarded relative to a wrong step; the paper gives no evidence that the label is applied consistently. Figure 3 is a warning sign in passing: a wrong final answer with an incorrect substitution still gets an intermediate score of 0.7, suggesting leniency the paper does not analyze. Because the word 'precisely' in the title and abstract leans on this, the missing validation is load-bearing, not cosmetic.\n\nMinor items: the abstract/introduction still has an 'xx.x%' placeholder, code and data are not yet released, and the filtering choices (K, T, removal threshold) are not sensitivity-analyzed. Those are fixable paper-work issues.\n\nWho this is for: people building, using, or critiquing multimodal reasoning benchmarks. The filtering analysis is worth reading even if you ignore the step scores. The paper deserves a serious referee: the construction is coherent, the filtering evidence is real, and a revision with a small human agreement study, a label distribution, and a robustness check on the averaging could make the step-score claim stand. I would engage with it in review, but I would not yet rely on any intermediate-step number.","headline":"The filtering story is solid and the benchmark deserves refereeing, but the GPT-4o step-scoring is unvalidated and the intermediate-step numbers are not yet trustworthy.","tokens_in":16474,"tokens_out":3184,"would_cite":false,"duration_ms":30323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMReason, a new open-ended multimodal benchmark, claims existing MLLMs score far lower on multi-step reasoning than MCQ tests suggest, with the best model, GPT-4o, at 25.7%.","keywords":["multimodal large language models","multi-step reasoning","reasoning benchmark","open-ended questions","guessability","memorization","intermediate step evaluation","ternary scoring"],"falsifier":"Run a human agreement study on a random sample of MMReason responses: have several human raters assign the same correct/unverifiable/incorrect labels to each step using the same reference solutions, and compare the averaged intermediate scores with GPT-4o's labels. If human raters frequently disagree with the model judge, or if the ranking of models by step score reverses under human grading, the benchmark's central claim that it precisely and reliably evaluates intermediate reasoning is not supported.","tokens_in":15374,"feed_emoji":"🧠","tokens_out":5584,"duration_ms":55026,"temperature":0.7,"pith_summary":"MMReason is a benchmark built to measure whether multimodal large language models can actually reason through multi-step problems, rather than guess or memorize. The paper argues that current multimodal benchmarks, dominated by multiple-choice questions, let models arrive at correct answers without valid reasoning, either by guessing among options or by recalling leaked training data. MMReason removes answer choices, filters out questions that any of four strong models can answer from text alone, and scores each intermediate reasoning step against a human-annotated reference solution. On the resulting 1,384 questions, the best evaluated model, GPT-4o, reaches only 25.7% final-answer accuracy, suggesting that state-of-the-art multimodal reasoning is much weaker than MCQ leaderboards imply. The paper's case matters because if true, progress toward AGI needs to be measured by step-level reasoning on open-ended multimodal problems, not by option-picking accuracy.","feed_headline":"Best multimodal model scores just 25.7% on new reasoning test","feed_subtitle":"MMReason strips away answer choices and checks every reasoning step, exposing how far MLLMs are from AGI-level reasoning.","key_machinery":"The load-bearing mechanism is the combination of two filters and one grader. First, multiple-choice questions with unique answers are rewritten in open-ended form to block guessing. Second, a multi-model voting procedure runs four strong MLLMs on text-only versions of each question for two rounds, deleting any question answered correctly without the image; this targets memorization and enforces visual relevance. Third, GPT-4o decomposes each model response into steps and scores each step as correct (1), unverifiable (0.5), or incorrect (0), using the annotated reference solution, then averages the step scores. The 'unverifiable' category is what makes the grader usable when a response follows a valid alternative solution that the reference does not cover.","core_discovery":"The central discovery is that a carefully filtered set of 1,384 open-ended multimodal questions, spanning six disciplines and several difficulty tiers, is sufficient to expose a large gap between appearance and reality in MLLM reasoning. After converting unique-answer multiple-choice items to open-ended format and removing text-solvable or memorized instances via iterative multi-model voting, the authors find that no model answers more than a quarter of questions correctly: GPT-4o leads with 25.7%, followed by Claude-3.7V Sonnet at 25.1% and Gemini-1.5 Pro at 24.9%. They also report intermediate-step scores from a reference-based ternary grader, with GPT-4o scoring 42.1% on steps, and an error analysis attributing most of GPT-4o's failures to reasoning and perception errors. The paper reads these results as evidence that current MLLMs have not yet acquired robust long-chain multimodal reasoning.","pith_inferences":["Editorial inference: because the filtering step removes every question that any of the four chosen models can answer from text, the benchmark's difficulty is partially a moving target; as those models improve, the retained set may shift and require periodic re-filtering.","Editorial inference: the three-level scoring could be extended to a 'partially correct' scale or to multiple reference solutions, which would make the intermediate-step scores more informative for training and error analysis.","Editorial inference: a direct testable extension is to use MMReason's step scores as a training signal or as a diagnostic to distinguish perception failures from reasoning failures, which the paper's small error analysis suggests is feasible."],"forward_implications":["If MMReason measures what it claims, then published MCQ-based accuracy numbers for MLLMs overstate their genuine multi-step reasoning ability.","The 25.7% ceiling for GPT-4o provides a concrete reference point for future model development on open-ended multimodal reasoning.","The discipline-wise breakdown shows engineering questions are the hardest across nearly all models, while math and health questions are relatively easier.","The filtering results, with text-only accuracy dropping from about 6-13% before filtering to under 1% after filtering, indicate the retained benchmark is genuinely vision-dependent.","The intermediate-step scores being higher than final-answer accuracy for closed-source models suggests these models often reason plausibly but fail at the final step or answer extraction."],"supporting_citations":[{"why":"GPT-4o system card; GPT-4o is the model used for step decomposition, ternary scoring, final-answer extraction, and as a filtering and evaluation model.","marker":"[14]"},{"why":"MathVista; its MCQ format and the authors' choice-removal experiment supply the evidence that guessability inflates scores.","marker":"[23]"},{"why":"MMMU; its MCQ-heavy design (94% multiple choice) is the primary motivation for reformulating questions into open-ended form.","marker":"[45]"},{"why":"MMMU-Pro; a source of reformulated questions with unique answers used in MMReason.","marker":"[46]"},{"why":"M3CoT; a source of multi-domain multi-step multimodal chain-of-thought questions reused in MMReason.","marker":"[5]"},{"why":"MMStar; a source of questions and a prior analysis of data leakage that the voting filter extends.","marker":"[3]"}],"fun_headline_variants":["New benchmark exposes MLLM reasoning gap: best scores 25.7%","MMReason: open-ended multi-step test stumps leading AI models","Top multimodal models fail new reasoning benchmark at 25%","Multimodal LLMs score under 26% on new open-ended reasoning test","Benchmark reveals MLLMs far from AGI: only a quarter correct"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's intermediate-step scores all depend on GPT-4o being a trustworthy step extractor and grader; if GPT-4o's step labels are biased, the claimed advantage over final-answer-only evaluation collapses.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes MLLM reasoning gap: best scores 25.7%","MMReason: open-ended multi-step test stumps leading AI models","Top multimodal models fail new reasoning benchmark at 25%","Multimodal LLMs score under 26% on new open-ended reasoning test","Benchmark reveals MLLMs far from AGI: only a quarter correct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2838,"prompt_tokens":1000,"completion_tokens":1838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":1741}},"tokens_in":616,"tokens_out":1838,"duration_ms":12497,"temperature":1.0,"reasoning_tokens":1741,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:37:35.621301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human agreement study on a random sample of MMReason responses: have several human raters assign the same correct/unverifiable/incorrect labels to each step using the same reference solutions, and compare the averaged intermediate scores with GPT-4o's labels. If human raters frequently disagree with the model judge, or if the ranking of models by step score reverses under human grading, the benchmark's central claim that it precisely and reliably evaluates intermediate reasoning is not supported.","supporting_citations":[{"cited_title":"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi","cited_arxiv_id":null,"evidence_quote":"MMMU; its MCQ-heavy design (94% multiple choice) is the primary motivation for reformulating questions into open-ended form."},{"cited_title":"M 3cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought","cited_arxiv_id":null,"evidence_quote":"M3CoT; a source of multi-domain multi-step multimodal chain-of-thought questions reused in MMReason."}],"review_version":1}