{"id":"58091a82-6357-4ae5-aac2-071ed6ba5bbe","arxiv_id":"2505.24182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new multi-image benchmark and graph-based scoring method show that MLLMs produce weak, poorly-grounded chains of thought on visual physics tasks, and that RL post-training can degrade spatial reasoning.","lead":"This paper introduces MVPBench, a new benchmark that tests whether AI vision-language models can reason step-by-step about physics using multiple images. It reports that even top models often fail, and that reinforcement-learning fine-tuning sometimes makes spatial reasoning worse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline findings depend on unvalidated GPT-4o step judgments: SAS, KSC, SRS, RVR, and PVR/PCS are scored by the very model class being evaluated, with no human agreement or calibration, so the 'poor visual reasoning' and 'RL post-training harms' conclusions could be evaluator artifacts.","rationale":"The reader's weakest assumption targeted the completeness of the reference graph and the reliability of GPT-4o step parsing. My independent read agrees that this is a real vulnerability, but I would put the weight one level up: the entire evaluation pipeline—SAS, KSC, SRS, RVR, and the PVR/PCS graph matching—is mediated by GPT-4o judgments, and there is no validation of that judge against human experts. The paper does release the dataset, includes a human diversity baseline, and makes a plausible case that current MLLMs struggle on these tasks; those are real strengths. However, the headline claims about 'weak image-text alignment' and 'RL post-training often harms spatial reasoning' require that the evaluation instrument be reliable and impartial. Since GPT-4o is both judge and evaluated model, and since post-training changes generation style, the observed degradation could in principle be an artifact of the judge's stylistic preferences. I also noticed a concrete internal inconsistency: Section H.1 says InternVL3-78B-Instruct underwent MPO post-training while InternVL3-78B remained unmodified, but Figure 5's caption labels the opposite. This discrepancy must be resolved before the InternVL-based RL comparison can be interpreted. These concerns do not overturn the reader's CONDITIONAL verdict; they reinforce it, so I recommend no change to the verdict. A human-agreement study on a 100-item sample, with the reference-graph completeness question included, is the single check that would settle whether the concern lands.","tokens_in":36323,"tokens_out":6844,"duration_ms":84760,"concrete_test":"Select a stratified random sample of 100 items (25 per subset) and the corresponding outputs from the base/post-trained pairs in Figures 5-6. Have three human physics experts independently score every step with the exact SAS, KSC, SRS, and RVR prompts, and also judge whether each model path edge is a valid physical inference not necessarily in the reference graph. Compute Krippendorff's alpha between GPT-4o and the human experts, then recompute Table 2 rankings and the base-vs-post-trained deltas using the human scores only. If alpha < 0.6, or if the sign of the RL-post-training delta flips on the sampled subset, the central claims are not established. In the same pass, verify the InternVL3-78B vs InternVL3-78B-Instruct post-training designation against model cards, since Section H.1 and Figure 5 contradict each other.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core empirical conclusions are produced entirely by a GPT-4o-based evaluation pipeline. Section 4.1 defines SAS and KSC as GPT-4o binary judgments of each model step against human-annotated key steps; Section 4.3 defines SRS and RVR using GPT-4o relevance and reflection judgments; Section 4.2's PVR/PCS rely on parsing model paths into a human-annotated reference graph that is assumed to be a complete inventory of valid reasoning chains. Limitation (ii) admits annotation subjectivity but does not test the completeness assumption, and no inter-annotator agreement or human re-scoring study is reported anywhere. Because GPT-4o is itself one of the evaluated systems, any systematic preference of the judge for certain step granularity, phrasing, or reasoning style will differentially affect models. This is especially consequential for the RL post-training claim: the post-trained checkpoints in Figures 5-6 have different generation styles from their base models, so a style-sensitive judge could manufacture the reported 'post-training harms spatial reasoning' result. The reference graph completeness assumption is also doing real work: KSC, PVR, and PCS penalize any valid edge absent from the human graph, so diversity and completeness scores may understate model ability. The section E limitations acknowledge dependence on GPT-4o outputs but do not validate that the judgments are correct, which leaves the central claims unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MVPBench, a benchmark of 1,211 carefully curated visual-physics examples with multi-image inputs and multi-path chain-of-thought (CoT) annotations across four domains: physics experiments, physics problems, spatial relations, and dynamic prediction. The authors propose a graph-based evaluation suite with correctness metrics (SAS, KSC, CRS), diversity metrics (PVR, PCS, CMS), and efficiency metrics (SRS, RVR), and report results on open- and closed-source MLLMs under single- and multi-image settings. The headline findings are that current MLLMs perform poorly on visually grounded physical reasoning, multi-image input helps most models, and RL-based post-training often degrades spatial reasoning and visual-physics scores.","tokens_in":36618,"tokens_out":3852,"duration_ms":50527,"significance":"If the evaluation methodology is validated, MVPBench fills a real gap: it combines real-world visual physics, multi-image inputs, and multi-path CoT annotations, and the graph-based metrics go beyond final-answer accuracy. The authors release data and code, report human diversity performance (Appendix B.1), and use human final-answer labels for the correctness component, which is a genuine strength. However, the step-level metrics that carry most of the paper's conclusions are produced by GPT-4o as an unvalidated judge, and no text-only baseline is provided, so the central claims are currently conditional on the reliability of that judge and on the completeness of the human-annotated reference graphs.","major_comments":[{"comment":"SAS, KSC, SRS, and RVR are all computed from GPT-4o judgments on step decomposition, correctness, relevance, and reflection validity, but no inter-annotator agreement or human rescoring is reported for these judgments. Limitation (iii) in Appendix E acknowledges dependence on proprietary models but does not validate that the judgments are correct. Since GPT-4o is also one of the evaluated systems, a systematic preference for certain step granularity, phrasing, or reasoning style could differentially affect models, especially post-trained checkpoints. Please add a validation study: human rescoring of a sample of model outputs across multiple models, reporting Cohen's kappa or similar, and a sensitivity analysis with an alternative judge.","section":"Section 4.1, Section 4.3, Appendix E"},{"comment":"The abstract claims that models exhibit 'weak image-text alignment' and that MVPBench minimizes text-prior shortcuts, but no text-only baseline is reported. The experiments only compare single-image versus multi-image inputs; neither condition removes the images. Without a text-only or image-blind control, the claim that failures are due to visual reasoning rather than language priors is unsupported. Please add a text-only baseline (e.g., the same questions without images, or with captions only) and report the accuracy gap.","section":"Section 5 and Table 2"},{"comment":"PVR and PCS assume that the human-annotated reference graph is a complete inventory of valid reasoning paths: any model edge not matching the graph is penalized, and PCS uses the longest matched sub-path. If annotators missed legitimate alternative chains, or if step parsing by GPT-4o fails, the diversity scores understate model ability. The average of 2.67 reference paths per sample is reported, but no completeness validation is provided. Please have human experts judge a sample of unmatched model paths as valid or invalid, and report the proportion of valid paths currently counted as errors.","section":"Section 4.2"},{"comment":"The adjusted path coverage score is defined as PCS * exp(-alpha * (Np/Ngt - 1)), where alpha controls the over-generation penalty, but the value of alpha is never reported anywhere in the manuscript. Since Table 2 and Table 3 report PCS and CMS numbers that depend on this adjustment, the results are not reproducible. Please report the alpha value used for all experiments, and include a sensitivity analysis over reasonable alpha ranges.","section":"Section 4.2"},{"comment":"The claim that RL post-training harms spatial reasoning is based on comparisons where base and post-trained models are evaluated with different generation settings. For example, Table 8 shows InternVL2.5-78B uses max_new_tokens=2048 while InternVL2.5-78B-MPO uses 1024, and do_sample differs across models. Since SAS, KSC, SRS, and diversity metrics depend on response length and sampling behavior, the reported degradations may be artifacts of the evaluation configuration rather than of post-training. Please control generation hyperparameters across matched model pairs and report bootstrap confidence intervals or paired significance tests for the post-training comparisons.","section":"Figures 5-6 and Table 8"}],"minor_comments":[{"comment":"The text says diversity is assessed via PVR and RCS, but the metric introduced in Section 4.2 is PCS (Path Coverage Score); please correct the abbreviation.","section":"Section 5, first paragraph"},{"comment":"The caption states RVR is set to 100 for models lacking reflection ability, but the table reports values such as 99.52 for OpenAI o3 and 99.32 for InternVL2.5-78B-MPO; please clarify how RVR is computed for models with no detected reflection steps.","section":"Table 2 caption"},{"comment":"There is a typo in 'Chinses Gaokao physics questions'; this should read 'Chinese Gaokao'.","section":"Appendix C.2"},{"comment":"For API-based models such as GPT-4o, o3, and Claude, the table lists torch.dtype, which is not a meaningful generation setting for those models; please remove or mark those entries as not applicable.","section":"Appendix H.2, Table 8"},{"comment":"The sentence 'SRS, similar to SCS, is defined as...' references a metric 'SCS' that is not defined elsewhere; it should likely refer to SAS or should be rephrased.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a useful contribution and the release of data/code is commendable, but the evaluation methodology needs validation before the headline claims can be accepted. The self-judging issue is the main technical risk, and the missing text-only baseline undermines the 'vision-centric' narrative. I would ask for the validation experiments and controlled post-training comparisons in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MVPBench is a real contribution as a benchmark resource, but the paper's headline claims—'poor visual reasoning' and 'RL post-training harms spatial reasoning'—rest on an unvalidated GPT-4o judge and on reference graphs assumed to be complete. Both need to be addressed before the strong conclusions can be trusted.\n\nWhat's genuinely new: the combination of real-world multi-image physics inputs, multi-path CoT annotations, and graph-based diversity metrics (PVR/PCS/CMS) is not in PhysBench, MathVerse, or MME-CoT. The benchmark construction looks careful: 1,211 examples, 4,701 images, multiple annotated reference chains per example, double annotation for at least some subsets, and a human diversity baseline. The single-image vs. multi-image manipulation is a clean way to isolate the effect of temporal visual context, and that result (multi-image helps) is the most solid finding in the paper. They also ship data and code, which is good practice.\n\nThe soft spots are substantial. Every step-level metric—SAS, KSC, SRS, RVR, PVR, PCS—passes through GPT-4o judgments. The prompts are detailed, but there is no human agreement study on the step judgments, no calibration, and no error bars on any reported number. GPT-4o is itself one of the evaluated models, so a systematic preference for certain step granularity or phrasing could differentially affect models. This is especially relevant for the post-training comparisons: the post-trained checkpoints (InternVL2.5-MPO, MM-Eureka, R1-VL) have different generation styles from their base models, and a style-sensitive judge could manufacture the 'post-training harms' conclusion. The results in Figure 5 and Table 3 are also mixed across subsets, so the abstract's blanket statement overreaches what the data show.\n\nThe reference graph completeness assumption is doing real work. PVR and PCS penalize any model edge that does not appear in the human-annotated graph. If the annotators missed legitimate alternative chains—and the paper gives no completeness analysis, only a stated assumption—the diversity scores understate model ability. The limitation section (E) acknowledges dependence on GPT-4o outputs and annotation subjectivity, but it never validates that the judge's step parses are correct or that the graphs cover all valid paths.\n\nMissing as well: a text-only baseline. Without one, the 'vision-centric' claim is not directly tested. The single vs multi-image comparison helps, but it doesn't show that the models couldn't answer from language priors alone.\n\nBottom line: the benchmark and metrics are worth having, and the paper deserves a serious referee. But as submitted, the headline empirical conclusions are not established. I'd send it to review with a clear request for judge validation, a text-only baseline, and error bars or significance tests on the post-training comparisons.","headline":"Useful benchmark, but the headline findings rest on an unvalidated GPT-4o judge and a completeness assumption that need to be fixed before the strong claims stand.","tokens_in":37174,"tokens_out":2663,"would_cite":true,"duration_ms":31931,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Leading multimodal models cannot reliably execute multi-step, visually grounded physical reasoning, and common RL post-training often makes spatial reasoning worse.","keywords":["visual physical reasoning","chain-of-thought","multimodal large language models","graph-based evaluation","multi-path reasoning","reinforcement learning post-training","spatial reasoning","benchmark"],"falsifier":"Fine-tune a base model with reinforcement learning whose reward explicitly scores step-level grounding against the MVPBench reference graphs; if that model outscores its untrained base on spatial-reasoning metrics, the paper's conclusion that RL post-training harms spatial reasoning fails for that reward family.","tokens_in":36140,"feed_emoji":"🧪","tokens_out":8073,"duration_ms":86478,"temperature":0.7,"pith_summary":"The paper sets out to test whether multimodal large language models can genuinely reason about physical scenes image by image, rather than merely describe them. It introduces MVPBench, a benchmark of 1,211 tasks that present multiple related images and require a final answer plus a step-by-step reasoning chain anchored to the visual evidence, with several human-annotated valid chains per task. On this benchmark, even the strongest tested models show low reasoning accuracy and weak image-text alignment, and reasoning errors dominate over perception errors in most subsets. The graph-based chain-of-thought metrics reveal that models often fail to follow valid logical paths, and RL-based post-training, commonly believed to improve visual reasoning, frequently lowers spatial-reasoning scores. If the paper is right, current alignment practices do not produce physically grounded reasoning, and reward design needs to become vision-centric.","feed_headline":"Seeing is not reasoning: MLLMs fail visual physics","feed_subtitle":"A 1,211-task benchmark shows top models miss grounded reasoning chains, and RL alignment can lower spatial scores.","key_machinery":"The load-bearing evaluation machinery is a graph-based chain-of-thought consistency metric. For each item, human-annotated key steps are nodes of a directed graph whose edges encode valid logical transitions, and multiple valid paths are recorded for each task. A model's generated reasoning is parsed into steps, embedded as edges in the same graph, and scored by Path Validity Rate (the share of model edges present in the reference graph), Path Coverage Score (the normalized longest matched sub-path), and their harmonic mean, the CoT Match Score, with adjusted versions that penalize over-generation. Quality is measured by Step Accuracy Score, Key Step Coverage, and a weighted CoT Reasoning Score, while efficiency is measured by Step Relevance Score and Reflection Validity Rate. The graph formalism is what lets the benchmark separate visually grounded multi-step reasoning from fluent but ungrounded narration.","core_discovery":"The central claim is that seeing is not reasoning: current multimodal models can perceive objects in images but cannot carry out coherent multi-step visual physical reasoning. Concretely, MVPBench asks models to answer physics questions from interleaved image sequences under chain-of-thought prompting, and the strongest closed model reaches a CoT Reasoning Score of about 75 in the multi-image setting while most open models score lower, with the Spatial Relations subset the hardest. The paper's error analysis attributes most failures to reasoning steps rather than visual perception, except in spatial-relation tasks where perception errors are far more common. The paper further reports that multi-image input helps almost all models, by up to 21 percentage points, and that RL-based post-training, tested with several open models and reward families, can reduce visual-physics and spatial-reasoning scores relative to their base counterparts.","pith_inferences":["Because the reference graph is treated as the complete set of valid paths, the diversity metrics may penalize legitimate chains that annotators did not list; crowd-sourcing many more valid paths per item and re-measuring would test whether the headline understates model ability.","The paper's 'RL harms spatial reasoning' result is conditional on the tested reward designs; it predicts that RL with explicit step-level visual-grounding rewards would reverse the drop, and MVPBench provides the measurement tool to check that.","The graph formalism transfers directly to other multi-step visual domains, such as medical image interpretation or driving scene understanding, where the same question of whether a model's text is anchored to what it sees arises.","The strong multi-image gains suggest a possible scaling relationship between the number of temporally spaced images and measured reasoning quality, which the paper does not vary systematically."],"forward_implications":["High scores on conventional visual question answering tell little about whether a model reasons from images; performance must be judged at the level of individual reasoning steps.","Multi-image input is not optional for physical reasoning: giving the full image sequence raised reasoning scores by up to 21 points, so single-frame evaluations understate both ability and failure.","RL-based post-training that optimizes final-answer or conversational fluency can degrade spatial reasoning, so alignment rewards should be redesigned to include step-level visual grounding.","Spatial relations, not physics knowledge, are the current bottleneck, with perception errors playing a much larger role there than in physics experiments and problems.","Current multimodal models should not be treated as trustworthy physical-world reasoners for embodied or science applications without step-level visual verification."],"supporting_citations":[{"why":"Supplies the dynamic-scene video samples and multi-path annotations that MVPBench adapts into its Dynamic Prediction subset.","marker":"[49]"},{"why":"Supplies the physics-problem examples and chain-of-thought style that MVPBench extends with visual grounding.","marker":"[44]"},{"why":"Motivates adaptive key-step extraction and warns against rigid ground-truth templates, which the graph-based metrics build on.","marker":"[25]"},{"why":"Provides the step-wise CoT scoring idea that MVPBench extends by weighting final-answer correctness.","marker":"[24]"},{"why":"Acts as both a strong evaluated model and the automated judge used for step parsing and scoring.","marker":"[6]"},{"why":"Provides the MPO-tuned model whose degraded spatial scores support the claim that post-training can harm reasoning.","marker":"[81]"},{"why":"Supplies the chain-of-thought prompting technique used throughout the zero-shot evaluation.","marker":"[88]"}],"fun_headline_variants":["MVPBench: MLLMs see but can't reason visually","RL alignment hurts visual physics in MLLMs","Perception isn't reasoning: MVPBench exposes gap","Multi-step visual reasoning fails in top MLLMs","Benchmark: MLLMs lack grounded visual CoT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human-annotated key-step graphs are a complete inventory of valid reasoning paths; if annotators missed legitimate alternative chains, or if the automated step parser fails, the diversity and quality scores understate model ability and the headline conclusion is weakened.","fun_headline_variants_meta":{"raw":{"variants":["MVPBench: MLLMs see but can't reason visually","RL alignment hurts visual physics in MLLMs","Perception isn't reasoning: MVPBench exposes gap","Multi-step visual reasoning fails in top MLLMs","Benchmark: MLLMs lack grounded visual CoT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1367,"prompt_tokens":987,"completion_tokens":380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":603,"tokens_out":380,"duration_ms":4194,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:30:37.869555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a base model with reinforcement learning whose reward explicitly scores step-level grounding against the MVPBench reference graphs; if that model outscores its untrained base on spatial-reasoning metrics, the paper's conclusion that RL post-training harms spatial reasoning fails for that reward family.","supporting_citations":[{"cited_title":"Lighthouse Laboratory","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique used throughout the zero-shot evaluation."}],"review_version":1}