{"id":"cb8ad0d0-6748-4eb1-9e88-4d1818e7d113","arxiv_id":"2506.22992","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"State-of-the-art multimodal language models perform at or near random chance on MARBLE, a new hard benchmark for spatial reasoning and planning.","lead":"A new benchmark called MARBLE tests AI models on two spatial puzzles: planning through Portal-style rooms and assembling a 3D cube from jigsaw pieces. The best current models score near zero, showing that multi-step spatial reasoning is still a major weakness for multimodal AI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0% CUBE accuracy is statistically indistinguishable from the 1e-5 random baseline, so it does not by itself show MLLM failure on M-Cube; the claim's strongest number is uninformative.","rationale":"The reader's verdict is CONDITIONAL and identifies the unverified ground truth (human annotations and the solution validator) as the weakest assumption. That concern is legitimate and worth addressing. However, I find a more immediate, non-speculative flaw in the central claim: the '0% accuracy on M-Cube' result is statistically uninformative because the random baseline is 1e-5. The paper's own numbers show that 0 successes in 200 trials is exactly what chance would produce, so the headline number cannot support the inference that models lack the relevant reasoning ability. The strength of the benchmark as evidence of MLLM limitations rests instead on M-Portal and CUBE-easy, where the random baselines are non-trivial and most models remain near or below chance. I therefore agree with the CONDITIONAL verdict, but for a slightly different primary reason: the paper should either report confidence intervals for the 0% CUBE results or reframe them as a floor effect and lean on the discrimnable subtasks. The proposed concrete test would settle whether the 0% figure contributes any evidence to the central claim. This is a statistical correctness risk, not a question of agreement with consensus, and it does not call the authors' integrity into question.","tokens_in":16107,"tokens_out":11617,"duration_ms":125337,"concrete_test":"Compute a 95% Clopper-Pearson confidence interval for each model's CUBE accuracy using the actual N (200 for closed models, 1000 for open models). If every interval contains the random baseline rate of 1e-5, the 0% results carry no information about model competence. As a second check, simulate 10,000 random-guess evaluations on the same 200-example subset; if random guesses produce 0 successes in most simulations, the paper should replace '0% accuracy' with 'indistinguishable from random' and base the M-Cube challenge claim on CUBE-easy alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim that MLLMs obtain '0% accuracy on M-Cube' (Abstract, Section 3.5, Table 3) is presented as demonstrating that complex multimodal reasoning remains unsolved. But Table 3 gives the random baseline for CUBE as 1e-5, and Appendix B states that closed-source models are evaluated on 200 representative examples, not the full 1,000. With 0 successes in 200 trials, the 95% Clopper-Pearson interval is [0%, 1.8%], which contains the random success rate of 0.001%. The experiment therefore has no power to distinguish a model with zero ability from one that guesses uniformly. Saying models 'completely fail' on CUBE is no more informative than saying a random guesser 'completely fails.' The meaningful evidence of difficulty on M-Cube comes from CUBE-easy (random baseline 3.1%, most models at or below 16%, GPT-o3 at 72%), not from the 0% on the full CUBE task. This does not invalidate the benchmark, but it removes one of the two headline pillars of the central claim, and the paper should not cite the 0% figure as evidence of a reasoning deficit.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents MARBLE, a benchmark aimed at evaluating multimodal spatial reasoning and planning in large multimodal language models. It consists of two tasks: M-Portal, derived from Portal 2 puzzles, with plan-correctness and fill-the-blanks subtasks, and M-Cube, a 3D jigsaw assembly task with a simplified CUBE-easy subtask. The authors evaluate 12 models, including open- and closed-weight MLLMs and text-only LLMs. They report near-random F1 on plan correctness, accuracy between 0% and 17.6% on fill-the-blanks, 0% accuracy on the full CUBE task, and between 0.3% and 72% on CUBE-easy. Additional experiments probe perception accuracy and a validator-assisted iterative refinement setup. The paper's central claim is that state-of-the-art models still lack the multi-step multimodal reasoning required by these tasks.","tokens_in":16364,"tokens_out":11553,"duration_ms":93622,"significance":"MARBLE addresses a genuine gap in multimodal reasoning evaluation by focusing on multi-step planning under spatial and physical constraints rather than single-step question answering. The benchmark design is thoughtful: the M-Cube-easy results provide a clear performance gradient (GPT-o3 at 72% versus the best other model at 16%), and the perception experiment isolates a measurable bottleneck. The data generation pipeline and the human-annotated Portal solutions are potentially reusable resources. However, the paper's headline statistical claims—'near-random performance on M-Portal' and '0% accuracy on M-Cube'—are not adequately supported as stated: the fill-the-blanks results are far above random for several models, and the 0% CUBE result lacks the statistical power to distinguish any non-random ability from chance. With the revisions described below, the benchmark would be a useful contribution.","major_comments":[{"comment":"The statement that 'all the 12 advanced models obtain near-random performance on M-Portal' is not supported by the reported fill-the-blanks accuracy. In Table 2, the random baseline is 3e-3, while GPT-o3 reaches 17.6%, Gemini-2.5-pro 16.1%, DeepSeek-R1-0528 8.4%, and Claude-3.7-Sonnet 6.8%—several orders of magnitude above random. The near-random claim is only true for the plan-correctness F1, whose interpretation is compromised by class imbalance (see next comment). Please revise the abstract and the corresponding discussion to state the results separately for the two subtasks, and do not describe the fill-the-blanks performance as near-random.","section":"Abstract, Section 3.4"},{"comment":"The plan-correctness task has an extreme class imbalance (1 correct candidate versus 31 incorrect per map). Under such imbalance, the F1 score for the positive class is highly insensitive: a model with 50% recall and 50% specificity on the 32 candidates would yield F1 approximately 6%, statistically indistinguishable from the random baseline. Thus the reported near-random F1 does not rule out partial reasoning ability. The authors should either report a metric that is informative under class imbalance (e.g., AUROC, or partial credit for detecting the presence or absence of mistakes), or explicitly state the diagnostic limits of this task and base the 'near-random M-Portal' claim primarily on fill-the-blanks.","section":"Section 3.1"},{"comment":"The 0% accuracy on CUBE is used as headline evidence of complete failure, but this result has insufficient statistical power to support that interpretation. For closed-source models evaluated on 200 examples (Appendix B), the 95% Clopper-Pearson interval for 0/200 is [0, 1.8%], which is orders of magnitude above the random baseline (approximately 1e-5). For open-source models evaluated on 1,000 examples, the interval is [0, 0.37%]. In both cases, observing zero successes is exactly what would be expected under random guessing, and the upper confidence bound does not distinguish zero ability from a success rate well above random. The authors should report binomial confidence intervals, avoid phrasing such as 'completely fail,' and rely on the CUBE-easy results (which show a clear contrast between GPT-o3 and other models) as the primary evidence of difficulty. The 0% result can be reported as 'no successes in N trials,' but not as evidence that models are at or below random.","section":"Section 3.5, Table 3, Appendix B"},{"comment":"The correctness of the M-Cube evaluation rests entirely on the solution validator, but the paper gives no description of its algorithm, its handling of rotational symmetries and edge complementarity, or any validation that it accepts all valid solutions and rejects all invalid ones. If the validator has false negatives, the reported 0% on CUBE could be an artifact. Please provide a detailed specification of the validator and report a sanity check on a set of known-valid and known-invalid configurations (e.g., generated from the ground truth) to demonstrate that it has no false positives or false negatives.","section":"Section 3.2"}],"minor_comments":[{"comment":"The 0% whole-piece accuracy is a deterministic consequence of the per-cell accuracy (0.7^25 is approximately 0.0008) and is not an independent finding; please report per-cell accuracy as the primary metric and describe the 0% as a derived quantity.","section":"Figure 5"},{"comment":"The selection procedure for the '200 representative examples' used for closed-source models on M-Cube is not described. Please specify how these were sampled (e.g., stratified by difficulty) and confirm that the subset is balanced across puzzle configurations.","section":"Appendix B"},{"comment":"The sentence 'despite more than 10,000 tokens spent on thinking' is contradicted by Table 3, where several models (Llama-4-Scout, Qwen2.5-VL, GPT-4o, InternVL3-78B, GPT-o4-mini, GPT-o3) use fewer than 2,000 tokens on CUBE. Please correct or qualify this statement.","section":"Section 3.5"},{"comment":"The units for the random baseline '1e-5' are ambiguous (fraction vs. percentage). Please clarify the caption and ensure consistency with the text.","section":"Table 3"},{"comment":"The model name appears inconsistently as 'GPT-4o-mini' (Appendix B) and 'GPT-o4-mini' (main text and Table 3). Please standardize.","section":"Table 4 and Section 3.3"},{"comment":"The reference list contains a duplicate entry for MathVista ([15] and [16] are the same paper).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially valuable and the CUBE-easy results provide a useful signal, but the paper currently overstates its headline findings. The abstract needs revision to accurately reflect the subtask-wise results, and the M-Cube narrative should lead with CUBE-easy and the perception experiment rather than the uninformative 0% on CUBE. The description of the solution validator must be expanded to convince readers that the benchmark's ground truth is reliable. These issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MARBLE is worth a serious referee. The two tasks are genuinely new—Portal-derived plan evaluation and synthetic Happy Cube assembly—and the amount of human annotation in M-Portal plus the validator in M-Cube put it above most new benchmarks. The bootstrapped standard deviations and the perception/reasoning decomposition are nice touches.\n\nThe soft spots are real but fixable. The headline result, 0% accuracy on M-Cube, is statistically empty: closed-source models were only run on 200 examples, and with a random success rate of 1e-5, zero successes is fully consistent with random guessing. The Clopper-Pearson upper bound is 1.8%, so you cannot distinguish a model with no ability from one guessing. That does not kill the benchmark, but it removes one of the two headline pillars. The difficulty of M-Cube is better supported by CUBE-easy, where only GPT-o3 gets far above random, and by the perception experiment showing 0% full-piece accuracy. The paper should stop citing 0% on CUBE as evidence of a reasoning deficit.\n\nThe M-Portal results are also weaker than they look at first. Plan-correctness is a 1-in-32 minority class, so all models sitting at F1 ~6% is exactly the random baseline; the task is too imbalanced to show whether models have partial ability. Fill-the-blanks is more informative, and the gap between best models (17.6%) and random (0.003%) is the real signal there.\n\nI also flag the validator and human annotations as unverified. The entire near-zero narrative depends on the ground truth being correct and the validator not rejecting valid assemblies. The paper does not include code or data links in the arXiv text, so this is not checkable yet. That is a fixable gap, not a fatal one.\n\nNet: the central claim that current MLLMs struggle with multi-step multimodal spatial reasoning is still supported once you ignore the 0% number. This paper should go to peer review with a request to reframe the M-Cube results, release the benchmark artifacts, and add a clearer statistical treatment of the 0% claim. It is a useful resource for anyone building or evaluating multimodal reasoning models.","headline":"MARBLE is a genuinely new benchmark worth referee time, but the 0% on M-Cube headline is statistically empty; the real difficulty signal is in CUBE-easy and the perception task.","tokens_in":16856,"tokens_out":2444,"would_cite":true,"duration_ms":25863,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current multimodal AI models score near-random on a new spatial-planning benchmark, with all 12 failing the harder 3D cube-assembly task.","keywords":["multimodal reasoning","spatial reasoning","planning","MLLM evaluation","Portal 2","Happy Cube","perception bottleneck","chain-of-thought"],"falsifier":"Take a MARBLE CUBE instance and hand-assemble the six pieces physically; if a human can assemble a configuration that the validator rejects, the reported 0% result is an artifact of the evaluation. Alternatively, run any current MLLM with the image replaced by the exact 2D arrays and ask for the assembly; if it solves a full CUBE instance, the claim that current models score 0% would fall.","tokens_in":15942,"feed_emoji":"🧩","tokens_out":5565,"duration_ms":57789,"temperature":0.7,"pith_summary":"This paper tries to establish that state-of-the-art multimodal language models cannot yet perform complex, multi-step spatial reasoning and planning when the input is genuinely multimodal. It introduces MARBLE, a benchmark with two tasks: M-Portal, which asks models to judge or complete long plans for Portal 2 puzzles, and M-Cube, which asks models to assemble six 3D jigsaw pieces into a cube. On M-Portal all 12 tested models score near the random baseline on the harder binary task, and on M-Cube all score 0%; only simplified subtasks separate the strongest models, with GPT-o3 reaching 17.6% and 72%. The paper also shows perception is a bottleneck, since models fail even at transcribing a rendered piece into a grid. If the results hold, the benchmark gives the community a way to measure progress in multimodal reasoning separately from visual perception and final-answer retrieval.","feed_headline":"AI models fail a new spatial-planning test: 0% on hard cube task","feed_subtitle":"MARBLE's Portal-2 and 3D-cube tasks show near-random scores across 12 multimodal models.","key_machinery":"The machinery is the benchmark itself: M-Portal and M-Cube. M-Portal takes 16 community-made Portal 2 maps, each with human-annotated ground-truth chain-of-thought solutions and five deliberately mistaken steps; these are combined into 32 candidate plans for the plan-correctness binary task and into masked fill-the-blanks questions with 2n options. M-Cube disassembles a 5x5x5 cube into six synthetic jigsaw pieces rendered from random viewpoints, and a solution validator decides whether a proposed face assignment and orientation actually closes the cube; the full task has 6! times $8^{6}$ candidate assemblies, while CUBE-easy provides the pieces as 2D arrays, forbids flips, and pre-places four pieces, leaving 32 candidates. These controls let the benchmark separate perception (reading the image) from reasoning (searching the candidate space).","core_discovery":"The paper claims that MARBLE, a new benchmark of two tasks, shows current MLLMs cannot do complex multimodal spatial reasoning and planning. On M-Portal, all 12 models score near the random baseline on plan correctness (minority-class F1 around 6%) and at best 17.6% on fill-the-blanks (GPT-o3). On M-Cube, every model scores 0% on the full CUBE task; on the simplified CUBE-easy, only GPT-o3 does well, at 72%, while most other models remain near random. The paper argues this failure is not just reasoning: even converting a rendered jigsaw piece into a 5x5 bump/gap array yields only about 70% per-cell and 0% whole-piece accuracy, so perception is a separate bottleneck.","pith_inferences":["I infer from the one-blank versus many-blank gap that M-Portal is less a test of physics knowledge than of compositional planning over many coupled decisions; a model that plans hierarchically might do well without new visual abilities.","A testable extension the paper leaves implicit is to use the M-Cube solution validator at training time as a reward signal for reinforcement learning, since the validator already gives dense edge-conflict feedback.","The perception failure on a seemingly trivial task (76% per-cell, 0% per-piece accuracy) suggests that 3D rendering-to-array transcription, not reasoning, may be the binding constraint; replacing images with exact arrays should isolate reasoning and could raise CUBE scores.","Future work could combine the validator with interleaved thinking and tool use, which the paper notes many current reasoning models do not retain across rounds, making the interactive setup a natural testbed for agentic reasoning."],"forward_implications":["If the MARBLE results are right, near-random performance on M-Portal and 0% on CUBE mean existing MLLMs cannot yet produce or verify long, physically grounded plans from visual input.","The perception bottleneck suggests that improving MLLMs' ability to extract structured information from images, such as converting a 3D piece into a 5x5 array, is a prerequisite for progress on these tasks.","The fill-the-blanks result, around 70% with one blank but below 1% with four or more, implies that current models can do local reasoning but fail to integrate long chains of interdependent steps.","The validator-assisted setup shows that diagnostic feedback helps on the simplified task (GPT-o4-mini rises from 10% to 28% over five rounds) but does not yet transfer to the full task.","A benchmark that rewards the reasoning process rather than the final answer gives future model development a concrete target for multi-step, multimodal planning evaluation."],"supporting_citations":[{"why":"Supplies the closest prior escape-room benchmark that M-Portal is compared against, where frontier models achieved high escape rates.","marker":"[26]"},{"why":"Provides the ARC-like-hard-test motivation for building a challenging multimodal reasoning benchmark.","marker":"[6]"},{"why":"Reports that most commercially available Happy Cube puzzles have only one solution, supporting the claim that M-Cube has a very small valid solution space.","marker":"[1]"},{"why":"Establishes the chain-of-thought stepwise reasoning paradigm that M-Portal and M-Cube are designed to evaluate.","marker":"[27]"},{"why":"Represents the class of existing multimodal benchmarks that MARBLE argues conflate perception with reasoning and overemphasize final-answer accuracy.","marker":"[30]"}],"fun_headline_variants":["0% on cube task: All 12 AI models fail MARBLE benchmark","MARBLE stumps all 12 multimodal models: 0% on hard cube task","Zero spatial reasoning: 12 models score 0% on MARBLE's cube task","Near-random on portal, zero on cube: MARBLE exposes AI limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's ground truth is correct: the human-annotated Portal solutions and mistaken steps and the M-Cube solution validator must accurately reflect which plans and assemblies actually work.","fun_headline_variants_meta":{"raw":{"variants":["0% on cube task: All 12 AI models fail MARBLE benchmark","MARBLE stumps all 12 multimodal models: 0% on hard cube task","Zero spatial reasoning: 12 models score 0% on MARBLE's cube task","Near-random on portal, zero on cube: MARBLE exposes AI limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3477,"prompt_tokens":961,"completion_tokens":2516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2430}},"tokens_in":577,"tokens_out":2516,"duration_ms":18337,"temperature":1.0,"reasoning_tokens":2430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:52:23.481666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a MARBLE CUBE instance and hand-assemble the six pieces physically; if a human can assemble a configuration that the validator rejects, the reported 0% result is an artifact of the evaluation. Alternatively, run any current MLLM with the image replaced by the exact 2D arrays and ask for the assembly; if it solves a full CUBE instance, the claim that current models score 0% would fall.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that most commercially available Happy Cube puzzles have only one solution, supporting the claim that M-Cube has a very small valid solution space."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Establishes the chain-of-thought stepwise reasoning paradigm that M-Portal and M-Cube are designed to evaluate."},{"cited_title":"MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi","cited_arxiv_id":null,"evidence_quote":"Represents the class of existing multimodal benchmarks that MARBLE argues conflate perception with reasoning and overemphasize final-answer accuracy."}],"review_version":1}