{"id":"400d241b-b5c5-4173-8fcd-1a145962e7f9","arxiv_id":"2412.04531","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MageBench introduces a 483-scenario benchmark showing current large multimodal models are far weaker than humans at agent tasks requiring continuous visual feedback and planning.","lead":"MageBench is a new benchmark that tests large multimodal models as agents in three lightweight environments: rebuilding web pages from screenshots, solving Sokoban puzzles, and playing football. It finds that even the best product-level models perform far below humans and often barely beat random action.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sokoban historical-best reward may inflate the random baseline, undermining the claim that LMMs are barely better than random at planning.","rationale":"The reader's weakest assumption correctly identified the hand-designed metrics as the load-bearing point. My stress-test agrees but narrows the problem to a specific, concrete flaw in the Sokoban reward definition (historical-best) that directly affects the 'better than random' comparison and the 'severe lack of planning' conclusion. This is not a rejection of the benchmark: the paper provides valuable environments, code, and a broad evaluation that likely supports a qualitative gap between LMMs and humans. However, the headline quantitative claim (only a few models above random, and barely) may be an artifact of a metric that rewards lucky early progress and suppresses real differences. The proposed test is a simple reanalysis of existing data, and conditional acceptance with this check is consistent with the reader's verdict. Other aspects—such as the WebUI PSO-calibrated AES and the human baseline—were considered but are less central because WebUI models clearly exceed random and the human gap is enormous. The Sokoban metric is the most delicate load-bearing element in the paper's main argument.","tokens_in":40269,"tokens_out":4713,"duration_ms":53155,"concrete_test":"Recompute the Sokoban reward using a final-state metric: e.g., cumulative reward at trajectory end (or number of boxes on targets at step 50) instead of historical-best reward, for the random baseline, GPT-4o, and Gemini on the same 182 levels. If the GPT-4o/random gap widens substantially (e.g., from 6.4 to >15 points) or if models previously below random move above it, the historical-best choice is a load-bearing artifact and the 'severe lack' claim needs re-evaluation. Also compute 95% confidence intervals for the Football reward differences to verify which models truly exceed random, given the reported δ values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that current LMMs 'severely lack' ViC-type reasoning rests on the small gap between model and random performance in Sokoban and Football. The Sokoban metric (Appendix A.2.3) uses the historically best cumulative reward minus the optimal-trajectory reward, explicitly retaining any early reward even if the agent later undoes its progress. This rewards lucky early pushes: a random policy that happens to push a box onto a target in the first few steps keeps that +4.5, while a deliberate planner that solves the level after more steps receives the same incremental credit. As a result, the random baseline (46.61) is likely inflated relative to a metric based on final state or completion, compressing the gap to GPT-4o (53.03) and Gemini (51.84). The paper's evidence that models are 'far from ideal for agent and robotics applications' depends on this gap being meaningful. In Football, the margin is also fragile: Gemini (18.33) is within the reported δ = ±2.2 of random (17.33), so only GPT-4o may actually exceed randomness. If the Sokoban metric does not credit planning progress that is undone, the Best-of-N analysis and the 'severe lack of visual imagination' conclusion lose force.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MageBench, a multimodal agent benchmark with three lightweight environments (WebUI, Sokoban, Football) designed to evaluate the \"vision-in-the-chain\" (ViC) reasoning abilities of large multimodal models (LMMs). The authors define two agent settings, Global (planning from the initial observation) and Online (interactive planning with per-step visual feedback), and evaluate 14 models, alongside idle/random baselines and human performance. Their headline finding is that only a few product-level models outperform random action selection, and all are far below human level, which they interpret as strong evidence that current LMMs lack ViC-type reasoning, visual imagination, and interleaved image-text long-context handling. The paper also reports Best-of-N scaling curves, error statistics, memory ablations, and qualitative generalization studies to robotics and structured visual generation. Code and data are promised to be released.","tokens_in":40487,"tokens_out":7022,"duration_ms":73390,"significance":"If the conclusions are robust, MageBench would be a useful, lightweight, reasoning-oriented benchmark for evaluating LMMs as agents, filling a gap between VQA-style benchmarks and heavy agent environments. The ViC formulation is a helpful conceptual contribution, and the careful engineering—repeated runs, random/idle/human baselines, and human-preference-calibrated WebUI metric—are strengths. The release of code and data would lower the barrier for the community. However, the central claim depends critically on the design of the three evaluation metrics; the Sokoban and Football rewards are manually constructed and the random baseline margins are small, so the paper's headline conclusions need sensitivity checks before they can be taken at face value.","major_comments":[{"comment":"The Sokoban evaluation uses the historical best cumulative reward (R = max_t sum R(tau) - R_best + 100) rather than a final-state or completion-based metric. Because early rewards are retained even if progress is later undone, a random policy that happens to push a box onto a target in the first few steps keeps that +4.5, while a planner that solves the level after many steps receives the same incremental credit. This likely inflates the random baseline (46.61) relative to a final-state metric, compressing the gap to GPT-4o (53.03) and Gemini (51.84). Since the claim in Sec. 5.1 that \"only GPT-4o and Gemini performed better than the random baseline\" in Sokoban-Online is load-bearing for the paper's central conclusion, please provide an ablation with an alternative metric (e.g., number of boxes on targets at trajectory end, or completion rate) to show the conclusion is not an artifact of the historical-max reward.","section":"Appendix A.2.3"},{"comment":"The Football reward is a manually weighted sum of six sub-rewards with fixed coefficients, and the random baseline (17.33) is within the reported δ = ±2.2 of Gemini's score (18.33). Under that precision, Gemini cannot be said to outperform random in Football; only GPT-4o (21.20) is clearly above the baseline. Nevertheless, Sec. 5.1 states that \"only GPT-4o and Gemini performed better than the random baseline\" in Football. This overclaim should be corrected, and the authors should report proper statistical tests (e.g., paired bootstrap) for the Football comparison. Additionally, the sensitivity of the conclusion to the hand-chosen λ values should be assessed (e.g., by re-evaluating with a range of weights or a secondary win-rate metric).","section":"Sec. 3.3, Eq. (1); Table 1"},{"comment":"The headline statement \"only a few product-level models are better than random acting\" is not true for the WebUI environment, where the random baseline is 0.00 and every evaluated model scores above 0. If the intended claim is restricted to Sokoban and Football, the abstract and Sec. 5.1 should say so explicitly; otherwise, this overgeneralization misrepresents the results and weakens the credibility of the central claim. Please revise the phrasing to match the actual data in Table 1.","section":"Abstract and Sec. 5.1"},{"comment":"The conclusion that models \"failed to boost the result with browser's rendering feedback\" is based on comparing WebUI-Global (generate all code at once) with WebUI-Online (modify the Global output using rendered images). These settings differ in more than the presence of visual feedback: the Online task involves code modification and a more complex multi-turn interaction, which may independently affect performance. Although the HD-CoT-NoIMG condition in Fig. 5 provides a no-vision control, the main Table 1 comparison (Global vs. Online) is not fully controlled. Please clarify whether the Online scores are computed from the same initial code for all models, and discuss the potential confound between task structure and visual feedback.","section":"Sec. 5.1 and Fig. 5"}],"minor_comments":[{"comment":"There is a spelling typo: \"enviroments\" should be \"environments\".","section":"Fig. 1"},{"comment":"The phrase \"to assist humen\" contains a typo (\"humen\" should be \"humans\").","section":"Sec. 1"},{"comment":"The word \"javascipt\" appears in prompt templates; it should be \"javascript\".","section":"Appendix E.1.4 and E.3.1"},{"comment":"The abstract states \"We release our code and data,\" but Sec. 6 says \"we will release the code and data as open-source in the near future.\" Please make these statements consistent, and indicate the exact release status.","section":"Sec. 6 vs. Abstract"},{"comment":"The abbreviation \"IFE\" is defined in the Table 1 caption, but Table 2 uses \"IA\" (invalid actions). Use consistent terminology (e.g., \"IA\" vs. \"IFE\") and define all abbreviations at first use.","section":"Table 1 and Table 2"},{"comment":"The gray-shaded areas are described as \"regions of variance\" but no definition is given. Specify whether these are standard deviations, confidence intervals, or min-max ranges.","section":"Fig. 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is carefully constructed and the paper is positioned well for the multimodal-agent community. The main reservations are the sensitivity of the Sokoban metric, the statistical significance of the Football result, and the overgeneralized abstract claim. If these are addressed with additional experiments and a revised presentation, the paper could become a solid contribution. I would also encourage the authors to release the code and data immediately, as the reproducibility of the claims will be a key factor for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MageBench is worth reading: it evaluates LMMs as interactive agents in three lightweight environments, and the central observation is credible — even the best product models are far below human-level when they must plan from continuous visual feedback. The ViC framing is a useful reframing, and the benchmark itself is carefully built: repeated runs, error analysis, a PSO-calibrated WebUI metric that agrees with human rankings 93% of the time, and an auto-rendering trick that cuts Football API calls by 80% without changing measured rewards. They release code and data, which is real evidence of reproducibility.\n\nThe main soft spot is the abstract's claim that \"only a few product-level models are better than random acting.\" That is true for Sokoban and Football, but not for WebUI, where every model beats a random baseline of 0. The claim should be scoped to the planning-heavy environments. The Sokoban metric also deserves scrutiny: using the historically best cumulative reward can credit a lucky early push even if the agent later undoes it, which likely inflates the random baseline. That said, the effect is symmetric, and the gap to human performance is enormous under any reasonable metric, so this weakens the \"barely better than random\" framing without sinking the overall conclusion. A robustness check using a final-state or completion-based metric would settle it.\n\nFootball is more fragile: Gemini-1.5-pro's 18.33 sits within the reported error bars of random (17.33), so only GPT-4o clearly clears randomness there. The paper should report this explicitly rather than lumping both together. The human baseline is also thin — a few PhD candidates, no error bars, and they practiced outside the mini set. That needs more documentation for a benchmark that makes human-level comparisons.\n\nOne internal inconsistency: Appendix B.1 calls the WebUI Online setting \"essentially non-functional,\" yet Table 1 reports Online scores. That needs clarification.\n\nThese are fixable issues, not load-bearing flaws. The benchmark is useful, the evaluation is broad, and the code release is a plus. I would send this to review, asking the authors to scope the random-level claim and add a robustness analysis for the Sokoban and Football metrics. A revised version would be a solid reference for anyone building or evaluating multimodal agents.","headline":"A well-engineered benchmark with a real finding, but the headline claim about random baselines is overbroad and needs scoping before this is publication-ready.","tokens_in":41058,"tokens_out":2585,"would_cite":true,"duration_ms":29500,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MageBench shows current LMMs severely lack vision-in-the-chain reasoning: only a few models barely beat random acting, and all fall far below human level.","keywords":["multimodal agents","vision-in-the-chain reasoning","LMM evaluation benchmark","visual feedback","Sokoban planning","spatial reasoning","web UI reconstruction","embodied AI"],"falsifier":"Check whether a policy that only moves in a circle or repeats a single action scores above random in Sokoban or Football; if such a trivial policy earns a high historical-best or dense reward, the metric does not isolate the claimed reasoning abilities. Alternatively, if a model trained purely on ViC-style image-action traces reaches human-level under the same unified prompt on all 483 scenarios, the claim that current models lack the underlying capability would be overturned.","tokens_in":40047,"feed_emoji":"🤖","tokens_out":6507,"duration_ms":63678,"temperature":0.7,"pith_summary":"MageBench introduces a benchmark of 483 lightweight scenarios across three environments, WebUI page reconstruction, Sokoban puzzles, and a football game, designed to test whether large multimodal models can reason when visual feedback arrives continuously during decision-making, a pattern the authors call vision-in-the-chain (ViC). The paper claims that current LMMs, both open and closed, perform near or below random-acting baselines in the online settings, and that even the strongest product-level models stay far below human-level performance. The intended upshot is that today's multimodal models are not yet reliable agents for robotics or other tasks that require adapting plans to what the eyes see, because the missing capability is not language reasoning but visual-feedback-driven reasoning, visual imagination, and spatial planning.","feed_headline":"MageBench: top LMMs barely beat random acting as agents","feed_subtitle":"Only a few models outperform random and none approach human-level across 483 scenarios.","key_machinery":"The load-bearing object is the vision-in-the-chain (ViC) reasoning paradigm, defined as decision-making in which image observations enter the reasoning chain at every step and plans must be revised from visual feedback. MageBench instantiates ViC through three lightweight but planning-heavy environments, WebUI (rebuild a webpage from a description and screenshots), Sokoban (push boxes in minimal steps), and Football (control the ball handler to score), and through two baseline agents: a Global planner that sees only the opening image and an Online planner that receives fresh frames and acts. The scoring machinery compares model performance against random and idle baselines and human reference performance, using a historical-best reward in Sokoban, a dense composite football reward, and a human-calibrated Atomic Element Similarity score for WebUI.","core_discovery":"The paper's central claim is that current large multimodal models severely lack vision-in-the-chain reasoning: the ability to keep looking, update a plan, and act in a loop, as opposed to reasoning over a fixed image with text-only chain-of-thought. MageBench operationalises this with 483 tasks in three environments, and under a unified agent protocol the evaluation shows that in the online, feedback-driven setting only two product-level models beat the random-acting baseline, while all models remain far below the human reference. It further finds that models do not improve when given rendered visual feedback to revise code, do not benefit from longer action and observation memory, and score poorly on best-of-N in Sokoban, which the paper interprets as evidence of absent visual imagination and think-ahead ability.","pith_inferences":["A plausible extension not made in the paper is that ViC failures will show up in any human-in-the-loop or robotics setting where the model must ground each decision in a fresh camera view; the qualitative clean-table and book-stacking probes in the appendix point in this direction.","The benchmark's metric design suggests a testable refinement: if the football reward is meant to measure tactical value, then an independent human ranking of the same decision trajectories should correlate with the dense reward; a mismatch would indicate the reward rewards intermediate proxies rather than winning play.","The paper leaves open whether ViC is a data problem or an architecture problem; a natural follow-up is to train a model on interleaved image-action traces in these same environments and see whether the human gap closes without architectural change."],"forward_implications":["If current models truly lack ViC reasoning, then gains on static image QA and text chain-of-thought benchmarks will not transfer to multimodal agent applications; agent capability must be measured with interactive visual feedback.","The WebUI results imply that relying on browser-rendered feedback for self-correction does not currently help models, so engineering agents that assume models can debug their own visual output will underdeliver.","The best-of-N results suggest football-style tasks may be amenable to reinforcement learning because sampling many trajectories yields strong improvements, while Sokoban-style spatial planning may require mechanism-level training changes rather than more sampling.","Models' failure to improve with longer action and observation memory indicates a bottleneck in interleaved image-text long-context processing, not just in planning."],"supporting_citations":[{"why":"Supplies the Sokoban rendering environment, reward scheme, and minimum-step levels that MageBench adapts.","marker":"[61]"},{"why":"Supplies the football simulation engine used to generate the 108 scenarios and simulate play.","marker":"[36]"},{"why":"Product-level model that is one of the two to beat random in the online setting; a central comparison point for the failure claim.","marker":"[2]"},{"why":"The other product-level model to beat random online; a central comparison point for the failure claim.","marker":"[67]"},{"why":"Closed product-level model that nearly reaches human WebUI-Global level, used to show feedback-driven online revision does not help.","marker":"[1]"},{"why":"Defines the zero-shot chain-of-thought prompt style used in the unified agent protocol.","marker":"[35]"},{"why":"Defines the reason-and-act online agent paradigm that MageBench's Online planner instantiates.","marker":"[91]"},{"why":"Existing agent benchmark whose environment-oriented design MageBench contrasts with its reasoning-oriented selection.","marker":"[47]"},{"why":"Existing multimodal agent benchmark whose environment choices the paper argues are less transferable to structured visual generation.","marker":"[48]"}],"fun_headline_variants":["MageBench: top LMMs barely beat random in agent tasks","Vision-in-the-chain: LMMs fail at visual feedback planning","MageBench: LMMs can't update plans from visual signals","LMM agents: no better than random when vision feeds reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that the three hand-designed metrics (Sokoban historical-best reward, dense football reward, and web Atomic Element Similarity) actually measure planning, feedback integration, and imagination, rather than rewarding lucky early progress or instruction-following artifacts.","fun_headline_variants_meta":{"raw":{"variants":["MageBench: top LMMs barely beat random in agent tasks","Vision-in-the-chain: LMMs fail at visual feedback planning","MageBench: LMMs can't update plans from visual signals","LMM agents: no better than random when vision feeds reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1167,"prompt_tokens":938,"completion_tokens":229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":153}},"tokens_in":554,"tokens_out":229,"duration_ms":3019,"temperature":1.0,"reasoning_tokens":153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:31:15.065238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether a policy that only moves in a circle or repeats a single action scores above random in Sokoban or Football; if such a trivial policy earns a high historical-best or dense reward, the metric does not isolate the claimed reasoning abilities. Alternatively, if a model trained purely on ViC-style image-action traces reaches human-level under the same unified prompt on all 483 scenarios, the claim that current models lack the underlying capability would be overturned.","supporting_citations":[],"review_version":1}