{"id":"c4018863-4184-476f-9f18-b8b583b63058","arxiv_id":"2607.15641","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IMBench is a 35-task robosuite benchmark with a three-stage evaluation showing current VLMs and robot policies fail to convert physical reasoning into executable manipulation.","lead":"IMBench is a new simulated manipulation benchmark that tests whether AI systems can turn physical understanding into real robot actions, spanning 35 tasks from sliding plates to balancing rods. Early results show vision-language models can describe what to do but fail when actually executing, exposing a gap current benchmarks were not designed to measure.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage 3 zero scores may confound missing intuitive manipulation with an inexpressive action-primitive interface; no privileged oracle using the same eight primitives is reported.","rationale":"The reader's weakest assumption identifies the most load-bearing threat to the central claim. The abstract's strongest inference is that VLMs cannot convert physical understanding into executable plans; that inference is drawn almost entirely from Stage 3, where only GPT-5.5 is run and where the only way to act is by selecting from the eight-primitive vocabulary in F.4. If that vocabulary cannot represent force-controlled or dynamically timed behaviors, a model with perfect physical intuition could still score 0% on tasks like balance-medium or slide-catch. The paper's own scripted oracles (C.3) use privileged state and raw delta actions; they demonstrate task solvability under the low-level action space, not the high-level primitive space used in Stage 3. This is exactly the missing control condition. I find no internal inconsistency or overt overclaiming beyond this: Table 2 and Table 15 show policies learning some contact-tolerant tasks and failing precision tasks, which is coherent with a benchmark that requires precision; the OOD drops (e.g., balance-medium 0.71 to 0.12) are large and qualitatively support the generalization claim. The benchmark artifact itself is detailed and internally consistent, with a plausible data collection and filtering pipeline. The main reason the central claim is not yet proven is the Stage 3 interface confound, compounded by the single-VLM design and unreported Stage 3 episode counts. The proposed oracle test would settle whether the interface explanation accounts for the zeros. This is a request for an additional control, not a refutation; hence the reader's CONDITIONAL verdict remains appropriate.","tokens_in":35585,"tokens_out":5878,"duration_ms":65811,"concrete_test":"Implement a privileged oracle control for Stage 3: for each of the 16 tasks in Table 10, replace the VLM's action selection with the task's canonical plan from Appendix B, executed by a scripted policy with ground-truth state that uses only the same eight primitives, the same 24-step budget, and the same termination conditions. Report success on the same evaluation seeds. If the oracle solves a task (e.g., balance-medium or recover-peg-insert) while GPT-5.5 scores 0%, the zero is evidence of the model's planning/execution failure; if the oracle also fails, that task's Stage 3 zero is an interface artifact and must be excluded or the primitive set expanded before concluding that VLMs cannot produce executable plans.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that VLMs 'fail to produce executable plans' rests on Table 10: GPT-5.5 scores 0% on 13 of 16 Stage 3 tasks. But Stage 3 (Appendix F.4) restricts the VLM to eight named primitives — MOVEEEF, CLOSEGRIPPER, OPENGRIPPER, LIFT, DESCEND, ATOMICACTION, HOLD, DONE — within a 24-step budget. Several zero-score tasks (recover-peg-insert, slide-catch, cup-extract, balance-medium) appear to require continuous force shaping, velocity-controlled release, tool pivoting, or iterative contact adjustment that may not be expressible as a short sequence of target-setting primitives. If these behaviors cannot be expressed, the 0% scores could reflect interface limitations rather than absence of intuitive reasoning. The paper never reports a control condition that decouples these factors: the scripted oracles in C.3 use raw 6-DoF delta actions and privileged state, not the Stage 3 primitive set, so they validate task solvability under a different interface. ATOMICACTION does allow raw 7D actions, but asking a VLM to synthesize raw action vectors is not equivalent to showing the primitive vocabulary is sufficient. Thus the headline inference — a missing capability axis in foundation models — is not yet cleanly separated from an interface artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IMBench, a robosuite-based benchmark of 35 manipulation tasks organized into seven physical-reasoning categories (P1–P7, plus two miscellaneous tasks). It proposes an Understand–Infer–Act decomposition and evaluates three capabilities: VLM constraint understanding, high-level plan generation, and closed-loop execution, alongside end-to-end visuomotor policies. The dataset contains ~14K curated trajectories. The headline results are that frontier VLMs achieve ~74% on constraint understanding but Stage 3 closed-loop execution for GPT-5.5 succeeds on only 3 of 16 tasks (11.3% mean), and VLAs score near zero zero-shot and remain low after finetuning; OOD perturbations cause large drops, e.g., balance-medium from 0.71 to 0.12. The paper interprets these results as evidence that intuitive manipulation is a missing capability axis in current models.","tokens_in":35928,"tokens_out":6895,"duration_ms":58274,"significance":"If the measurement is valid, IMBench fills a useful gap: it is more integrated than physical-reasoning QA benchmarks and more physics-focused than standard manipulation benchmarks, and the three-stage design separates understanding, planning, and execution. The task suite is structured around explicit inference bottlenecks, the OOD axes are specified per task, and the policy evaluation covers several baselines. However, the central Stage 3 finding is currently confounded by the action-primitive interface, and the human-evaluation rubrics are not released. These issues must be addressed before the benchmark's main claim can be considered established.","major_comments":[{"comment":"The claim that VLMs 'fail to produce executable plans' rests on the Stage 3 scores, but the action interface may itself be the binding constraint. Appendix F.4 restricts the agent to eight named primitives (MOVEEEF, CLOSEGRIPPER, OPENGRIPPER, LIFT, DESCEND, ATOMICACTION, HOLD, DONE) within a 24-step budget. Several zero-score tasks — recover-peg-insert, slide-catch, cup-extract, balance-medium — plausibly require continuous force shaping, velocity-matched release, or iterative contact adjustment that this vocabulary cannot express. The scripted oracles in Appendix C.3 use raw 6-DoF delta actions and privileged state, so they validate solvability under a different interface, not under the Stage 3 primitive set. Please add a control condition: a privileged oracle or scripted policy that selects from the same eight primitives with perfect state information, with per-task success rates. If t","section":"§4.2.1 / Table 10 / Appendix F.4"},{"comment":"Stage 1 and Stage 2 scores are human-evaluated, but the task-specific rubrics are not provided in the manuscript or supplement, and no inter-annotator agreement is reported. The text says full prompt templates and thresholds are 'provided in the supplementary material,' but the submission contains no such material. The Stage 2 criterion — a human verifier finds the plan sufficient — is underspecified. Since the Q1/Q2 results and the understanding-to-plan gap are central to the paper's narrative, please include the actual rubrics (or a representative sample covering each category) and report inter-annotator agreement (e.g., Cohen's or Fleiss' kappa) on a subset of responses.","section":"§4.1 / Appendix F.2–F.3"}],"minor_comments":[{"comment":"The Table 2 P2 mean for π0.5 FT is 0.09, but summing Table 15 task values (0.12 + 0.28 + 0.04 + 0 + 0 + 0.10 + 0 = 0.54 over 7 tasks) gives 0.077, not 0.09. Please correct or explain.","section":"Table 2 vs Table 15"},{"comment":"The phrase 'rely heavily on memorized patterns' is stronger than the evidence. OOD success drops show lack of extrapolation, not necessarily memorization. Suggest rephrasing to 'do not generalize along the target physical axis' unless a representation-level analysis is added.","section":"§4.3"},{"comment":"For a benchmark contribution, the release of data, code, evaluation harness, and full annotation rubrics is essential; currently only a 10-episode sample is available. Also remove the placeholder acknowledgment paragraph.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The Stage 3 primitive-interface confound is the main technical risk. If the authors add a same-interface oracle control and report per-task seed counts, the paper could become a solid benchmark contribution. I do not see grounds for rejection, provided the Stage 3 conclusions are appropriately qualified if the control reveals interface limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: IMBench is a real contribution and the core benchmark claim is defensible, but the paper's headline inference about a missing capability axis in foundation models is supported mainly by Stage 3, and Stage 3 is not cleanly separated from an interface limitation.\n\nWhat's actually new: a 35-task suite built on robosuite with tasks organized around explicit inference bottlenecks across seven categories; a three-stage protocol that separates constraint understanding, high-level plan correctness, and closed-loop execution; and about 14K curated trajectories with both teleop and scripted demonstrations. The KinDER comparison is fair and the benchmark is differentiated from physical-reasoning QA suites and standard manipulation benchmarks. That is a useful artifact.\n\nThe paper does several things well. The task designs are thoughtful, with clear canonical plans. The internal numbers are coherent: Stage 3 means in Table 1 match Table 10, and OOD perturbation axes are specified rather than left vague. The limitations section is honest about simulation-only evaluation, the Franka embodiment, fixed task descriptions, and lack of deformables. The policy baselines (diffusion policy, π0.5, GR00T) are reasonable and the per-task rollout analysis is genuinely informative.\n\nThe soft spots are real. First and most important: Stage 3 restricts the VLM to eight motor primitives with a 24-step budget, while the scripted oracles that validate task solvability use raw 6-DoF delta actions and privileged state. For tasks like recover-peg-insert, slide-catch, cup-extract, and balance-medium, the zero scores may reflect an inexpressive action interface rather than absent intuitive reasoning. No control condition uses the same primitives with a privileged or scripted policy, so the interface explanation is not ruled out. The stress-test note lands.\n\nSecond, Stage 3 is evaluated with a single VLM (GPT-5.5), with per-task success rates that are 0 or 100 and no variance or confidence intervals. The cost explanation is understandable, but it still limits the strength of the inference.\n\nThird, the paper claims to release ~14K trajectories, but the full text says only a 10-episode subset is released for review and the complete benchmark will be released upon acceptance. That is a mismatch between the contribution statement and the current artifact. Human evaluation rubrics and inter-annotator agreement are also unspecified.\n\nNone of this sinks the core benchmark, but the missing-capability-axis conclusion is stronger than the Stage 3 evidence alone cleanly supports. I would send this to peer review, with required revisions: release the data and code, add a Stage 3 control using the same primitives with privileged or scripted execution, broaden Stage 3 to at least another VLM, and report rubric agreement. For the robot-learning community, this is worth engaging with.","headline":"IMBench is a genuinely useful benchmark artifact with coherent internal numbers, but the headline claim about a missing capability axis rests on a Stage 3 evaluation whose zero scores are confounded by an unvalidated action-primitive interface, and the full dataset is not actually released yet.","tokens_in":36440,"tokens_out":2824,"would_cite":true,"duration_ms":30808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IMBench shows that reasoning about physics and acting on it are sharply separated capabilities in current AI systems.","keywords":["intuitive manipulation","robotic manipulation benchmark","physical reasoning","vision-language-action models","embodied AI","constraint understanding","closed-loop execution","generalization"],"falsifier":"Run the paper's own hand-written scripted policies—which solve the tasks using privileged simulator state—through the same eight-primitive action interface used for the vision-language agent. If the oracle also fails on the 13/16 tasks where the agent scored zero, the primitives are the bottleneck; if the oracle succeeds, the failures are attributable to the model's reasoning and grounding.","tokens_in":35507,"feed_emoji":"🤖","tokens_out":5020,"duration_ms":50884,"temperature":0.7,"pith_summary":"The paper introduces IMBench, a benchmark of 35 simulated manipulation tasks designed to test whether AI systems can do what humans do almost effortlessly: read a physical scene, infer hidden constraints, and act on that understanding. It argues that current evaluation separates physical reasoning from execution, and that the two should be measured as one integrated capability, called 'intuitive manipulation'. Using a three-stage protocol—constraint understanding, plan proposal, and closed-loop execution—the authors show that frontier vision-language models describe constraints with moderate accuracy (about 74%) but produce executable plans less often, and that success collapses to roughly 11% when plans must be executed. The benchmark's evidence supports the claim that intuitive manipulation is a missing axis: a capability current foundation models and robot policies do not yet possess. The release of 14K curated trajectories and scenario-generation tools is positioned as a measurement instrument for the field.","feed_headline":"AI models can't turn physics understanding into robot action","feed_subtitle":"A 35-task benchmark shows a sharp gap between describing physical constraints and executing them, even for frontier models.","key_machinery":"The load-bearing instrument is the three-stage Understand–Infer–Act protocol, together with the task design principle of an 'inference bottleneck'. Stage 1 probes whether a vision-language model can extract task-critical physical constraints; Stage 2 has the model output an ordered sequence of sub-goals; Stage 3 executes those plans through a closed-loop reasoning-acting agent armed with eight motor primitives (move end-effector, lift, descend, open/close gripper, hold, atomic action, done). The 35 tasks are grouped into seven categories, each isolating one intuition type: geometry-constrained grasping, dynamics/trajectory prediction, causal/indirect action, tool use, hidden state, reactive","core_discovery":"The central claim is that intuitive manipulation—the conversion of physical understanding into goal-directed, constraint-satisfying action—is a distinct capability axis that existing benchmarks do not isolate and current models do not exhibit. IMBench operationalizes this claim by building 35 tasks around explicit 'inference bottlenecks': hidden facts about geometry, dynamics, causality, hidden state, or stability that must be inferred from observation to succeed. On these tasks, the paper measures a consistent cascade: vision-language models identify task constraints with moderate success (best mean 74.1%), convert them into correct high-level plans slightly less often (69.5%), and almost n","pith_inferences":["A likely confound not resolved by the paper: the Stage 3 action interface supplies only eight predefined primitives; if those primitives cannot express continuous force shaping or compliant sliding, the near-zero execution scores could understate what the evaluated models could do with a richer action space. A scripted oracle executing correct high-level plans through the same primitives would sep","The paper leaves open whether adding force/torque feedback (which the benchmark records but the baselines do not use) would close part of the Stage 3 gap; a natural extension is to test policies trained with wrist force and tactile inputs on the same tasks.","If the intuitive-manipulation axis is real, the benchmark could serve as a selection criterion for embodied foundation models, complementing trajectory-following metrics.","The mirror-pick-place task suggests a cheap diagnostic: because humans adapt to a flipped camera frame within tens of trials, it could serve as a minimal test of whether a system can learn new observation-action correspondences quickly."],"forward_implications":["If IMBench measures what it claims, then high scores on physical-reasoning QA benchmarks do not imply the ability to manipulate objects; reasoning and execution must be evaluated jointly.","Generalist robot policies that saturate existing manipulation benchmarks should be expected to drop sharply on tasks whose success depends on physical inference.","Training or fine-tuning on physics-constrained tasks like IMBench's may be necessary for foundation models to ground reasoning in actionable plans.","The three-stage protocol offers a reusable template for localizing where a system loses competence: understanding, planning, or execution.","The benchmark's OOD perturbations provide a direct test of whether a policy generalizes along physical axes rather than memorizing patterns."],"fun_headline_variants":["AI understands physics but can't act on it","New benchmark exposes AI's missing 'intuitive manipulation' skill","Benchmark finds AI can reason about physics but not execute plans","35-task test shows AI fails to turn physics knowledge into action","IMBench: Why AI can describe physics but not handle a tool"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The Stage 3 execution results assume the eight predefined action primitives are expressive enough to realize the physical strategies each task requires; if a task demands continuous force shaping or compliant contact that those primitives cannot express, zero percent scores would reflect interface limits rather than absent intuitive reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI understands physics but can't act on it","New benchmark exposes AI's missing 'intuitive manipulation' skill","Benchmark finds AI can reason about physics but not execute plans","35-task test shows AI fails to turn physics knowledge into action","IMBench: Why AI can describe physics but not handle a tool"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00072,"raw_usage":{"total_tokens":3060,"prompt_tokens":728,"completion_tokens":2332,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":2248}},"tokens_in":472,"tokens_out":2332,"duration_ms":15819,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:41:07.615669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own hand-written scripted policies—which solve the tasks using privileged simulator state—through the same eight-primitive action interface used for the vision-language agent. If the oracle also fails on the 13/16 tasks where the agent scored zero, the primitives are the bottleneck; if the oracle succeeds, the failures are attributable to the model's reasoning and grounding.","supporting_citations":[],"review_version":1}