{"id":"9af8b61a-eb98-4748-86de-c59c7e5c0210","arxiv_id":"2607.27180","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new full-body benchmark shows that current VLMs can recognize targets but cannot reliably tell where their own body is, whether it arrived, or whether it collided; the best solves only 16.8% of episodes.","lead":"This paper introduces HumanCLAW, a way to test whether a vision-language model can drive a simulated human body by choosing small whole-body actions, with motor mistakes removed from the score. Tested on 1,218 indoor episodes, the best model finishes only 16.8% of tasks, failing mostly because it loses track of its own body.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Half-physics decoupling is asserted, not validated: contact artifacts (jamming/sliding) could be mislabeled as VLM body-awareness failures, undermining 'failures belong to the decision maker.'","rationale":"The paper's central scientific assertion is that observed failures are decision-level because motor execution is factorized out. The reader's weakest assumption targets precisely the unvalidated half-physics decoupling, and I agree that this is the most load-bearing condition. Table 1 validates the skill conditioner, not the contact behavior of the simulator; the 'unaware jammed' criterion could absorb simulator blocking. The authors themselves concede in Section 6 that the absence of a tactile or proprioceptive channel may make body awareness intrinsically difficult, which further weakens the 'VLMs lack self-awareness' framing. A parameter-sensitivity check of λ and Δt is the minimal experiment that would settle whether half-physics artifacts contaminate the attribution; without it the headline claim is an overreach. The benchmark, staged metrics, and root-cause analysis remain useful, so CONDITIONAL remains the right verdict.","tokens_in":34577,"tokens_out":7980,"duration_ms":91474,"concrete_test":"On the 100-episode mini-val with one strong model (e.g., Gemini-3.1), sweep the half-physics parameters: passive stiffness λ ∈ {0.1, 1.0, 10} and simulator step Δt ∈ {1/60, 1/120, 1/240}s, holding all VLM prompts fixed. Record NavSR, InteractSR, and the Appendix B 'unaware jammed' label rate; also log median contact penetration and foot-sliding distance per episode. If any metric shifts by more than ~5 points (or median penetration/sliding exceeds a few cm), the half-physics layer is not neutral and the 'failures belong to the decision maker' attribution fails. If the metrics are invariant across settings, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 replaces joint-torque actuation with velocities derived from generated poses (passive stiffness λ=1.0, 120Hz), and the abstract plus Section 4.4 conclude 'with motor execution factored out by construction, these failures belong to the decision maker.' The only quantitative support is Table 1, which measures skill fidelity in free motion (walk 0.966, side_step 1.002, turn 0.994, etc.); it does not measure what happens when the velocity-driven body meets a wall, stair edge, or movable object. The Appendix B rule 'unaware jammed' labels ≥8 consecutive locomotion steps with per-step displacement <0.1m as a body-awareness failure. Under half-physics this exact pattern can be produced by the kinematic velocity command pressing against a rigid contact while the passive body fails to advance—an execution-side artifact, not necessarily a decision to keep moving. No full-dynamics baseline, no contact-penetration/foot-sliding metric, and no sensitivity analysis over λ or Δt is reported. Therefore the decisive premise that motor artifacts are absent in all 41 HSSD scenes is unsupported; the reported 'self-awareness' failures may be simulator artifacts rather than VLM deficits.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HumanCLAW, an evaluation framework intended to separate a VLM's action decision-making from low-level motor execution. A frozen off-the-shelf VLM issues atomic whole-body skill commands (walk, turn, sit, etc.) with continuous parameters; a skill-conditioned motion generator produces 0.5s kinematic motion chunks; and a 'half-physics' simulator (AI Habitat + Bullet, 120Hz, passive stiffness lambda=1.0) executes them under gravity and contact while abstracting away balance and torque-level motor control. The authors build HumanCLAW-Bench with 1,218 find-navigate-interact episodes across 41 HSSD indoor scenes, evaluate nine VLMs, and report that the best model completes only 16.8% of episodes. They argue that perception is not the bottleneck; rather, the models fail at embodied self-awareness: knowing where the body is, whether it has arrived, and whether it has collided. Success metrics are staged (FindSR/NavSR/InteractSR) with geometric objective criteria plus model-acknowledged completion, and root-cause labels are assigned by deterministic rules in Appendix B.","tokens_in":34788,"tokens_out":5797,"duration_ms":64631,"significance":"If the half-physics decoupling is sound, HumanCLAW provides a useful middle layer between symbolic agent benchmarks and full-dynamics VLA evaluation: it keeps a full body in a closed-loop physical world while removing balance and motor-tracking failures, and it ships a large benchmark with transparent, rule-based error attribution and per-stage metrics. The comparison across nine models and the skill-fidelity measurements are also useful empirical assets. However, the central scientific claim—that the reported failures 'belong to the decision maker' and reveal a missing embodied self-awareness—depends on a validation step that the paper does not provide: demonstrating that contact interactions in the half-physics simulator are free of motor-side artifacts. The paper's own limitation paragraph also concedes that the interface gives the agent no proprioceptive or contact signal, which confounds the 'missing faculty' interpretation. The contribution is valuable and the core idea is defensible, but the headline claims currently exceed what the evidence supports.","major_comments":[{"comment":"The load-bearing premise is that half-physics 'factors out' motor execution while preserving physical interaction, and the abstract/Section 4.4 conclude 'with motor execution factored out by construction, these failures belong to the decision maker.' Table 1 supports free-space skill fidelity, but it does not characterize what happens when the velocity-driven body contacts a wall, stair edge, or movable object. The 'unaware jammed' rule (≥8 consecutive locomotion steps with displacement <0.1m) can be produced by a kinematic command pressing against rigid contact while the passive body fails to advance—an execution artifact rather than a decision to keep moving. Without a full-dynamics baseline, contact-penetration/foot-sliding metrics, and sensitivity analysis over λ=1.0 and Δt=1/120s across the 41 scenes, the decision-level attribution is unsupported.","section":"Section 2.4 and Appendix B (Table 7)"},{"comment":"The central conclusion that 'current VLMs lack embodied self-awareness' is confounded by the observation interface. The agent receives only egocentric RGB and text history, with no proprioceptive or contact signal, as Section 6 acknowledges: 'a body-state or contact signal could be the missing input rather than a missing faculty.' The benchmark therefore cannot distinguish a missing faculty from a missing input modality. The abstract and Finding 6 should be qualified accordingly; as written they overstate what the experiments establish.","section":"Section 4.4 (Finding 6) and Section 6"},{"comment":"The evaluated system is not the off-the-shelf VLM alone but the VLM plus the HumanCLAW harness, whose skill-specific verifier is an external component doing substantial decision-level work. Table 3 shows that removing the verifier collapses NavSR from 27.0% to 2.0% and InteractSR from 18.9% to 0.0%. The claim that failures 'belong to the decision maker' is therefore relative to a specific scaffold. The paper should state clearly that the benchmark evaluates VLMs under this harness, not the VLM's unaided action intelligence. This is acknowledged indirectly in Section 6 but not in the abstract.","section":"Section 2.2, Table 3, and Section 6"}],"minor_comments":[{"comment":"The text says the method 'tracks every commanded magnitude at ≈1,' but stair-climbing ratios are 0.794±0.019 and 0.738±0.052. These are stable gains, not near-unity tracking. The wording should distinguish displacement-tracking skills from skills with a calibratable gain.","section":"Table 1 / Section 4.1"},{"comment":"The listed percentages appear to sum to 102% (6+9+12+23+38+3+10+1). This is likely rounding, but the figure should be reconciled or a note added.","section":"Figure 8(b)"},{"comment":"The velocity conversion formula only states linear joint velocity; rotational components are described as 'computed from relative rotations in the corresponding tangent space.' The exact quaternion/log-map formulation should be given for reproducibility.","section":"Section 2.4"},{"comment":"The baseline on the 100-episode mini-val reports InteractSR=18.9% for Gemma-4-31B, while the full-val Table 2 reports 11.1% for the same model. The discrepancy is large enough that confidence intervals or a note on mini-val sampling variance should be provided.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main weakness is validation of the half-physics decoupling, not circularity: the metrics and root-cause rules are objective and reproducible. I would not reject, because the authors can add the missing full-dynamics comparison, contact-quality metrics, and λ/Δt sensitivity analysis, and can qualify the self-awareness claim to match the interface limitations they already acknowledge. My concern is that the abstract and Section 4.4 make a stronger claim than the experiments currently support; if the authors are unwilling to qualify those claims, I would move toward rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is a real contribution: a testbed for VLM action intelligence with a clean separation between skill selection and low-level motion, plus a staged benchmark that shows all nine tested models fail badly (best 16.8% on the full task). But the headline conclusion—'failures belong to the decision maker'—goes beyond the evidence. The half-physics simulator that does the decoupling isn't validated for contact scenarios, and the self-awareness diagnosis is oversold.\n\nWhat's genuinely good: the atomic skill interface with plug-and-play ControlNet adapters is a smart way to let a frozen VLM drive a whole-body humanoid; Table 1 shows skills track commanded magnitudes (walk 0.97, turn 0.99, etc.), so in free motion execution isn't the issue. The benchmark is carefully staged—FindSR/NavSR/InteractSR separate geometric success from acknowledged success, and the difficulty tiers are transparent. The root-cause rules in Appendix B are deterministic and reproducible, and the ablation showing the verifier is decisive is informative.\n\nThe soft spots are real. The half-physics setup is the load-bearing wall, and it's only validated for free motion. The 'unaware jammed' rule labels ≥8 steps with displacement <0.1m as a body-awareness failure, but in a kinematic-driven passive body pressing against a rigid contact, the same pattern emerges without the model making any decision error—it's a simulator artifact. There's no sensitivity analysis on λ or timestep, no contact/foot-sliding metrics, no full-dynamics baseline. That means the strong attribution claim in the abstract isn't backed. Also missing: error bars, code/data release, and a random-policy baseline that would calibrate the benchmark's difficulty. The authors do acknowledge in Section 6 that a missing tactile or proprioceptive channel could be the real problem, which undercuts the abstract's 'embodied self-awareness' framing. That's the right caveat; it should be in the abstract too.\n\nBottom line: the negative result (frontier VLMs can't do embodied find-navigate-interact) is credible and useful. The diagnosis of what's missing is a hypothesis, not a proven fact, until the simulator is validated and baselines are added. This deserves peer review—the framework is reusable and the results are sharp enough to be worth referee time—but it needs a revision that either validates the half-physics contact behavior or scales back the claim to match what's shown. I'd recommend sending it to review with that expectation.","headline":"A useful benchmark and a credible negative result, but the 'self-awareness' diagnosis goes beyond what the half-physics validation supports.","tokens_in":35446,"tokens_out":4202,"would_cite":true,"duration_ms":42603,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HumanCLAW claims that current vision-language models fail at embodied action because they lack embodied self-awareness, not because they cannot perceive targets.","keywords":["embodied self-awareness","vision-language models","action intelligence","half-physics simulation","humanoid motion generation","find-navigate-interact benchmark","egocentric spatial reasoning"],"falsifier":"Run the same 1,218 episodes under a fully dynamic torque-controlled humanoid and under pure kinematic playback. If half-physics outcomes diverge from both, or if commanded skill parameters are realized differently when the body is pressed against obstacles than in free space, the decoupling claim collapses. A simpler probe: check whether the passive stiffness λ=1.0 and 120 Hz integration yield jamming or sliding that a torque controller would not produce.","tokens_in":34400,"feed_emoji":"🤖","tokens_out":5692,"duration_ms":58724,"temperature":0.7,"pith_summary":"HumanCLAW is an evaluation framework and benchmark for whether vision-language models (VLMs) can act through a physical body. The paper's claim is that when low-level motor execution is removed from the loop, current frontier VLMs still fail an indoor find-navigate-interact task, and fail specifically because they lose track of their own body: where it is, whether it has reached the goal, and whether it has collided with the world. Across 1,218 episodes in 41 scenes, nine off-the-shelf VLMs top out at 16.8 percent full-task success, with perception (seeing the target) nearly solved. The diagnosis is a missing faculty the paper calls embodied self-awareness, and because motor errors are factored out by construction, the failures are attributed to the decision maker. The framework is built on half-physics simulation, a skill-conditioned motion generator, and a verifier that keeps the decision loop physically grounded.","feed_headline":"Nine AI models fail body-control benchmark; top score: 16.8%","feed_subtitle":"New benchmark strips out motor control and finds the real gap: models lose track of their own body.","key_machinery":"Half-physics simulation is the load-bearing mechanism: the world follows rigid-body physics—gravity, contact, friction, movable objects—while the humanoid body is driven by equivalent kinematic velocities from a skill-conditioned motion generator, with no joint torques, so balance and motor-tracking failures cannot occur. Around it, the harness exposes a fixed set of atomic whole-body skills (walk, turn, side-step, step back, climb, sit, stop) with continuous parameters in a text/JSON interface; a skill-specific verifier rejects unsafe or premature proposals; and a frozen base motion diffusion transformer with per-skill ControlNet adapters realizes each skill as a half-second full-body motio","core_discovery":"The central claim is that today's VLMs reason about the scene but not about the body they control. With balance and motor tracking removed by the half-physics simulator, the measured success rates—best 16.8 percent for full episodes, 42.4 percent for navigation alone—cannot be blamed on locomotion. The paper then traces the failures with an automated root-cause analysis: once a target is rendered in the egocentric view it is almost always recognized, but 68 percent of found targets are never reached, and among those reached, 71 percent of sits fail. The dominant error modes are stopping while still far, walking on after arrival, sitting into empty air, and colliding with visible obstacles wh","pith_inferences":["A testable extension is to feed the VLM a proprioceptive or contact signal; the paper's own discussion predicts this may be the missing input, and if success rises substantially, 'embodied self-awareness' would be a sensor-channel problem as much as a reasoning deficit.","The half-physics premise is unvalidated against a full-dynamics baseline; if kinematic driving distorts contact outcomes (jamming, sliding, unrealistic blocking), some attributed decision failures could actually be simulator artifacts. A direct comparison would settle this.","The same half-physics loop should generalize to manipulation and tool use, where the finest body-placement failures (sitting on air, wrong object) become hand and tool placement failures—an even stiffer test of anticipatory spatial reasoning.","Because the interface omits tactile feedback, the benchmark may underestimate models that could otherwise use contact information; a purely vision-based ceiling may not be the ceiling for the same reasoner with different sensors."],"forward_implications":["Embodied evaluation can now separate decision quality from motor skill, so VLM reasoning can be compared across embodiments without a locomotion confound.","The bottleneck being self-localization and termination suggests proprioceptive or contact feedback, or persistent spatial memory, may be more valuable than larger context windows.","The plug-and-play skill adapters mean new action vocabularies can be added without retraining the decision maker, so benchmark coverage can grow cheaply.","The staged metrics and root-cause labels turn the benchmark into a diagnostic: progress on body awareness should show up as higher navigation and interaction success, not just better target finding.","The verifier ablation (navigation success drops from 27.0% to 2.0% without it) shows that short-context verification, not longer memory, is what closes the loop in this setting."],"supporting_citations":[{"why":"Supplies the half-physics passive-body simulation method that the framework's decoupling relies on.","marker":"(Siyao et al., 2025)"},{"why":"Provides the HSSD scenes used to build the 1,218 episodes across 41 indoor houses.","marker":"(Khanna et al., 2024)"},{"why":"Supplies Habitat, the platform on which the physical simulation environment is built.","marker":"(Savva et al., 2019)"},{"why":"Supplies the Bullet physics engine implementing contact, collision, and gravity.","marker":"(Coumans, 2015)"},{"why":"Provides the receding-horizon motion-continuation base DiT architecture used as the skill-agnostic motion prior.","marker":"(Zhang et al., 2025c)"},{"why":"Provides MoMask, the text-to-motion baseline that demonstrates the skill-fidelity gap.","marker":"(Guo et al., 2024)"}],"fun_headline_variants":["AI models lose track of their own bodies—best score 16.8%","Vision-language models flunk physical tasks: top score 16.8%","New benchmark: AI sees the world but not itself","Embodied AI fails: 68% of targets never reached","Why AI can't sit down: it doesn't know where its body is"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The half-physics simulator removes balance and motor-tracking failures exactly, while preserving contact, collision, gravity, and object displacement without introducing motor artifacts—so every failure can be attributed to the decision maker.","fun_headline_variants_meta":{"raw":{"variants":["AI models lose track of their own bodies—best score 16.8%","Vision-language models flunk physical tasks: top score 16.8%","New benchmark: AI sees the world but not itself","Embodied AI fails: 68% of targets never reached","Why AI can't sit down: it doesn't know where its body is"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1128,"prompt_tokens":807,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":551,"tokens_out":321,"duration_ms":4645,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:21:51.507845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,218 episodes under a fully dynamic torque-controlled humanoid and under pure kinematic playback. If half-physics outcomes diverge from both, or if commanded skill parameters are realized differently when the body is pressed against obstacles than in free space, the decoupling claim collapses. A simpler probe: check whether the passive stiffness λ=1.0 and 120 Hz integration yield jamming or sliding that a torque controller would not produce.","supporting_citations":[],"review_version":3}