{"id":"97c73e24-11aa-4d14-8c09-411b579db3d8","arxiv_id":"2608.13049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"H2R-Bench is a benchmark that evaluates whether video generation models can transfer human manipulation demonstrations into robot videos with correct embodiment, contact, and task completion, and it finds current models largely fail.","lead":"H2R-Bench turns human demonstration videos into test cases for robot video generation, checking whether AI video models can replace a human actor with a robot gripper without breaking the task. Eleven current video models were tested, and most failed to keep the embodiment, contact, and task execution consistent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"M3/M4 scores may be artifacts of the 25-frame MLLM protocol; no ground-truth contact/embodiment check currently exists.","rationale":"The reader's weakest_assumption identifies the reliability of MLLM judges for contact and embodiment from 25 frames, and that is also the single most load-bearing concern here. The paper's central claim about model limitations and the quality-transfer decoupling depends on M3 and M4 being valid diagnostics. The reported human-MLLM agreement is encouraging, but it uses the same rubric and does not validate the construct against any independent ground truth or against denser temporal evidence. A direct frame-budget and full-video sensitivity study would settle whether the sparse sampling protocol materially changes scores and rankings. The native-interface confound and Qwen-as-judge circularity are real but secondary: they affect attribution of failures to models versus interfaces or to a single annotator family, whereas the frame-budget issue threatens the validity of the transfer metrics themselves. The reader's CONDITIONAL verdict is appropriate; no verdict change is needed, but the proposed check should be a condition for treating H2R-Bench as a settled standard.","tokens_in":28626,"tokens_out":5951,"duration_ms":65240,"concrete_test":"Take a stratified sample of 60 generated videos covering both target embodiments and the range of model tiers, and have the same three MLLM judges score M3 and M4 twice: once with the standard 25 uniformly sampled frames and once with the full video (or 100 frames). Also collect human scores on the same full videos. If the 25-frame M3/M4 scores are within 0.05 in mean and preserve the full-video model ranking (Kendall tau greater than 0.9), the sparse-frame protocol is adequate. If not, the benchmark's central findings must be re-derived with a denser frame budget before the conclusion about model limitations is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that current video world models 'often fail in embodiment consistency, functional interaction, and task execution' rests on M3 (functional contact transfer) and M4 (embodiment correctness), which are scored by three MLLMs from 25 uniformly sampled frames (Section B.6). This is load-bearing because a 5-second clip sampled at roughly 5 fps can miss brief contact onsets, releases, and end-effector swaps between frames; M3's 'temporally supported object response' is explicitly visual-temporal evidence rather than physical verification; and M4's end-effector correctness can be ambiguous under motion blur, occlusion, or small gripper details. The human validation in Section B.7 uses the same rubric and does not independently establish that visible-contact or embodiment judgments correspond to actual functional contact or robotic morphology. If M3/M4 are miscalibrated or insensitive, the headline finding of weak quality-transfer association (Spearman rho = 0.14) and the conclusion that even leading models fail could reflect judge artifacts rather than model capability. The benchmark's stated scope is visible evidence, so the concern is not that physical executability is unmeasured; it is that the visible evidence itself may be inadequately sampled and verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"H2R-Bench introduces a benchmark for evaluating human-to-robot manipulation video generation: given an egocentric human demonstration, a target embodiment (parallel-jaw gripper or dexterous hand), and a text prompt, a video generator must produce a robot manipulation video that preserves the source task goal, required action events, functional contact, and object responses while realizing the execution with the requested robot morphology. The benchmark is built from 120 EgoDex test clips, paired with two embodiments to yield 240 transfer cases across six manipulation families. Eleven video generators are evaluated with their native source-conditioning interfaces. Five metrics are defined: M1 goal-state completion, M2 action-event completion, M3 functional contact transfer, M4 embodiment correctness, and M5 task-agnostic video quality. M1-M4 are scored by three MLLM judges (Gemini 3.5 Flash, Qwen3.7-Plus, GPT-5.4) from 25 uniformly sampled frames; M5 combines MUSIQ, a CLIP-based aesthetic predictor, adjacent-frame stability, and AMT-S interpolation consistency. The aggregate H2RCore weights contact and embodiment at 0.30 each, goal and action at 0.15 each, and quality at 0.10. The main findings are that video-conditioned models (Seedance 2.0, Wan2.7, Kling-V3) rank highest, that M5 scores lie in a narrow range while H2RCore spans 30.0-84.6, and that the rank correlation between M5 and H2RCore is weak (Spearman rho = 0.14).","tokens_in":28790,"tokens_out":5522,"duration_ms":58907,"significance":"If the benchmark measurements are valid, H2R-Bench fills a real gap: existing video-generation benchmarks do not evaluate source-relative, cross-embodiment transfer from human demonstrations to robot embodiments. The paper's diagnostic decomposition into goal, action, contact, embodiment, and quality is well motivated, and the empirical claim that generic video quality is nearly decoupled from transfer quality (rho = 0.14) is important and falsifiable. The authors are methodologically careful in several ways: all annotations are manually verified against source videos; the evidence budget for M1-M4 is fixed across models; paired bootstrap confidence intervals are reported for the leading model comparisons; weight sensitivity is checked; and human validation is conducted on a substantial sample of 660 videos. The limitations section is unusually candid, acknowledging that the native-interface protocol confounds model capability with conditioning interface and that MLLM judgments from sampled frames can be uncertain.","major_comments":[{"comment":"","section":"§B.6 (M3/M4) and §D"},{"comment":"","section":"§A.2 and §B.6"},{"comment":"","section":"Experiments, Native-Interface Protocol, and §D"}],"minor_comments":[{"comment":"","section":"Figure 4"},{"comment":"","section":"§B.5 and Figure 4"},{"comment":"","section":"§B.2"},{"comment":"","section":"Table S7"},{"comment":"","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"This is a solid benchmark paper with unusually honest limitations. The main risk is not the benchmark's usefulness but the construct validity of the two most heavily weighted metrics: M3 and M4 are scored by MLLMs from 25 frames, and the human validation uses the same rubric and evidence. I also take the Qwen annotation-judge overlap seriously, though it may be mitigated by human annotation verification and three-judge averaging. If the authors add a dense-frame validation subset and a judge-ablation, I would be willing to support acceptance. The native-interface confound is real but partly acknowledged; tightening the claims would be sufficient if the cross-interface ablation is not expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"H2R-Bench is worth engaging. It gives the video-world-model community a source-relative H2R transfer evaluation that separates goal, action, contact, and embodiment, and the headline empirical finding—generic video quality and transfer quality barely correlate (Spearman rho = 0.14)—is new and useful. The paper does the right methodological work: bootstrap confidence intervals, paired comparisons, weight sensitivity, three human raters on 660 videos, and inter-judge agreement. The limitations section is unusually honest about the interface confound and the visible-evidence scope.\n\nSoft spots, in proportion. The native-interface protocol confounds model capability with conditioning interface; the authors acknowledge this, and the matched Seedance ablation (video vs nine images) shows the interface matters. Readers should not interpret the video-conditioned vs frame-conditioned gap as pure model capability. The MLLM-judged M3/M4 are the main suspect. Could 25 uniformly sampled frames miss brief contact onsets or subtle embodiment violations? Yes, in principle. But the human raters saw the full clips, and their M1–M4 scores correlate r = 0.88–0.93 with the MLLM scores, which suggests the 25-frame budget preserves most of the evidence humans rely on. The caveat is that the human rubric is the same visible-evidence rubric, so this is not an independent check on physical contact or true morphology. The paper explicitly limits itself to visible evidence, so I do not count this as a fatal flaw—just a limitation to remember when citing.\n\nThe Qwen annotation-then-judge overlap is a mild circularity, mitigated by manual verification and three-judge averaging. The missing artifact links (code, data, annotations, commit hash) are the most easily fixable defect.\n\nBottom line: a solid benchmark contribution, carefully reported, and a clear yes on peer review. I would ask the authors to release the full artifacts and add a small independent check of M3 on a subset with full-video human raters using an explicit contact-presence annotation, but I would not block on it. Treat the scores as measuring visible H2R transfer, which is a legitimate and useful target.","headline":"A well-built benchmark for the source-relative H2R transfer setting with an honest, carefully analyzed empirical finding; the MLLM-judged contact and embodiment metrics are the main soft spot but not fatal.","tokens_in":29397,"tokens_out":4818,"would_cite":true,"duration_ms":47567,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark shows that video world models which produce polished videos still fail to transfer human manipulation demonstrations to specified robot embodiments.","keywords":["human-to-robot video transfer","video world models","manipulation benchmark","cross-embodiment generation","functional contact transfer","embodiment correctness","video generation evaluation","robot learning from human videos"],"falsifier":"Have a physical robot or a contact-physics simulator attempt the manipulations in the videos that the benchmark scores highest on contact and embodiment, and check whether the object state actually follows from the visible contact; if high-scoring videos consistently fail physical execution, the benchmark's transfer scores are measuring appearance rather than functional transfer.","tokens_in":28392,"feed_emoji":"🤖","tokens_out":11060,"duration_ms":96083,"temperature":0.7,"pith_summary":"Video world models are generators that simulate how a scene evolves; the hope is that they can turn abundant egocentric human manipulation videos into robot demonstrations. This paper introduces H2R-Bench, a benchmark that tests whether eleven such models can actually do the transfer: given a human demo and a target robot embodiment, the model must produce a video that preserves the task goal, the required action events, the functional contact between end-effector and object, and the requested robot morphology. The benchmark scores 240 cases built from 120 egocentric clips across six manipulation families and two embodiments, and reports that current models are far from reliable: the best H2RCore is 84.6 out of 100, several models sit near 30, and the task-agnostic video-quality score is almost uncorrelated with transfer quality (Spearman ρ = 0.14). If the benchmark is right, visual polish is not evidence that a generated robot video would be usable as robot training data.","feed_headline":"Video quality doesn't predict human-to-robot transfer","feed_subtitle":"New scoring of 11 video generators shows goal, contact, and embodiment are what separate robots from humans in the output.","key_machinery":"The load-bearing mechanism is the source-relative scoring protocol wrapped around the H2RCore aggregate. The benchmark defines each case as a triple of human source video, target embodiment, and generation prompt, and keeps evaluation annotations separate from what the generator sees. Three AI vision-language judges independently score M1–M4 on 25 uniformly sampled frames against the source-derived annotation, with M3 comparing source and generated frames for functional contact and M4 isolating robot presence, absence of human hands, embodiment category, end-effector correctness, and structural consistency. The aggregate $H2RCore = 100(0.15 S_{goal} + 0.15 S_{action} + 0.30 S_{contact} + 0.30 S_{emb} + 0.10 S_{video})$ gives contact and embodiment 60% of the weight, which is what separates visually plausible videos from transfer-valid ones. M5 measures imaging quality, aesthetics, temporal stability, and motion smoothness without reference to the task.","core_discovery":"The central claim is that source-relative evaluation is required to judge human-to-robot transfer, and under that evaluation current video world models largely fail. Each test case specifies a human source video, a target embodiment (parallel-jaw gripper or dexterous hand), and a text prompt; the source video is treated as evidence of what happened, not a style reference. The benchmark annotates each source with weighted final-state predicates, required action events, functional-contact specifications, and embodiment constraints, then scores generated videos with five metrics: goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and task-agnostic video quality. On the 240-case main evaluation, the three full-video-conditioned models lead with H2RCore between 74.5 and 84.6, frame-conditioned models cluster near 30–61, and actions or final states are often preserved even when contact and embodiment are wrong. The paper reads this as showing that task recognition and visual polish are not enough: embodiment consistency, functional interaction, and task execution remain unsolved for current video world models.","pith_inferences":["Editorial inference: the near-zero correlation between visual quality and H2RCore implies that pipelines filtering generated robot videos by aesthetics or text-video alignment are selecting on the wrong signal; a cheap transfer-aware check such as the M3/M4 rubric would retain more usable training data.","Editorial inference: because M3 judges visible evidence and does not claim to verify physical causation, the benchmark likely measures apparent contact, not verified contact; a natural extension is to feed high-scoring generations into a contact-physics simulator or real gripper and test whether the object state follows from the visible interaction.","Editorial inference: the consistent dexterous-hand advantage suggests current generators transfer more readily to embodiments that resemble the human hand; this predicts that adding a humanoid hand or a soft gripper embodiment would systematically narrow the gap, which a larger embodiment set could test."],"forward_implications":["Source-conditioning interface matters: the three full-video-conditioned models occupy the top of the leaderboard, while frame-conditioned models often leave the human as the active manipulator or replace the end effector incorrectly.","Visual quality and transfer quality are decoupled: quality scores range only 0.73–0.81 while H2RCore spans 30.0–84.6, with rank correlation ρ = 0.14, so a model can look polished and still fail transfer.","The requested embodiment changes difficulty: all eleven models score higher on contact transfer for the dexterous hand, and nine of eleven score higher overall on H2RCore for the hand than for the parallel-jaw gripper.","A target-robot reference image is not a reliable fix: it substantially helps one model with the gripper while hurting two others, so embodiment grounding is model-dependent.","Current video world models are not yet dependable generators of robot-centric training data from human demonstrations, because leading models still fail on at least one of contact, embodiment, or task execution across the six manipulation families."],"supporting_citations":[{"why":"Supplies the 120 EgoDex egocentric source clips that define the transfer cases.","marker":"Hoque et al. 2025"},{"why":"Provides VBench, the video-quality reference shown to be nearly uncorrelated with H2RCore.","marker":"Huang et al. 2024"},{"why":"Provides WorldModelBench, the existing world-model evaluation that inspects plausibility but not H2R transfer.","marker":"Li et al. 2026a"},{"why":"Provides Mitty-EPIC14B, a frame-conditioned human-to-robot generation baseline evaluated in the benchmark.","marker":"Song et al. 2025"},{"why":"Supplies MUSIQ, the imaging-quality model used in the M5 video-quality component.","marker":"Ke et al. 2021"},{"why":"Supplies AMT-S, the frame-interpolation model used to measure motion smoothness in M5.","marker":"Li et al. 2023"},{"why":"Supplies CLIP features used by the LAION aesthetic predictor in M5.","marker":"Radford et al. 2021"}],"fun_headline_variants":["Video polish hides robot transfer failure","H2R-Bench: World models flunk robot transfer","Human video to robot: quality isn't enough","Embodiment and contact trip video world models","Source-relative scoring exposes transfer gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions depend on three AI vision-language judges being able to tell, from only 25 sampled frames of each video, whether a robot really made functional contact and whether the embodiment is correct.","fun_headline_variants_meta":{"raw":{"variants":["Video polish hides robot transfer failure","H2R-Bench: World models flunk robot transfer","Human video to robot: quality isn't enough","Embodiment and contact trip video world models","Source-relative scoring exposes transfer gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1608,"prompt_tokens":1022,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":638,"tokens_out":586,"duration_ms":5593,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:50:41.642037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a physical robot or a contact-physics simulator attempt the manipulations in the videos that the benchmark scores highest on contact and embodiment, and check whether the object state actually follows from the visible contact; if high-scoring videos consistently fail physical execution, the benchmark's transfer scores are measuring appearance rather than functional transfer.","supporting_citations":[],"review_version":1}