{"id":"dc1b5400-c1e0-4b09-bfac-d9f74ca6035b","arxiv_id":"2505.09694","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EWMBench is a benchmark dataset and evaluation toolkit for scoring embodied world models on scene consistency, motion correctness, and semantic alignment, applied to seven video generation models.","lead":"This paper introduces EWMBench, a benchmark that evaluates whether video generation models can act as embodied world models for robot manipulation tasks. It measures scene consistency, motion correctness, and semantic alignment, and ranks seven current video generation models on these criteria.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-alignment claim is statistically unsupported: Section 4.2 reports no annotator count, item count, or agreement metric, so the Figure 6 comparison against VBench could reflect noise rather than genuine alignment.","rationale":"The reader's weakest-assumption identification points to detector reliability in Appendix A.1/A.3.1, which is a real risk: the paper itself notes that Hailuo outputs often look cartoon-like and LTX/COSMOS often render human hands, so a YOLO-World detector fine-tuned on real Agibot-World frames may fail on generated frames, corrupting HSD, nDTW, and DYN. That concern is substantive. However, for the specific strongest claim about alignment with human judgments, the more load-bearing gap is the human evaluation itself: without reported annotator counts, item counts, agreement statistics, or a described VBench scoring procedure, the comparison in Figure 6 cannot be separated from chance. This is a missing-support issue in Section 4.2 rather than an internal inconsistency in the metrics. The paper still has useful components: a task-decomposed dataset, a multi-dimensional evaluation toolkit, and a public repository, so the CONDITIONAL verdict remains appropriate. The headline human-alignment claim, though, should be treated as unverified until the human-study evidence is provided or re-collected.","tokens_in":12399,"tokens_out":8497,"duration_ms":89767,"concrete_test":"Request or re-collect per-item human ratings: at least 5 annotators rating 50 videos per model for the four models in Section 4.2, using a consistent 3/2/1/0 rank scoring; compute inter-annotator agreement (e.g., Fleiss' kappa) and Kendall's tau between (a) the human aggregate and EWMBENCH overall score and (b) the human aggregate and VBench overall score, with bootstrap 95% confidence intervals on the difference. If the confidence interval includes zero, or if kappa is below 0.6, the headline alignment claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4.2, Figure 6) is that EWMBENCH rankings align more closely with human judgments than VBench rankings. The only evidence is a human evaluation described as 'annotators ranked the predictions... assigning 3 points... 2... and 0 to the worst.' No number of annotators, number of rated videos, per-item scores, or inter-annotator agreement is reported; 'multiple review rounds ensuring annotation reliability' is asserted without supporting data. The comparison against VBench is also opaque: no VBench score definitions, no per-model VBench scores, and no test of whether the difference in rank agreement is statistically significant. With only four models, random variation in either the human or the VBench ranking can change the conclusion. The claim is therefore not yet established, independent of the automated metric details.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"EWMBench is a proposed benchmark for embodied world models (EWMs), evaluating image/text-conditioned video generation on three dimensions: visual scene consistency, motion correctness (via end-effector trajectory metrics HSD, nDTW, and DYN), and semantic alignment (via MLLM captioning, CLIP scores, and logic-error penalties). The dataset is curated from Agibot-World across ten manipulation tasks with diversity-based trajectory sampling, and seven video generation models are compared. The paper claims that EWMBench rankings align more closely with human judgments than VBench rankings (Section 4.2) and provides qualitative analyses of model failure modes. The dataset and evaluation tools are publicly available.","tokens_in":12468,"tokens_out":7083,"duration_ms":62233,"significance":"The benchmark addresses a genuine gap: existing video generation benchmarks focus on perceptual quality and lack embodied task grounding. The decomposition into scene/motion/semantics, the publicly released evaluation toolkit, and the controlled experiment showing complementary behavior of HSD, nDTW, and DYN (Figure 6C) are useful contributions. The qualitative failure-mode analysis (e.g., human-hand bias in COSMOS/LTX, viewpoint drift in COSMOS) is also informative. However, the central human-alignment claim and the reliability of the automated trajectory/scene detectors are not yet established; several methodological choices (best-of-three selection, additive overall score, arbitrary DYN weights) affect the benchmark's validity as currently presented.","major_comments":[{"comment":"Best-of-three selection is circular for HSD. In Section 3.3, the protocol generates three candidate trajectories per sample and selects 'the best trajectory... based on Hausdorff distance.' That same HSD value is then reported in Table 2. As a result, the reported HSD is the minimum over three draws of a distance (equivalently the maximum of the reciprocal score), which inflates scores and disproportionately benefits models with high generation variance. Because the Overall column in Table 2 includes HSD, the ranking is directly affected. Please report all candidates or, at minimum, use a selection criterion independent of the reported metric and justify the choice.","section":"3.3, Table 2"},{"comment":"The Overall score is an unweighted arithmetic sum of components with different ranges. SceneC contributes at most 1, the motion 'Avg' is actually the sum HSD+Dyn+nDTW (at most 3), and the semantic 'Avg' is the sum Diversity+BLEU+CLIP+Logics (at most 4). Thus the overall ranking implicitly weights semantics four times and motion three times relative to scene, and the labels 'Avg.' are misleading. For example, EnerVerse_FT's lead over Kling is driven substantially by the larger number of semantic sub-scores. Please report a stated aggregation rule (e.g., average within each dimension, then a declared weighted combination) and show sensitivity to the aggregation.","section":"Table 2"},{"comment":"The DYN score in Eq. (7) depends on two constants alpha=0.007 and beta=0.003 with no stated justification. These weights determine the relative contribution of velocity and acceleration, and the resulting DYN magnitudes feed into the Overall score in Table 2. Since the paper argues that DYN captures a distinct failure mode, the constant choice should be supported (e.g., calibrated on a labeled set or reported with a sensitivity analysis) rather than asserted.","section":"A.3.2, Eq. (7)"},{"comment":"The human-alignment claim is not statistically supported. Section 4.2 reports no number of annotators, no number of rated videos, no per-item scores, and no inter-annotator agreement; it only states that 'multiple review rounds' ensured reliability. The comparison with VBench in Figure 6(B) lacks the per-model VBench scores and any test of whether the difference in ranking agreement is significant. With only four models, random variation in either the human or VBench ranking can change the conclusion. Please provide the full evaluation protocol, per-item data, and a significance test (e.g., Kendall's tau with bootstrap).","section":"4.2, Figure 6"},{"comment":"The trajectory and scene metrics depend on detectors whose reliability on generated videos is unmeasured. Trajectory extraction uses a YOLO-World model fine-tuned on 1,451 annotated Agibot-World frames, and scene consistency uses DINOv2 fine-tuned on Agibot-World. Generated videos can contain human hands instead of robot end-effectors (as the paper itself notes for COSMOS and LTX in Appendix A.4), cartoon-like scenes (Hailuo), or viewpoint drift (COSMOS). No detection accuracy is reported on generated frames, so HSD, nDTW, and DYN may be unreliable for exactly the models the benchmark is discriminating. A manual audit of trajectories on a sample of generated videos, or a report of detector confidence on the generated corpus, is needed.","section":"A.1, 3.3"}],"minor_comments":[{"comment":"The number of benchmark episodes is inconsistent: the introduction says '30 candidate samples across ten tasks' while Section 4 says ten tasks with ten ground-truth episodes each, and Appendix A.2.2 says 100 instances per task with 10 selected. Please clarify the exact dataset size.","section":"1, 4, A.2.2"},{"comment":"In Table 2, the column headers are ambiguous: the 'Avg.' columns are sums, and the Overall column is a sum of sums; a clearer breakdown with per-dimension averages would aid reproducibility.","section":"Table 2"},{"comment":"The scoring scheme assigns 3, 2, and 0 to best, second-best, and worst, with no value for third-best; please clarify whether this is deliberate and report the exact scoring rubric.","section":"4.2"},{"comment":"The global semantic alignment score uses BLEU to compare an MLLM-generated caption to the raw task instruction; BLEU is sensitive to lexical overlap and may be a poor measure for this open-ended comparison. Please report the actual BLEU values or motivate the choice.","section":"3.3, A.3.2"},{"comment":"Appendix A.1 reports precision/recall of the fine-tuned YOLO-World on held-out Agibot-World tasks, but not on generated videos; if detector performance on generated data is available, add it here.","section":"A.1"}],"recommendation":"major_revision","confidential_remarks":"No concerns about novelty or citation practices. The main issue is methodological validation; the paper would fit the journal after the proposed revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: EWMBench is a useful early step toward task-specific evaluation of embodied world models. It contributes a curated 10-task subset of Agibot-World with step-level annotations and a three-axis scoring protocol (scene, motion, semantics) that is more appropriate for manipulation than general video benchmarks. The core idea is sound, but the paper's central validation claim—that EWMBENCH's rankings align better with human judgment than VBench—is not yet supported by the reported evidence, and the metric aggregation has self-referential softness.\n\nWhat's new and what works: the dataset construction is the real contribution. Selecting tasks with action-ordering constraints, decomposing them into atomic sub-actions with step captions, and using a greedy voxel-IoU trajectory sampler to diversify motion patterns is solid, reproducible work. The evaluation toolkit combines fine-tuned DINOv2 for scene consistency, YOLO-World plus BoT-SORT for trajectory extraction, nDTW/HSD/DYN for motion, and MLLM prompts for semantic alignment and logical-error penalties. Most components are known, but the embodied-specific combination and the open release give the community a usable harness. The qualitative failure-mode analysis—empty grasps, human-hand bias, viewpoint drift—matches what the metrics flag, which lends face validity to the dimensions.\n\nSoft spots, in order of severity:\n\n1. The human-alignment claim (Section 4.2, Figure 6) does not stand on the reported data. No annotator count, no number of rated videos, no agreement statistic; the VBench comparison lacks score definitions and any significance test. With only four models, random variation can flip the conclusion. This is the headline claim, so it matters.\n\n2. Best-of-three selection uses the same HSD metric that is later reported as a motion score. Selecting on the evaluation metric biases the reported score upward. The protocol should either use a different selection criterion or report all candidates.\n\n3. The overall ranking is a raw sum of sub-scores with different scales and arbitrary weights (e.g., DYN alpha/beta). Table 2 ranks by a number that depends on these choices. Per-dimension reporting or principled aggregation is needed.\n\n4. Sample-size statements are inconsistent: the intro says 30 candidate samples, the experiments say 10 tasks x 10 episodes x 3 videos = 2,100 videos, and Appendix A.2.2 says 100 video instances per task. The actual evaluation set needs to be stated precisely.\n\nMinor: the \"first\" claim deserves qualification against EVA, which also covers trajectory and scene dimensions.\n\nWho this is for: researchers building or evaluating embodied world models. It deserves a serious referee—the dataset and protocol are worth the field's attention—but the current version needs major revision before its rankings are used as ground truth. I would send it to review and ask for the human-eval statistics, a selection-protocol fix, and clean aggregation. If those are addressed, this could become a standard harness.","headline":"Useful early benchmark for embodied world models, but the human-alignment claim outruns the evidence and the metric aggregation needs a principled rework before the scores are used as ground truth.","tokens_in":13078,"tokens_out":3115,"would_cite":true,"duration_ms":29343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EWMBench ranks embodied video models closer to human judgments than VBench does.","keywords":["embodied world models","video generation evaluation","robotic manipulation","trajectory metrics","scene consistency","semantic alignment","human evaluation","benchmark dataset"],"falsifier":"Compute EWMBench and human rankings on a held-out set of generated videos that contain human hands instead of robot arms. If the benchmark ranks models higher than humans do on those cases, the detector assumption fails and the human-alignment claim does not generalize.","tokens_in":12139,"feed_emoji":"🤖","tokens_out":5408,"duration_ms":50180,"temperature":0.7,"pith_summary":"This paper argues that judging embodied world models—video generators asked to produce the next frames of a robotic manipulation—requires metrics for visual scene consistency, motion correctness, and semantic alignment, not just general video quality or human preference. It introduces EWMBench, a benchmark built on real manipulation data, with 30 curated episodes across 10 tasks, automated trajectory and scene detectors, and a prompt suite that makes a video MLLM caption, break down steps, and penalize logic errors. The paper's central claim is that EWMBench's rankings agree with human annotators more closely than VBench's rankings do, and that domain-adapted models (EnerVerse_FT, LTX_FT) outperform commercial and open-source generators on these embodied criteria. If the claim holds, the benchmark gives embodied-AI developers a way to measure progress on physically grounded, action-consistent generation rather than on visual polish alone.","feed_headline":"Embodied-video benchmark beats VBench on matching human judgment","feed_subtitle":"Domain-adapted models EnerVerse and LTX_FT top scene, motion, and semantic scores for robot manipulations.","key_machinery":"The machinery has three parts. Fine-tuned YOLO-World detects the end-effector in every frame and BoT-SORT links detections into trajectories; the largest-spatial-extent hand's trajectory is compared with ground truth. The trajectory metrics are Symmetric Hausdorff Distance (maximum spatial deviation), Normalized Dynamic Time Warping (spatial-temporal alignment and task sequencing), and Dynamic Consistency (Wasserstein distance between velocity and acceleration distributions, amplitude-normalized). Scene consistency uses a DINOv2 ViT-B/14 fine-tuned for 20,000 iterations on embodied data, computing cosine similarity between patch embeddings of initial and consecutive frames. Semantic evaluation uses a video MLLM with three prompt levels—global caption compared by BLEU to the instruction, step descriptions compared by CLIP to ground-truth sub-action captions, and logic-error penalties for hallucinations and impossible spatial relations. The benchmark protocol takes initial frames, language instruction, and an optional action trajectory, normalizes outputs to 640x480 at 30 FPS, and requires three generated videos per episode with best-of-three selection.","core_discovery":"On its own terms, the paper establishes EWMBench as a benchmark that scores generated manipulation videos along three axes: scene consistency (static layout, object permanence, viewpoint coherence via fine-tuned DINOv2 cosine similarity), motion correctness (end-effector trajectory compared to ground truth with Hausdorff distance, normalized dynamic time warping, and Wasserstein-based dynamic consistency), and semantic alignment (global captions matched by BLEU, step descriptions by CLIP, plus explicit logic-error penalties). In the paper's experiments, EWMBench ranks the domain-adapted EnerVerse_FT and LTX_FT above commercial models like Kling and Hailuo and open-source models like COSMOS, OpenSora, and LTX, with EnerVerse_FT and Kling showing the strongest semantic alignment. The paper reports that in a human evaluation over four representative models, EWMBench's ranking matched the aggregated human ranking more closely than VBench's ranking did. It also shows that the three trajectory metrics respond differently to sequence reversal, outlier insertion, and frame repetition, which the paper reads as evidence that all three are needed.","pith_inferences":["A natural extension, not pursued in the paper, would be to run the same human study on all seven models, not just four; if the ranking agreement persists, the case for EWMBench over VBench strengthens.","Because EWMBench's motion scores depend on detecting a single end-effector, models that render human hands instead of robot arms could receive inflated trajectory scores if the detector tracks the hand; a targeted audit of hand-vs-arm cases would clarify whether the metric measures task correctness or merely object tracking.","The logic-error penalty in the MLLM prompt suite could be turned into a standalone audit protocol for physical-commonsense violations in generated manipulation videos, independent of trajectory extraction.","The trajectory-diversity sampling with voxelized IoU and greedy selection suggests a recipe for building harder evaluation splits: choose episodes whose ground-truth trajectories are least similar, which may better expose models that only reproduce average motion."],"forward_implications":["The benchmark's ranking of domain-adapted models above commercial and open-source ones supports the paper's conclusion that domain-specific fine-tuning improves embodied generation quality.","Because the trajectory metrics diverge on reversed, outlier, and repeated sequences, measuring embodied motion requires all three rather than any single distance.","EWMBench's closer agreement with human rankings over VBench indicates that general benchmarks' foreground-background entanglement is a measurable limitation for embodied evaluation.","The benchmark currently restricts itself to end-effector trajectories and fixed viewpoints, scoping its conclusions to that setting."],"supporting_citations":[{"why":"Serves as the baseline benchmark whose ranking agreement with human judgment is compared against EWMBench.","marker":"[18]"},{"why":"Supplies the Agibot-World dataset that provides the ten manipulation tasks and ground-truth trajectories.","marker":"[2]"},{"why":"EnerVerse is the domain-adapted model that receives the highest overall score in the benchmark.","marker":"[17]"},{"why":"YOLO-World is the detector fine-tuned to extract end-effector trajectories from generated frames.","marker":"[9]"},{"why":"DINOv2 is the visual backbone fine-tuned on embodied data for the Scene Consistency metric.","marker":"[28]"},{"why":"Provides dynamic time warping, the basis of the Normalized DTW trajectory metric.","marker":"[27]"},{"why":"Defines the Symmetric Hausdorff Distance used as the spatial-alignment metric.","marker":"[32]"},{"why":"Provides the Wasserstein distance used by the Dynamic Consistency metric for velocity and acceleration.","marker":"[36]"},{"why":"COSMOS is an evaluated open-source model that the paper contrasts with domain-adapted models.","marker":"[1]"},{"why":"LTX is an evaluated open-source model, and LTX_FT is its fine-tuned variant ranked second overall.","marker":"[13]"}],"fun_headline_variants":["EWMBench ranks embodied video models beyond VBench in human agreement","New benchmark scores robot-video scene, motion, and semantics","EWMBench beats VBench on matching human quality judgments","EWMBench: sharper read on embodied video quality than VBench"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The end-effector tracker and the scene-consistency feature extractor, both fine-tuned on the Agibot-World dataset, reliably measure the true task-relevant motion and scene structure in videos that those models were not trained on, including generated frames that show human hands, cartoon-like scenes, or drifting viewpoints.","fun_headline_variants_meta":{"raw":{"variants":["EWMBench ranks embodied video models beyond VBench in human agreement","New benchmark scores robot-video scene, motion, and semantics","EWMBench beats VBench on matching human quality judgments","EWMBench: sharper read on embodied video quality than VBench"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000441,"raw_usage":{"total_tokens":2245,"prompt_tokens":961,"completion_tokens":1284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1212}},"tokens_in":577,"tokens_out":1284,"duration_ms":8571,"temperature":1.0,"reasoning_tokens":1212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:26:53.913368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute EWMBench and human rankings on a held-out set of generated videos that contain human hands instead of robot arms. If the benchmark ranks models higher than humans do on those cases, the detector assumption fails and the human-alignment claim does not generalize.","supporting_citations":[{"cited_title":"Vbench: Comprehensive benchmark suite for video generative models","cited_arxiv_id":null,"evidence_quote":"Serves as the baseline benchmark whose ranking agreement with human judgment is compared against EWMBench."},{"cited_title":"Agibot world.https://agibot-world.com, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the Agibot-World dataset that provides the ten manipulation tasks and ground-truth trajectories."},{"cited_title":"Hausdorff distances and interpolations.Computational Imaging and Vision, 12:107–114, 1998","cited_arxiv_id":null,"evidence_quote":"Defines the Symmetric Hausdorff Distance used as the spatial-alignment metric."},{"cited_title":"The wasserstein distances.Optimal transport: old and new, pages 93–111, 2009","cited_arxiv_id":null,"evidence_quote":"Provides the Wasserstein distance used by the Dynamic Consistency metric for velocity and acceleration."}],"review_version":1}