Pith. sign in

REVIEW 4 major objections 6 minor 68 references

WorldScape Policy 2.0 reaches 94.3% average success in bimanual manipulation by layering event-level memory over short-term visual memory.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 14:09 UTC pith:MSBUMKDP

load-bearing objection A real engineering advance in WAM memory and event-grounded pretraining, but the headline RoboTwin result is not established — the gap to the runner-up is likely within noise and the comparison is not data-matched. the 4 major comments →

arxiv 2607.18840 v1 pith:MSBUMKDP submitted 2026-07-21 cs.RO

WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory

classification cs.RO
keywords world action modelrobot manipulationlong-horizon planningmemory-augmented policyevent-grounded pretrainingmultimodal robot controldual-arm manipulationvision-language-action model
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

WorldScape Policy 2.0 argues that long-horizon robot manipulation fails without two complementary forms of temporal context: semantic event memory, which tracks what subgoals are done and what comes next, and frame-level visual memory, which preserves contact and motion details. The paper introduces a single world action model that couples these memories, so the same backbone can plan autonomously from a high-level instruction, follow fine-grained subtask captions, and adapt from a goal image or a demonstration video. To train it, the authors assemble ManipEvent-5M, nearly five million event segments with aligned actions, captions, goal images, and videos, and a three-stage curriculum that ends with semantic forcing tying latent planning tokens to explicit subtask semantics. If the claims hold, generalist robot policies gain progress-awareness and multimodal control without an external planner.

Core claim

The central claim: a world action model that jointly predicts future video and robot actions becomes controllable and long-horizon capable when history is organized at two levels—a causal buffer of recent visual latents and an event memory of VLM outputs stored as global-history, local-active, and event-boundary views. Gated retrieval fuses that memory into current reasoning tokens; a semantic-forcing loss aligns them with fine-grained caption embeddings, transferring event semantics into autonomous planning. On RoboTwin 2.0 it reports 94.3% average success (94.3% clean, 94.2% randomized) and 47.9% on the clean-to-randomized protocol; real-world dual-arm tasks reach 75–80% on folding, cleani

What carries the argument

The load-bearing mechanism is the reasoning-augmented long short-term memory: a causal short-term visual memory feeds recent observation latents into the diffusion transformer as prefill, while a long short-term event memory organizes historical vision-language-model outputs into global-history, local-active, and event-boundary views, retrieved by learned gated attention into current perception and planning tokens. These memory-enhanced tokens form an implicit subgoal condition, and a semantic-forcing loss aligns them with fine-grained caption embeddings, transferring explicit event semantics into autonomous planning. ManipEvent-5M supplies the captions, goal images, and video prompts that m

Load-bearing premise

The load-bearing premise is that ManipEvent-5M's automatically generated event boundaries and fine-grained captions are accurate enough to supervise fine-grained grounding and semantic forcing; the paper reports no human agreement or error rate, only that outputs were parsed and validated as structured records.

What would settle it

Shuffle or delay the event boundaries in ManipEvent-5M by a few seconds for a matched pretraining run, or replace fine-grained captions with the episode-level instruction; if success on RoboTwin 2.0 barely moves, then event-level semantic supervision is not the active ingredient. A reader could also compare the model trained on the automatic labels against one trained on a small human-verified subset of the same episodes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported numbers are representative, memory-grounded world action models can replace the separate planner-plus-executor stacks used in many long-horizon systems.
  • The negligible clean-to-randomized gap (94.3% vs 94.2%) implies the design generalizes to visual and physical perturbations when trained on both, while the 47.9% clean-to-randomized score shows a meaningful gain even without seeing randomized data.
  • Unified multimodal prompting—text, goal image, video—becomes a single interface, so the same backbone can be steered at different granularities depending on what the user supplies.
  • Since ablations show monotonic gains from short-term memory, event memory, and latent reasoning, the paper's design claims each component is load-bearing rather than decorative.
  • The dataset contributes a reusable asset: nearly five million event-segmented manipulation trajectories with aligned captions and visual prompts, which other world-model and vision-language-action training pipelines could adopt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension: apply the same reasoning-augmented memory to a non-world-model vision-language-action policy; if gains persist, the benefit is the memory structure, not joint video-action pretraining.
  • A stronger test of the causal story would be a human-verified subset of ManipEvent-5M; if caption noise is removed and performance does not improve, then semantic forcing may be learning from scale rather than from event semantics.
  • The event-boundary detection via latent cosine change suggests a generic progress-tracking prior that could transfer to other long-horizon domains, such as navigation or assembly, where subtask boundaries are not annotated.
  • The paper does not report breakdowns by data source; a natural follow-up is to measure how much of the gain comes from the egocentric human-video portion versus real robot data, which would speak to the data-efficiency of the approach.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes WorldScape Policy 2.0, a world action model that couples a causal short-term visual memory (recent frames as DiT prefill) with a long-term event memory built from VLM outputs (global-history, local-active, and event-boundary views) and an autoregressively generated latent subgoal condition. A semantic-forcing loss aligns the latent planning tokens with fine-grained event captions. To support training, the authors construct ManipEvent-5M, an event-grounded dataset of about 4.89M segments with hierarchical captions, goal images, and video prompts, and use a three-stage curriculum. The paper reports 94.3% average success on RoboTwin 2.0, monotonic gains in component and curriculum ablations, and real-robot results on dual-arm tasks including folding, table cleaning, shell game, and cross-embodiment stacking.

Significance. If the claims hold, the paper makes a useful step toward long-horizon, steerable WAMs: the combination of frame-level and event-level memory within a single WAM is well motivated, and the real-robot evaluation spans four capabilities. The constructed ManipEvent-5M dataset and the three-stage training recipe are potentially valuable contributions. The internal ablations (Tables 5 and 6) are a strength: they show monotonic improvements when memory stages and semantic forcing are added. However, the headline comparative claims are not established as stated: the main RoboTwin table is not data-matched, no variance is reported, the C2R comparison omits the closest baselines, and the automatically generated event labels are not validated against human judgments. These issues are fixable and do not invalidate the approach, but they must be addressed before the central claims can be accepted.

major comments (4)
  1. [§4.3, Table 2] The main comparison is not data-matched. WorldScape Policy 2.0 is pretrained on ManipEvent-5M (§3.5, Table 1, ~4.89M segments) and initialized from Wan2.2-5B, while the baselines are described only as fine-tuned for 50K steps on clean-plus-randomized data. Reported advantages over Abot-M0.5 (94.3% vs 94.1%) and LingBot-VA 2.0 (94.3% vs 93.6%) are within a range that can easily be sampling noise; no error bars, confidence intervals, or significance tests are provided anywhere. This prevents attributing the gains to the proposed memory and semantic-forcing mechanisms. Please report variance estimates and, if possible, data-matched baselines (e.g., the same backbone trained with and without ManipEvent-5M, or with episode-level captions only).
  2. [Figure 10] In the C2R benchmark, the paper compares only against pi0, pi0.5, and Fast-WAM and omits Abot-M0.5 and LingBot-VA 2.0, which are the two closest competitors in Table 2. Without their C2R scores, the claim that WorldScape Policy 2.0 'achieves the highest average success rate' at 47.9% is unsupported. Report those numbers or explicitly explain why they are unavailable.
  3. [Tables 5 and 6] The component and curriculum ablations show monotonic improvements, and that is a strength. However, no multiple-seed variance is reported. The increments from Stage-2 mid-training and semantic forcing are small (average 47.30% to 47.89% in Table 6; randomized-condition gains around 0.4–0.7 points), and with 100 trials per task across 50 tasks these differences may be within noise. Please add seeds, error bars, and, ideally, a significance test, particularly for the contribution of semantic forcing, which is central to the paper's claim that event semantics transfer to latent subgoal planning.
  4. [§3.5, Eq. (16)] The event boundaries and captions in ManipEvent-5M are generated automatically by Qwen3-VL-32B. The paper states only that outputs are 'parsed and validated as structured records' with retries and fallbacks; no human agreement, error rate, or manual sample audit is reported. These captions are the supervision targets of the semantic-forcing loss (Eq. 16) and the event-boundary selections in Eq. (11). If the labels are noisy or boundaries are misaligned, the claimed progress-aware reasoning is trained on unreliable targets. Please provide a human-validated evaluation of caption and boundary accuracy on a representative sample, and, if feasible, a label-noise sensitivity analysis.
minor comments (6)
  1. [Table 1] The segment counts for RoboMIND, RoboCOIN, and DROID are shown as '–', while 'Single-Seg. Ratio' is 100%. Clarify whether these datasets are treated as single-segment episodes and whether their segment counts are included in the total or not.
  2. [Eqs. (10)–(11)] The notation TopKΔ and the 'Expand' operation are used without a precise formal definition. Specify the greedy selection procedure with the minimum temporal separation Δ and the exact slot expansion used for global-history latents.
  3. [§4.2] Several hyperparameters that appear in the method are not given concrete values: S_v (short-term visual window), S_b and Δ (event-boundary count/separation), λ_s, and α. The text gives S_e=8 and K=4, but the others should be listed for reproducibility.
  4. [Figure 2] The caption text appears garbled: 'First-view of Goal ImageThird-view of Goal Image' is missing a separator. Also, the figure's 'Long Short-Term Memory WAM' label is ambiguous because the paper distinguishes short-term visual memory from long-term event memory.
  5. [§3.3] The sentence 'Following the implementation' is informal and does not identify which implementation is meant. Replace it with a precise description of how the compact full-history bank is concatenated with the three memory views.
  6. [References] Reference [62] for WorldScape Policy 1.0 is formatted as an organization-only citation with no authors or technical report number. If this is a self-citation to an unpublished predecessor, please provide full bibliographic information or a publicly accessible technical report.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims are empirical results of supervised pretraining and external-benchmark evaluation, not reconstructions of their inputs.

full rationale

The derivation chain is not circular. Semantic forcing (Eq. 16) uses fine-grained captions as training-time targets only; at inference the caption and its T5 embedding are absent (Eq. 17 selects the memory-enhanced latent q_hat), so the model is trained to predict a latent subgoal from observations, memory, and the high-level instruction rather than copying the caption into the output. Event-boundary selection (Eq. 11) computes boundary indices from historical VLM latents, but these are memory-retrieval anchors, not the predicted actions or the benchmark success labels; no equation retrofits the reported success rate into the model's input. ManipEvent-5M's auto-generated captions and boundaries (Sec. 3.5) are a supervision-quality risk, since validation is only described as 'parsed and validated as structured records,' but that is a data-quality/correctness concern, not an equivalence-by-construction. The self-citation to WorldScape Policy 1.0 [62] and other author-affiliated works is used as a baseline or contextual comparison, not as the load-bearing justification for the central result. The benchmark gaps over Abot-M0.5 are small and lack reported variance, which is a statistical-robustness concern, not circularity. Overall, the core training and evaluation pipeline is self-contained against external RoboTwin 2.0 and real-robot tasks, so no circular step meets the quoting-and-reduction standard.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

The central claims rely on the accuracy of automatically generated event labels and on several hand-chosen architectural hyperparameters. The invented entities are internal representations and an unreleased dataset, none of which provide an independent falsifiable handle outside the paper's own evaluation.

free parameters (6)
  • Short-term visual memory window S_v = 4 chunks
    Implementation Details: 'the short-term visual memory retains up to 4 recent chunks.' Chosen by hand; affects the balance between context and compute.
  • Long-term event memory size S_e = 8 chunks (default)
    Implementation Details: 'The long-term memory keeps 8 history chunks by default, and this length can be adjusted.' Chosen by hand.
  • Planning tokens K = 4
    Implementation Details: 'autoregressively generates K=4 planning tokens.' Selected by hand; if larger, the latent plan might be richer but inference cost grows.
  • Event-boundary count S_b and separation Delta = not specified
    Eq. (11) uses TopKDelta with S_b slots and minimum separation Delta; values are not reported. Hand-chosen hyperparameters of the memory construction.
  • Semantic forcing weight lambda_s = 0.001 when active
    Sec. 3.6: 'lambda_s is 0.001 when semantic forcing is active and zero otherwise.' Chosen by hand.
  • Residual scaling factor alpha = not specified
    Eq. (14) introduces alpha as a residual scaling factor; no value or schedule is reported.
axioms (4)
  • domain assumption ManipEvent-5M event boundaries and captions generated by Qwen3-VL (with robot-signal priors) are accurate enough to serve as training targets.
    Sec. 3.5 describes fully automatic two-stage labeling without human verification; the paper implicitly assumes label correctness.
  • domain assumption T5 embeddings of fine-grained captions form a semantic space to which planning latents can be aligned via L_sem.
    Eq. (16) defines L_sem as cosine similarity between normalized T5 summary and projected planning summary; this assumes the T5 space is semantically meaningful for action subgoals.
  • domain assumption K=4 greedy-decoded VLM tokens provide a sufficient latent subgoal representation.
    Eq. (7)-(9): the planning latent q_t is formed from 4 auto-regressive tokens; no evidence is given that 4 is sufficient or that greedy decoding is stable.
  • domain assumption The Wan2.2-5B text-to-video pretrained DiT is a suitable backbone for joint video-action modeling.
    Implementation Details: the video DiT is initialized from Wan2.2-5B; the paper relies on this pretrained model's capabilities.
invented entities (3)
  • Implicit subgoal latent (planning tokens) no independent evidence
    purpose: Represents the inferred next atomic action/subgoal in latent space for autonomous planning.
    Introduced in Sec. 3.3-3.4; its only evidence is the internal L_sem alignment and downstream task success, no external falsifiable handle.
  • Three memory views (global-history, local-active, event-boundary latents) no independent evidence
    purpose: Organize historical VLM outputs for progress-aware retrieval.
    Sec. 3.3; internal representations whose contribution is shown only via ablations.
  • ManipEvent-5M dataset no independent evidence
    purpose: Event-grounded pretraining data with captions, goals, videos.
    Introduced in Sec. 3.5 but not released; cannot be independently inspected or used.

pith-pipeline@v1.3.0-alltime-deepseek · 23635 in / 12761 out tokens · 106156 ms · 2026-08-01T14:09:05.411909+00:00 · methodology

0 comments
read the original abstract

World Action Models (WAMs) offer a promising paradigm for robotic manipulation by jointly modeling visual state transitions and robot actions. However, existing WAMs are constrained by limited temporal context, coarse episode-level language supervision, and predominantly text-only conditioning, which hinder task-progress tracking and fine-grained language-video-action grounding while limiting visual-context reasoning and cross-embodiment transfer. In this paper, we introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory. Its causal short-term visual memory supplies recent observations as DiT prefill to preserve local interaction dynamics, while its long short-term event memory organizes historical VLM outputs into global-history, local-active, and event-boundary representations for progress-aware retrieval. The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning; semantic forcing further transfers event-level instruction semantics into this latent planning pathway. To establish fine-grained multimodal controllability, we construct ManipEvent-5M, an event-grounded embodied pretraining dataset containing nearly 5 million event segments with aligned action trajectories, episode-level task instructions, segment-level subtask captions, goal images, and video demonstrations. These designs provide a unified interface for autonomous planning from high-level instructions and controllable execution from fine-grained text, goal-image, or video-context prompts. Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following and in-context adaptation.

Figures

Figures reproduced from arXiv: 2607.18840 by Baorun Li, Chen Gao, Chengming Hu, Haisheng Su, Haoxuan Dou, Jianjie Fang, Junchi Yan, Ruiyan Xu, Wei Wu, Xin Jin, Xin Zhang, Xue Yang, Yong Li, Zhanwang Liu, Zhenjie Yang, Zongdai Liu.

Figure 1
Figure 1. Figure 1: Comparison of different WAM paradigms. Existing models use (a) static observations, (b) short-term visual history, or (c) full-history visual memory without progress-aware reasoning or multimodal control. (d) WorldScape Policy 2.0 introduces multimodal controllability and reasoning-augmented long short-term memory, enabling interactive video-action modeling for long-horizon robotic manipulation. expands ac… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the WorldScape Policy 2.0 framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of Reasoning-Augmented Long Short-Term Memory. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Attention mask of WorldScape Policy 2.0. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Construction pipeline of ManipEvent-5M. We aggregate heterogeneous manipulation sources, including human-arm ego videos, robot-free UMI data, simulated trajectories, and real-robot demonstrations, and convert them into event-level training samples. Each trajectory is segmented into ordered atomic subgoals and annotated with paired action trajectories, episode-level global instructions, segment-level fine-g… view at source ↗
Figure 6
Figure 6. Figure 6: Examples of hierarchical captioning in ManipEvent-5M. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution statistics of ManipEvent-5M. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Multimodal prompt example from ManipEvent-5M: [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Multimodal prompt example from ManipEvent-5M: [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison on the RoboTwin 2.0 standard and C2R benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Illustration of real-world tasks on the dual-arm PiPER platform. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 1 canonical work pages

  1. [1]

    Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models.arXiv preprint arXiv:2602.08971, 2026

    Yu Shang, Zhuohang Li, Yiding Ma, Weikang Su, Xin Jin, Ziyou Wang, Lei Jin, Xin Zhang, Yinzhou Tang, Haisheng Su, et al. Worldarena: A unified benchmark for evaluating perception and functional utility of embodied world models.arXiv preprint arXiv:2602.08971, 2026

  2. [2]

    Robosense: Large-scale dataset and benchmark for egocentric robot perception and navigation in crowded and unstructured environments

    Haisheng Su, Feixiang Song, Cong Ma, Wei Wu, and Junchi Yan. Robosense: Large-scale dataset and benchmark for egocentric robot perception and navigation in crowded and unstructured environments. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 27446–27455, 2025

  3. [3]

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, ...

  4. [4]

    𝜋0.7: A steerable generalist robotic foundation model with emergent capabilities, 2026

    Physical Intelligence, Bo Ai, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Greg Balke, Kevin Black, et al. 𝜋0.7: A steerable generalist robotic foundation model with emergent capabilities, 2026. URLhttps: //arxiv.org/abs/2604.15483

  5. [5]

    Drivemoe: Mixture-of-expertsforvision-language-actionmodelinend-to-endautonomousdriving

    Zhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li, Yuqian Shao, Xuekai Zhu, Haisheng Su, and Junchi Yan. Drivemoe: Mixture-of-expertsforvision-language-actionmodelinend-to-endautonomousdriving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10678–10688, 2026

  6. [6]

    Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

    Tianyuan Yuan, Zibin Dong, Yicheng Liu, and Hang Zhao. Fast-wam: Do world action models need test-time future imagination?arXiv preprint arXiv:2603.16666, 2026

  7. [7]

    Motus: A unified latent action world model

    Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A unified latent action world model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 35101–35113, June ...

  8. [8]

    Unified 4d world action modeling from video priors with asynchronous denoising.arXiv preprint arXiv:2604.26694, 2026

    Jun Guo, Qiwei Li, Peiyan Li, Zilong Chen, Nan Sun, Yifei Su, Heyun Wang, Yuan Zhang, Xinghang Li, and Huaping Liu. Unified 4d world action modeling from video priors with asynchronous denoising.arXiv preprint arXiv:2604.26694, 2026

  9. [9]

    Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026

    Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Hengtao Li, Jie Li, Jindi Lv, Jingyu Liu, et al. Gigaworld-policy: An efficient action-centered world–action model.arXiv preprint arXiv:2603.17240, 2026

  10. [10]

    Causal world modeling for robot control

    Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Luyao Zhang, Mingrui Yu, Zelin Gao, Nan Xue, Boyu Zhou, Xing Zhu, Mingyu Ding, Yujun Shen, and Yinghao Xu. Causal world modeling for robot control. In Proceedings of Robotics: Science and Systems, 2026. URLhttps://roboticsconference.org/program/ papers/16/

  11. [11]

    World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

    Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies.arXiv preprint arXiv:2602.15922, 2026

  12. [12]

    MemoryWAM: Efficient world action modeling with persistent memory.arXiv preprint arXiv:2606.20562, 2026

    Sizhe Yang, Juncheng Mu, Tianming Wei, Chenhao Lu, Xiaofan Li, Linning Xu, Zhengrong Xue, Zhecheng Yuan, Dahua Lin, Jiangmiao Pang, and Huazhe Xu. MemoryWAM: Efficient world action modeling with persistent memory.arXiv preprint arXiv:2606.20562, 2026. 20

  13. [13]

    DINO-WM: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann Lecun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 79115–79135. PMLR, 2025. URLhttps: //proceedings.mlr.press/v267/zhou25t.html

  14. [14]

    Enerverse: Envisioning em- bodied future space for robotics manipulation

    Siyuan Huang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Yue Liao, Zhengkai Jiang, Yue Hu, Peng Gao, Hongsheng Li, Maoqing Yao, and Guanghui Ren. Enerverse: Envisioning em- bodied future space for robotics manipulation. InAdvances in Neural Information Processing Sys- tems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ ...

  15. [15]

    Video prediction policy: A generalist robot policy with predictive visual representations

    Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. InProceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 24...

  16. [16]

    Dreamgen: Unlocking generalization in robot learning through video world models

    Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loïc Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xiaohui Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke Zettlemoyer, Dieter Fox, Jan Kautz, Scott Reed, Yuke Zh...

  17. [17]

    Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    YilunDu, SherryYang, BoDai, HanjunDai, OfirNachum, JoshTenenbaum, DaleSchuurmans, andPieterAbbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

  18. [18]

    Vidar: Embodied video diffusion model for generalist bimanual manipulation.arXiv preprint arXiv:2507.12898, 2025

    Yao Feng, Hengkai Tan, Xinyi Mao, Guodong Liu, Shuhe Huang, Chendong Xiang, Hang Su, and Jun Zhu. Vidar: Embodied video diffusion model for generalist bimanual manipulation.arXiv preprint arXiv:2507.12898, 2025

  19. [19]

    Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation

    Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation. InProceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, page...

  20. [20]

    Tsi: temporal saliency integration for video action recognition.arXiv preprint arXiv:2106.01088, 2021

    Haisheng Su, Kunchang Li, Jinyuan Feng, Dongliang Wang, Weihao Gan, Wei Wu, and Yu Qiao. Tsi: temporal saliency integration for video action recognition.arXiv preprint arXiv:2106.01088, 2021

  21. [21]

    Discovering a variety of objects in spatio-temporal human-object interactions

    Yong-Lu Li, Hongwei Fan, Zuoyu Qiu, Yiming Dou, Liang Xu, Hao-Shu Fang, Peiyang Guo, Haisheng Su, Dongliang Wang, Wei Wu, et al. Discovering a variety of objects in spatio-temporal human-object interactions. arXiv preprint arXiv:2211.07501, 2022

  22. [22]

    Collaborative distillation in the parameter and spectrum domains for video action recognition.arXiv preprint arXiv:2009.06902, 2020

    Haisheng Su, Jing Su, Dongliang Wang, Weihao Gan, Wei Wu, Mengmeng Wang, Junjie Yan, and Yu Qiao. Collaborative distillation in the parameter and spectrum domains for video action recognition.arXiv preprint arXiv:2009.06902, 2020

  23. [23]

    RoboDreamer: Learning compositional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning compositional world models for robot imagination. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 61885–61896. PMLR, 2024. URLhttps://proceedings.mlr.press/v235/zhou24f.html

  24. [24]

    Drivemamba: Task-centric scalable state space model for efficient end-to-end autonomous driving.arXiv preprint arXiv:2602.13301, 2026

    Haisheng Su, Wei Wu, Feixiang Song, Junjie Zhang, Zhenjie Yang, and Junchi Yan. Drivemamba: Task-centric scalable state space model for efficient end-to-end autonomous driving.arXiv preprint arXiv:2602.13301, 2026. 21

  25. [25]

    Egofsd: Ego-centricfullysparseparadigmwithuncertainty denoising and iterative refinement for efficient end-to-end self-driving

    HaishengSu,WeiWu,ZhenjieYang,andIsabelGuan. Egofsd: Ego-centricfullysparseparadigmwithuncertainty denoising and iterative refinement for efficient end-to-end self-driving. In2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026

  26. [26]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

  27. [27]

    Gigaworld-0: Worldmodelsasdataenginetoempowerembodiedai.arXivpreprint arXiv:2511.19861, 2025

    GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, KeruiLi,MengyuanXu, etal. Gigaworld-0: Worldmodelsasdataenginetoempowerembodiedai.arXivpreprint arXiv:2511.19861, 2025

  28. [28]

    Cosmos policy: Fine-tuning video models for visuomotor control and planning

    Moo Jin Kim, Yihuai Gao, Tsung-Yi Lin, Yen-Chen Lin, Yunhao Ge, Grace Lam, Percy Liang, Shuran Song, Ming-Yu Liu, Chelsea Finn, and Jinwei Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning. InInternational Conference on Learning Representations, 2026. URLhttps://iclr.cc/virtual/ 2026/poster/10006732

  29. [29]

    OA-WAM: Object-addressable world action model for robust robot manipulation, 2026

    YushanLiu,PeiboSun,ShoujieLi,YifanXie,LingfengZhang,XintaoChao,ShiyuanDong,FangChen,Xiao-Ping Zhang, and Wenbo Ding. OA-WAM: Object-addressable world action model for robust robot manipulation, 2026. URLhttps://arxiv.org/abs/2605.06481

  30. [30]

    Long-VLA: Unleashing long-horizon capability of vision language action model for robot manipulation

    Yiguo Fan, Shuanghao Bai, Xinyang Tong, Pengxiang Ding, Yuyang Zhu, Hongchao Lu, Fengqi Dai, Wei Zhao, Yang Liu, Siteng Huang, Zhaoxin Fan, Badong Chen, and Donglin Wang. Long-VLA: Unleashing long-horizon capability of vision language action model for robot manipulation. InProceedings of The 9th Conference on Robot Learning, volume 305 ofProceedings of Ma...

  31. [31]

    VLA-OS: Structuring and dissecting planning representations and paradigms in vision-language-action models

    Chongkai Gao, Zixuan Liu, Zhenghao Chi, Junshan Huang, Xin Fei, Yiwen Hou, Yuxuan Zhang, Yudi Lin, Zhirui Fang, and Lin Shao. VLA-OS: Structuring and dissecting planning representations and paradigms in vision-language-action models. InAdvances in Neural Information Processing Sys- tems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/pape...

  32. [32]

    MEM: Multi-scale embodied memory for vision language action models, 2026

    Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, et al. MEM: Multi-scale embodied memory for vision language action models, 2026. URLhttps://arxiv.org/abs/2603.03596

  33. [33]

    MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manipulation

    Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language-action models for robotic manipulation. InInternational Conference on Learning Representations, 2026. URLhttps://openreview. net/forum?id=54U3XHf7qq

  34. [34]

    Long-horizon manipulation via trace-conditioned VLA planning, 2026

    Isabella Liu, An-Chieh Cheng, Rui Yan, Geng Chen, Ri-Zhao Qiu, Xueyan Zou, Sha Yi, Hongxu Yin, Xiaolong Wang, and Sifei Liu. Long-horizon manipulation via trace-conditioned VLA planning, 2026. URLhttps: //arxiv.org/abs/2604.21924

  35. [35]

    Goal2Skill: Long-horizon manipulation with adaptive planning and reflection,

    Zhen Liu, Xinyu Ning, Zhe Hu, Xinxin Xie, Weize Li, Zhipeng Tang, Chongyu Wang, Zejun Yang, Hanlin Wang, Yitong Liu, and Zhongzhu Pu. Goal2Skill: Long-horizon manipulation with adaptive planning and reflection,

  36. [36]

    DSWAM: A dual-system world action foundation model for fine-grained robot manipulation,

    JianZhu,JianjunZhang,TaiyiSu,TianbinLiu,ZhangyuanWang,KaiXie,ZitaiHuang,ChongMa,YouzhangHe, Tianjian Wang, et al. DSWAM: A dual-system world action foundation model for fine-grained robot manipulation,

  37. [37]

    Barry, Kris Kitani, and George Konidaris

    Jiahui Fu, Junyu Nan, Lingfeng Sun, Hongyu Li, Jianing Qian, Jennifer L. Barry, Kris Kitani, and George Konidaris. NovaPlan: Zero-shot long-horizon manipulation via closed-loop video language planning, 2026. URL https://arxiv.org/abs/2602.20119. 22

  38. [38]

    Qwen-VLA: Unifying vision-language-action modeling across tasks, environments, and robot embodiments, 2026

    Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, Sicheng Xie, Yitao Liu, Junhao Chen, Zhixuan Liang, Jie Zhang, Xintong Hu, et al. Qwen-VLA: Unifying vision-language-action modeling across tasks, environments, and robot embodiments, 2026. URLhttps://arxiv.org/abs/2605.30280

  39. [39]

    URLhttps://arxiv.org/abs/2607.04927

  40. [40]

    PixelVLA: Advancing pixel-level understanding in vision-language-action model

    Wenqi Liang, Gan Sun, Yao He, Jiahua Dong, Suyan Dai, Ivan Laptev, Salman Khan, and Yang Cong. PixelVLA: Advancing pixel-level understanding in vision-language-action model. InInternational Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id=7M6ryCABIc

  41. [41]

    SpatialVLA: Exploring spatial representations for vision-language-action models

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Jiayuan Gu, Zhigang Wang, Yan Ding, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring spatial representations for vision-language-action models. InProceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. doi: 10.15607/RSS.2025. XXI.011

  42. [42]

    FineVLA: Fine-grained instruction alignment for steerable vision-language-action policies, 2026

    Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, et al. FineVLA: Fine-grained instruction alignment for steerable vision-language-action policies, 2026. URLhttps://arxiv.org/abs/2605.27284

  43. [43]

    SG- VLA: Learning spatially-grounded vision-language-action models for mobile manipulation, 2026

    Ruisen Tu, Arth Shukla, Sohyun Yoo, Xuanlin Li, Junxi Li, Jianwen Xie, Hao Su, and Zhuowen Tu. SG- VLA: Learning spatially-grounded vision-language-action models for mobile manipulation, 2026. URLhttps: //arxiv.org/abs/2603.22760

  44. [44]

    TraceVLA: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies

    Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. TraceVLA: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. InInternational Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum?id=b1CVu9l5GO

  45. [45]

    ReconVLA: Reconstructive vision-language-action model as effective robot perceiver

    Wenxuan Song, Ziyang Zhou, Han Zhao, Jiayi Chen, Pengxiang Ding, Haodong Yan, Yuxin Huang, Feilong Tang, Donglin Wang, and Haoang Li. ReconVLA: Reconstructive vision-language-action model as effective robot perceiver. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 18549–18557, 2026. doi: 10.1609/aaai.v40i22.38921. URLhtt...

  46. [46]

    Learning generalizable robot policy with human demonstration video as a prompt, 2025

    Xiang Zhu, Yichen Liu, Hezhong Li, and Jianyu Chen. Learning generalizable robot policy with human demonstration video as a prompt, 2025. URLhttps://arxiv.org/abs/2505.20795

  47. [47]

    AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems

    AgiBot-World Team. AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3549–3556. IEEE, 2025. doi: 10.1109/IROS60139.2025.11247088. URLhttps://ieeexplore.ieee.org/ document/11247088/

  48. [48]

    InProceedingsoftheIEEE/CVFConferenceon Computer Vision and Pattern Recognition (CVPR), pages 1702–1713, June 2025

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han,ChelseaFinn,AnkurHanda,Tsung-YiLin,GordonWetzstein,Ming-YuLiu,andDonglaiXiang.CoT-VLA:Vi- sualchain-of-thoughtreasoningforvision-language-actionmodels. InProceedingsoftheIEEE/CVFConferenceon Computer Vision and Pattern Recognition (CVPR), pages 1702...

  49. [49]

    RoboCOIN: An open-sourced bimanual robotic data collection for integrated manipulation,

    ShihanWu,XuechengLiu,ShaoxuanXie,PengweiWang,XinghangLi,ZheLi,KaiZhu,HongyuWu,YihengLiu, Zhaoye Long, et al. RoboCOIN: An open-sourced bimanual robotic data collection for integrated manipulation,

  50. [50]

    DROID: A large-scale in-the-wild robot manipulation dataset

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. DROID: A large-scale in-the-wild robot manipulation dataset. InProceedings of Robotics: Science and Systems, Delft, Netherlands, July

  51. [51]

    Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation

    Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju, Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, et al. Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation. InRobotics: Science and Systems (RSS) 2025. Robotics: Science and Systems Foundation, 2025. URLhttps://www.roboticsproceedings.org/rss21/p152.pdf

  52. [52]

    LIBERO: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. InAdvances in Neural Information Processing Systems, volume 36, pages 44776–44791, 2023. URLhttps://proceedings.neurips.cc/paper_files/paper/2023/hash/ 8c3c666820ea055a77726d66fc7d447f-Abstract-Datasets_a...

  53. [53]

    URLhttps://arxiv.org/abs/2511.17441

  54. [54]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

  55. [55]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

  56. [56]

    RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation

    Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan-ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. RoboTwin 2.0: A scalable d...

  57. [57]

    URLhttps://icml.cc/virtual/2026/poster/62192

  58. [58]

    Ronghan Chen, Yandan Yang, Zuojin Tang, Dongjie Huo, Tong Lin, Haoning Wu, Haoyun Liu, Yuzhi Chen, Lulu Zheng, Botai Yuan, et al. Abot-m0. 5: Unified mobility-and-manipulation world action model.arXiv preprint arXiv:2607.00678, 2026

  59. [59]

    Yoon, Mouli Sivapurapu, and Jian Zhang

    Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. InInternational Conference on Learning Representations, 2026. URLhttps://mlanthology.org/iclr/2026/hoque2026iclr-egodex/

  60. [60]

    Holobrain-0 technical report.arXiv preprint arXiv:2602.12062, 2026

    XuewuLin,TianweiLin,YunDu,HongyuXie,YiweiJin,JiaweiLi,ShijieWu,QingzeWang,MengdiLi,Mengao Zhao, et al. Holobrain-0 technical report.arXiv preprint arXiv:2602.12062, 2026

  61. [61]

    Native video-action pretraining for generalizable robot control.arXiv preprint arXiv:2607.08639, 2026

    Qihang Zhang, Lin Li, Luyao Zhang, Shuai Yang, Yiming Luo, Shuaiting Li, Ruilin Wang, Junke Wang, Jiahao Shao, Gangwei Xu, et al. Native video-action pretraining for generalizable robot control.arXiv preprint arXiv:2607.08639, 2026

  62. [62]

    InProceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Laura Smith, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Z...

  63. [63]

    X-VLA: Soft-prompted transformerasscalablecross-embodimentvision-language-actionmodel

    Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, Tai Wang, Ya-Qin Zhang, Jingjing Liu, and Xianyuan Zhan. X-VLA: Soft-prompted transformerasscalablecross-embodimentvision-language-actionmodel. InInternationalConferenceonLearning Representations, 2026. URLhttps://openreview.net/...

  64. [65]

    A pragmatic vla foundation model.arXiv preprint arXiv:2601.18692, 2026

    Wei Wu, Fan Lu, Yunnan Wang, Shuai Yang, Shi Liu, Fangjing Wang, Qian Zhu, He Sun, Yong Wang, Shuailei Ma, et al. A pragmatic vla foundation model.arXiv preprint arXiv:2601.18692, 2026

  65. [68]

    Worldscape policy: Generalizable robotic learning via a foundation world model, 2026

    Manifold AI. Worldscape policy: Generalizable robotic learning via a foundation world model, 2026. 24

  66. [2024]

    doi: 10.15607/RSS.2024.XX.120. 23

  67. [2025]

    URLhttps://proceedings.mlr.press/v305/black25a.html

  68. [2026]

    URLhttps://arxiv.org/abs/2604.13942