pith. sign in

arxiv: 2606.23085 · v1 · pith:EC4VPAZVnew · submitted 2026-06-22 · 💻 cs.RO

Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents

Haoran Zhang , Yifu Lu , Boyang Wang , Xuhui Kang , Yen-Ling Kuo , Zezhou Cheng , Mengdi Wang , Odest Chadwicke Jenkins This is my paper

Pith reviewed 2026-06-26 08:41 UTC · model grok-4.3

classification 💻 cs.RO
keywords failure detectionlong-horizon manipulationworld modelsrobotic manipulationconformal predictionvision-language-action policies
0
0 comments X

The pith

Action-conditioned world model latents enable reliable failure detection in long-horizon robotic manipulation using only final task success labels.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Foresight, a framework that detects failures during long robotic manipulation tasks by monitoring latent embeddings produced by an action-conditioned world model. The system is trained exclusively on whether each full trajectory ended in success or failure, without any labels marking the exact moment a failure began. This approach works uniformly across different policies and uses functional conformal prediction to set adaptive detection thresholds. A sympathetic reader would care because real-world deployments often involve extended tasks where failures start ambiguously and dense annotations are impractical to collect.

Core claim

Foresight monitors manipulation trajectories using latent representations from an action-conditioned world model. Foresight is trained using only final task-level success or failure labels. By leveraging predictive world-model embeddings, the method provides a unified framework for failure detection across different policies and further uses functional conformal prediction to calibrate detection thresholds adaptively, with evaluation on state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K plus real-robot validation on a ReactorX-200 arm and a Franka arm.

What carries the argument

Action-conditioned world-model embeddings, which serve as scalable predictive representations of future states given actions and are used to monitor trajectories for signals of failure onset.

If this is right

  • Failure detection becomes feasible for long-horizon tasks without requiring dense temporal annotations.
  • A single monitoring method applies across multiple vision-language-action policies.
  • Functional conformal prediction supplies policy-specific adaptive thresholds.
  • The same embeddings support both simulation benchmarks and real-robot hardware validation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Detection signals could be used to trigger online policy recovery or replanning during execution.
  • The embeddings might be inspected to classify distinct failure modes rather than only binary detection.
  • Similar latent monitoring could transfer to other sequential domains such as autonomous driving or game agents.

Load-bearing premise

Latent representations from an action-conditioned world model trained only on final task success or failure labels contain enough information to detect the onset of ambiguous failures.

What would settle it

A controlled test in which replacing the action-conditioned world model with a non-action-conditioned version or removing the success/failure labels causes detection performance to fall to chance level on the same long-horizon benchmarks.

Figures

Figures reproduced from arXiv: 2606.23085 by Boyang Wang, Haoran Zhang, Mengdi Wang, Odest Chadwicke Jenkins, Xuhui Kang, Yen-Ling Kuo, Yifu Lu, Zezhou Cheng.

Figure 1
Figure 1. Figure 1: Overview of Foresight. Foresight consists of three stages. Stage 1: we fine-tune an action-conditioned world model (WM-AC) on robot rollouts consisting of image observations I1:T and actions a1:T −1. Stage 2: for each timestep t, the world model encodes the current observation context into hidden latents z h t and predicts action-conditioned future latents z p t using the policy￾predicted action chunk At. … view at source ↗
Figure 2
Figure 2. Figure 2: Real-Robot Setup. Left: real-world robot setting for three table-top manipulation tasks using ReactorX-200 arm. Right: real-world robot setting for a three-toy picking task using Franka arm. where TPR denotes the true positive rate and TNR denotes the true negative rate. Balanced accuracy assigns equal weight to successful and failed rollouts, making it robust to class imbalance. We evaluate all baselines … view at source ↗
Figure 3
Figure 3. Figure 3: Benchmark tasks overview 16 [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LIBERO-Long tasks overview 17 [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: ManiSkill-Long tasks overview task-level rollout statistics for π0-FAST. In total, we collect 319 valid rollouts across four tasks. Compared with LIBERO-Long, ManiSkill-Long requires longer execution horizons. Successful π0- FAST rollouts require 93 policy calls and 1,484 simulation control steps on average. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Behavior-1k tasks overview BEHAVIOR-1K evaluates long-horizon mobile manipulation in large-scale household environ￾ments. We select four tasks (as shown in [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Real-world experiment task overview. Franka / GR00T N1.5. We collect 44 episodes of the “pick 3 toys” task using GR00T N1.5 [21] on a Franka arm, with an average of 38 policy calls and an exec horizon of 45 steps per call (∼1700 total executed steps), achieving 48% success. 12 Ablation Studies This section studies which components of Foresight are responsible for performance. 12.1 World-Model Backbone Cosm… view at source ↗
Figure 8
Figure 8. Figure 8: LIBERO-Long (True Negative) (α=0.02, Task 0). “Put both the alphabet soup and the tomato sauce in the basket.” The failure score st (blue) remains below the FCP threshold δt (red dashed) throughout all inference steps; no alarm is raised and all frame borders are green [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: LIBERO-Long (True Positive) (α=0.02, Task 5). “Pick up the book and place it in the back compartment of the caddy.” Foresight raises an alarm before episode termination as the action-conditioned world model’s predicted states increasingly diverge from observed states. The robot failed the task because it dropped the book during the middle of execution. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: ManiSkill-Long (True Negative) (α=0.02, Task 2: Cubes into Bowl). “Put three cubes into the bowl.” [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: ManiSkill-Long (True Positive) (α=0.02, Task 3: Stack 3 Cubes ). “Stack 3 cubes together, starting with the red cube.” The robot failed to stack the red cube on the blue cube, leading to the final failure [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: BEHAVIOR-1K (True Negative) (α=0.20, Task 3: Setting Mousetraps). “Take four mousetraps from the bathroom cabinet and place at least two next to the same sink.” 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: BEHAVIOR-1K (True Positive) (α=0.20, Task 47: Cook Hot Dogs). “Take two hot dogs from the refrigerator and cook them in the microwave.” The robot fails during this task because it did not grasp the first hot dog [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Real-world (ReactorX / ACT) (True Negative) (α=0.10, Pick Banana and toy lion task). “Pick up banana and lion toy into basket.” No false alarm is raised, showing Foresight does not penalize successful executions [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Real-world (ReactorX / ACT) (True Positive) (α=0.10, Pick Banana and toy lion task). “Pick up banana and lion toy into basket.” A failing real-robot episode from the same task. The robot failed to pick up the banana, leading to final task failure. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
read the original abstract

Long-horizon tasks are common in real-world robotic deployments, yet failure detection for such tasks remains underexplored. Detecting failures in long-horizon robotic tasks is particularly challenging because failure onset is often ambiguous and dense temporal annotations are typically unavailable. We present Foresight, a failure detection framework that monitors manipulation trajectories using latent representations from an action-conditioned world model. Foresight is trained using only final task-level success or failure labels. By leveraging predictive world-model embeddings, our method provides a unified framework for failure detection across different policies. We further use functional conformal prediction (FCP) to calibrate detection thresholds adaptively. We evaluate Foresight with state-of-the-art vision-language-action policies in simulation on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K, compare it against state-of-the-artfailure detection methods, and validate it on real robots with three long-horizon tasks on a ReactorX-200 arm and one task on a Franka arm. Our results suggest that action-conditioned world-model embeddings provide a scalable representation for reliable failure monitoring in long-horizon manipulation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper introduces Foresight, a failure detection framework for long-horizon robotic manipulation that monitors trajectories using latent representations from an action-conditioned world model. The framework is trained solely on final task-level success/failure labels (no dense temporal annotations) and uses functional conformal prediction (FCP) to calibrate detection thresholds adaptively. It is evaluated in simulation against SOTA failure detection baselines on LIBERO-Long, ManiSkill-Long, and BEHAVIOR-1K using vision-language-action policies, and validated on real robots (ReactorX-200 arm with three tasks; Franka arm with one task). The central claim is that action-conditioned world-model embeddings supply a scalable, policy-agnostic representation for reliable failure monitoring when failure onset is ambiguous.

Significance. If the empirical results and ablations hold, the work would be significant for robotics because it directly tackles an underexplored but practically critical problem: failure detection in long-horizon tasks without expensive dense labels. The unified treatment across policies and the grounding in predictive world-model latents could improve safety and reliability in real deployments. The combination of world-model embeddings with FCP for calibration is a coherent and externally grounded approach that avoids circularity in the stated construction.

minor comments (2)
  1. [Abstract] Abstract: The abstract states the method, training regime, and evaluation setup but supplies no quantitative results, ablation details, or key performance numbers. Including at least the main comparative metrics (e.g., detection rates or AUC on the simulation benchmarks) would allow readers to assess the strength of the central claim directly from the summary.
  2. The manuscript should clarify in the methods or experiments section how the world-model latents are extracted at inference time (e.g., which layer or timestep) and whether any additional fine-tuning occurs beyond the terminal-label training described.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive assessment of the work, including the recognition of its significance for long-horizon failure detection without dense labels and the coherent use of world-model latents with functional conformal prediction. We appreciate the recommendation for minor revision and will incorporate any suggested improvements accordingly.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper describes a standard pipeline: train an action-conditioned world model, extract latents, and train a failure detector using only terminal success/failure labels, with FCP for calibration. No equations, derivations, or self-citations are shown that reduce the claimed performance or representations to quantities defined by the method itself. The central hypothesis (that these latents carry failure signal) is presented as the claim under test rather than presupposed, and evaluation uses external benchmarks and real-robot tasks. The derivation chain is therefore self-contained against external data.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Central claim rests on the domain assumption that world-model latents encode failure-relevant information from task-level labels alone; no free parameters or invented physical entities are named in the abstract.

axioms (1)
  • domain assumption Action-conditioned world model latents contain information sufficient to detect failure onset from task-level labels only
    Invoked in the description of how Foresight monitors trajectories without dense annotations.
invented entities (1)
  • Foresight framework no independent evidence
    purpose: Unified failure detection using world-model latents and FCP
    New method introduced to solve the stated problem; no independent evidence outside the paper is provided.

pith-pipeline@v0.9.1-grok · 5758 in / 1267 out tokens · 23402 ms · 2026-06-26T08:41:18.432234+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 6 canonical work pages · 2 internal anchors

  1. [1]

    C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina. Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies.arXiv preprint arXiv:2503.08558, 2025

    arXiv 2025

  2. [2]

    Q. Gu, Y . Ju, S. Sun, I. Gilitschenski, H. Nishimura, M. Itkina, and F. Shkurti. Safe: Multitask failure detection for vision-language-action models.arXiv preprint arXiv:2506.09937, 2025

    arXiv 2025

  3. [3]

    Yeh, K.-H

    J.-F. Yeh, K.-H. Hung, P.-C. Lo, C.-M. Chung, T.-H. Wu, H.-T. Su, Y .-T. Chen, and W. H. Hsu. Aed: Adaptable error detection for few-shot imitation policy. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  4. [4]

    J. Duan, W. Pumacay, N. Kumar, Y . R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Man- dlekar, and Y . Guo. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation.arXiv preprint arXiv:2410.00371, 2024

    arXiv 2024

  5. [5]

    N. He, S. Li, Z. Li, Y . Liu, and Y . He. ReDiffuser: Reliable decision-making using a diffuser with confidence estimation. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pag...

    2024

  6. [6]

    Bardes, Q

    A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y . LeCun, M. Assran, and N. Ballas. Revisiting feature prediction for learning visual representations from video.arXiv preprint arXiv:2404.08471, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    M. Ho, M. F. Ginting, I. R. Ward, A. Reinke, M. J. Kochenderfer, A.-a. Agha-Mohammadi, and S. Omidshafiei. World model failure classification and anomaly detection for autonomous inspection, 2026. URLhttps://arxiv.org/abs/2602.16182

    arXiv 2026

  8. [8]

    Assran, A

    M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. Robert Hogan, D. Dugas, P. Bojanowski, V . Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y . Li, X. Ma, S. Chandar, F. Meier, Y . LeCun, M. Rabbat, and N. Ballas. V-jepa 2: Self- superv...

    Pith/arXiv arXiv 2025

  9. [9]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. InProceedings of the 31st International Conference on Neu- ral Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY , USA, 2017. Curran Associates Inc. ISBN 9781510860964

    2017

  10. [10]

    Algorithmic Learning in a Random World

    V . V ovk, A. Gammerman, and G. Shafer.Algorithmic Learning in a Random World. Springer, New York, NY , 2005. ISBN 978-0-387-00152-4. doi:10.1007/b106715

    work page doi:10.1007/b106715 2005

  11. [11]

    Diquigiovanni, M

    J. Diquigiovanni, M. Fontana, and S. Vantini. The importance of being a band: Finite-sample exact distribution-free prediction sets for functional data.Statistica Sinica, 35(2):853–871,

  12. [12]

    doi:10.5705/ss.202022.0087

    work page doi:10.5705/ss.202022.0087

  13. [13]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning.arXiv preprint arXiv:2306.03310, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    S. Tao, F. Xiang, A. Shukla, Y . Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y . Liu, T. kai Chan, Y . Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V . N. Rajesh, Y . W. Choi, Y .-R. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai.Robotics: Science and Systems, 2025

    2025

  15. [15]

    C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Mart ´ın-Mart´ın, C. Wang, G. Levine, W. Ai, B. Martinez, H. Yin, M. Lingelbach, M. Hwang, A. Hiranaka, S. Garlanka, A. Ay- din, S. Lee, J. Sun, M. Anvari, M. Sharma, D. Bansal, S. Hunter, K.-Y . Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. H. Tang, C. Tang, F. Xia, Y . Li, S. Savarese, H. Gweon...

    Pith/arXiv arXiv 2024

  16. [16]

    M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Shukor, D

    M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Ar- actingi, C. Pascal, M. Russi, A. Marafioti, S. Alibert, M. Cord, T. Wolf, and R. Cadene. Smolvla: A vision-language-action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,

    Pith/arXiv arXiv

  19. [19]

    doi:10.48550/arXiv.2410.24164

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.24164

  20. [20]

    FAST: Efficient Action Tokenization for Vision-Language-Action Models

    K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine. Fast: Efficient action tokenization for vision-language-action models.arXiv preprint arXiv:2501.09747, 2025. doi:10.48550/arXiv.2501.09747

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.09747 2025

  21. [21]

    Black, N

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke...

    2025

  22. [22]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023. 10

    Pith/arXiv arXiv 2023

  23. [23]

    Bjorck, N

    NVIDIA, J. Bjorck, N. C. Fernando Casta ˜neda, X. Da, R. Ding, L. J. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y . L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y . Xie, Y . Xu, Z. Xu, S. Ye, Z. Yu, ...

    2025

  24. [24]

    C. Agia, R. Sinha, J. Yang, Z. Cao, R. Antonova, M. Pavone, and J. Bohg. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. InProceedings of The 8th Conference on Robot Learning, volume 270 ofProceedings of Machine Learning Research, pages 689–723. PMLR, 2025

    2025

  25. [25]

    Pacaud, R

    P. Pacaud, R. Garcia, S. Chen, and C. Schmid. Scaling cross-environment failure reasoning data for vision-language robotic manipulation, 2026. URLhttps://arxiv.org/abs/2512. 01946

    2026

  26. [26]

    P. Yi, Y . Ma, W. Xu, Y . Hao, S. Gan, W. Li, and S. Zhong. Critic in the loop: A tri-system vla framework for robust long-horizon manipulation, 2026. URLhttps://arxiv.org/abs/ 2603.05185

    arXiv 2026

  27. [27]

    E. Zhou, Q. Su, C. Chi, Z. Zhang, Z. Wang, T. Huang, L. Sheng, and H. Wang. Code-as- monitor: Constraint-aware visual programming for reactive and proactive robotic failure de- tection, 2025. URLhttps://arxiv.org/abs/2412.04455

    arXiv 2025

  28. [28]

    Grislain, H

    C. Grislain, H. Rahimi, O. Sigaud, and M. Chetouani. I-failsense: Towards general robotic failure detection with vision-language models. InProceedings of the International Conference on Robotics and Automation (ICRA), 2026. URLhttps://arxiv.org/abs/2509.16072

    arXiv 2026

  29. [29]

    I. R. Ward, M. Ho, H. Liu, A. Feldman, J. Vincent, L. Kruse, S. Cheong, D. Eddy, M. J. Kochenderfer, and M. Schwager. Foundational world models accurately detect bimanual ma- nipulator failures, 2026. URLhttps://arxiv.org/abs/2603.06987

    arXiv 2026

  30. [30]

    H. Liu, Y . Zhang, V . Betala, E. Zhang, J. Liu, C. Ding, and Y . Zhu. Multi-task interactive robot fleet learning with visual world models, 2024. URLhttps://arxiv.org/abs/2410.22689

    arXiv 2024

  31. [31]

    Agarwal, A

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation.Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  33. [33]

    B. Wang, N. Sridhar, C. Feng, M. Van der Merwe, A. Fishman, N. Fazeli, and J. J. Park. This&that: Language-gesture controlled video generation for robot planning. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 12842–12849. IEEE, 2025

    2025

  34. [34]

    NVIDIA, A. Ali, J. Bai, M. Bala, Y . Balaji, A. Blakeman, T. Cai, J. Cao, T. Cao, E. Cha, Y .-W. Chao, P. Chattopadhyay, M. Chen, Y . Chen, Y . Chen, S. Cheng, Y . Cui, J. Diamond, Y . Ding, J. Fan, L. Fan, L. Feng, F. Ferroni, S. Fidler, X. Fu, R. Gao, Y . Ge, J. Gu, A. Gupta, S. Gururani, I. El Hanafi, A. Hassani, Z. Hao, J. Huffman, J. Jang, P. Jannaty...

    Pith/arXiv arXiv 2025

  35. [35]

    Larchenko, G

    I. Larchenko, G. Zarin, and A. Karnatak. Task adaptation of vision-language-action model: 1st place solution for the 2025 behavior challenge, 2025. URLhttps://arxiv.org/abs/ 2512.06951

    arXiv 2025

  36. [36]

    J. J. Kuffner and S. M. LaValle. Rrt-connect: An efficient approach to single-query path planning. InProceedings 2000 IEEE International Conference on Robotics and Automation (ICRA), volume 2, pages 995–1001, 2000. doi:10.1109/ROBOT.2000.844730

    work page doi:10.1109/robot.2000.844730 2000

  37. [37]

    Sucan, M

    I. A. S ¸ucan, M. Moll, and L. E. Kavraki. The Open Motion Planning Library.IEEE Robotics & Automation Magazine, 19(4):72–82, December 2012. doi:10.1109/MRA.2012.2205651. https://ompl.kavrakilab.org. 12 Appendix 7 More Implementation Details World-model feature extraction.We use V-JEPA 2-AC [8] as the action-conditioned world- model backbone, initialized ...

    work page doi:10.1109/mra.2012.2205651 2012