Pith. sign in

REVIEW 2 major objections 4 minor 50 references

Robots can recover how their body and cameras work from a few random self-probes, then act under novel viewpoints without fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Prepending a few self-generated random interaction clips as context lets VLA policies identify novel camera viewpoints and morphologies at test time and outperform multi-view baselines without parameter updates.

T0 review reviewed 2026-07-12 challenge →

load-bearing objection Clean, practical idea: use self-generated random probes as in-context system ID rather than task demos; controlled multi-view and real-robot results back the comparative claim without circularity. the 2 major comments →

arxiv 2606.26025 v3 pith:ZX3VUJRM submitted 2026-06-24 cs.RO cs.CV

In-Context World Modeling for Robotic Control

Siyin Wang , Junhao Shi , Senyu Fei , Zhaoyang Fu , Li Ji , Jingjing Gong , Xipeng Qiu This is my paper
classification cs.RO cs.CV
keywords vision-language-actionin-context learningsystem identificationrobot manipulationviewpoint generalizationworld modelingtest-time adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard vision-language-action policies treat camera pose, robot shape, and similar system details as fixed constants baked into training. When those details change at deployment, performance collapses and expensive re-training is required. This paper shows that the same transformer already used for action prediction can instead treat a short history of the robot's own random, task-agnostic movements as a context window that reveals the current system configuration. After a few seconds of self-probing, the model implicitly rebuilds the missing mapping between actions and observations and proceeds with the real task, without any parameter updates or task-specific demos. Simulation and real-robot results show large gains on held-out camera angles and further resilience to table-texture changes and altered link lengths. The practical claim is that generalist robots can self-calibrate on the fly rather than relying on human intervention for every new setup.

Core claim

System identification for robot control can be cast as in-context inference: a short prefix of self-generated, task-agnostic observation-action-observation clips is sufficient for a shared VLA backbone to recover a usable representation of the latent configuration, so the policy can adapt its actions to novel viewpoints (and, to a lesser extent, semantic and morphological shifts) without any gradient steps or expert demonstrations.

What carries the argument

In-Context World Modeling (ICWM): the interaction context T of N random probing transitions is prepended to the task query; the same transformer that predicts actions attends to T first, inducing a configuration-aware hidden state Ψ(T) that then conditions action generation.

Load-bearing premise

A handful of purely random movements performed away from task objects always carry enough information for the model to recover a usable picture of the current cameras and body geometry.

What would settle it

Run the same trained model on a held-out camera angle after replacing the random probes with either (a) no probes or (b) probes recorded under a deliberately wrong camera; if success rates stay high under (a) or do not collapse under (b), the claim that the model is truly reading configuration from the context is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that standard VLA policies of the form π(at|ot,l) fail under novel system configurations ψ (camera viewpoint, morphology) because they never condition on ψ. It proposes In-Context World Modeling (ICWM): before task execution the robot collects a short sequence of self-generated, task-agnostic probing transitions T={(os_i,ai,oe_i)}, prepends them as context, and lets a shared transformer backbone form an implicit representation Ψ(T) that recovers the current ψ, yielding the adapted policy π(at|ot,l,Ψ(T)) without parameter updates or task demonstrations. Training simply prepends random multi-view interaction clips to ordinary next-action prediction. Controlled experiments on LIBERO (8 train / 6 OOD azimuths) and a 12-camera UR5e platform show consistent gains over multi-view BC and explicit-angle baselines; ablations, false-context controls, probing-strategy variants, t-SNE of Ψ(T), and further tests under semantic and morphological shifts support the system-identification interpretation.

Significance. If the result holds, ICWM supplies a practical, demonstration-free route to test-time adaptation for VLAs that is orthogonal to scale and multi-view data collection. The contribution is cleanly isolated by the multi-view BC and explicit-configuration baselines, by the false-context negative-transfer result, and by the morphological generalization experiments (spacers on UR5e and link-length interpolation on WindowX). The information-theoretic motivation (Prop. 1 under A1–A2) and the zero-extra-parameter design are additional strengths. Absolute OOD success rates remain modest on hard suites, yet the comparative claim is well supported and the protocol is immediately usable.

major comments (2)
  1. Absolute OOD numbers on LIBERO-Object (15.9 %) and LIBERO-Long (25.0 %) remain low even for ICWM (Table 4). The paper correctly emphasizes relative gains over MV/EXP, but the abstract and conclusion should state more clearly that the method mitigates rather than solves viewpoint generalization; otherwise readers may over-interpret the practical readiness of the approach.
  2. The real-robot evaluation (Fig. 5, 600 trials) reports only the multi-view BC baseline. Adding the explicit-configuration (EXP) condition already present in simulation would strengthen the claim that interaction context is superior to simply telling the model the camera angle, and would make the real-world evidence fully parallel to the simulation design.
minor comments (4)
  1. Figure 4 caption and the surrounding text use both “+13.0 %” and suite-specific percentages; a single consistent reporting convention (absolute points vs. relative) would improve readability.
  2. Proposition 1 and Appendix A are clear, but a short intuitive sentence in the main text explaining why random actions still increase I(ψ;T) would help readers who skip the appendix.
  3. N=5 is fixed throughout; a brief sensitivity plot or table for N∈{1,3,5,10} would be useful, even if only in the appendix.
  4. A few typographical issues remain (e.g., “ithe”, “T ongji”, occasional missing spaces around equations).

Circularity Check

0 steps flagged

No significant circularity: empirical next-action training with prepended clips evaluated on held-out configurations

full rationale

The paper's central claim is an empirical engineering result: a standard VLA backbone trained by ordinary next-token (action-chunk) prediction on multi-view trajectories, with randomly sampled task-agnostic interaction clips prepended as context, yields higher success rates on held-out camera viewpoints, semantic distractors, and morphological offsets than an identical multi-view baseline without context (and than an explicit-angle baseline). Proposition 1 is a short mutual-information argument under two mild assumptions (partial observability and information-preserving transitions) whose proof is self-contained in the appendix and does not feed back into any fitted quantity. Training loss (Eq. 7) is ordinary negative log-likelihood; no parameter is fitted to a subset of the evaluation metric and then re-labeled a prediction. Self-citations (e.g., prior world-modeling surveys or LIBERO-plus) appear only as background and are not load-bearing for the adaptation claim. Ablations (false context hurts, different probing strategies all help, t-SNE clusters by viewpoint) further confirm that the model conditions on the actual content of T rather than on a definitional identity. Absolute OOD numbers remain modest, but that is a performance limitation, not circularity. The derivation chain therefore contains no self-definitional step, no fitted-input-called-prediction, and no uniqueness or ansatz smuggled via self-citation.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on a small set of modeling assumptions (POMDP decomposition, partial observability, information-preserving transitions) plus a handful of free design choices (context length, probing distribution, shared backbone). No new physical entities are postulated; the invented objects are algorithmic constructs whose utility is measured by the reported success rates.

free parameters (4)
  • N (number of context clips) = 5
    Fixed at 5 for main results; latency and success both depend on this hand-chosen integer.
  • action chunk size = 5
    Set to 5; affects temporal smoothness and how much of the context is used per step.
  • peak learning rate and warmup = 5e-5
    5e-5 after 50k-step warmup; standard but free training hyper-parameter that influences final performance.
  • probing workspace bounds
    Defined from forward kinematics to keep motions task-object-free; the exact box is a free design choice that determines the information content of T.
axioms (4)
  • domain assumption Latent state decomposes as sk = (ψ, ξk) with ψ time-invariant system configuration
    Stated in Sec. 3.2; required for the claim that interaction history can recover ψ.
  • domain assumption A1: partial observability H(sk|ok) > 0
    Used in Prop. 1 (App. A) to argue a single image is insufficient.
  • domain assumption A2: information-preserving transitions I(s0; sk | a1:k) > 0
    Used in Prop. 1 to guarantee that random actions still enrich information about ψ.
  • ad hoc to paper Shared transformer parameters can implement both action prediction and configuration inference
    Design choice in Sec. 4.1 that lets Ψ share weights with πθ; not forced by prior theory.
invented entities (2)
  • In-Context World Modeling (ICWM) protocol no independent evidence
    purpose: Framework that prepends self-generated interaction clips for test-time system identification
    The paper’s central algorithmic contribution; evaluated only inside this work.
  • Ψ(T) — implicit configuration representation no independent evidence
    purpose: Hidden states induced by the interaction context that stand in for the latent ψ
    Defined as the transformer’s intermediate states after attending to T; no external measurement of Ψ is provided.

reviewed 2026-07-12 · how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context World Modeling for Robotic Control." pith.science (2026). https://pith.science/paper/ZX3VUJRM

@misc{pith2026260626025,
  author       = {Pith},
  title        = {Pith review of: In-Context World Modeling for Robotic Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZX3VUJRM}},
  note         = {Machine review of arXiv:2606.26025}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 26 linked inside Pith

  1. [1]

    Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalew...

    2023

  2. [2]

    Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov , Ethan Foster, Grace Lam, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. ArXiv, abs/2406.09246, 2024. ...

    Pith/arXiv arXiv 2024

  3. [3]

    π0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language-action flow model for general robot control. corr, abs/2410.24164, 2024. doi: 10.48550. arXiv preprint ARXIV .2410.24164

    Pith/arXiv arXiv 2024

  4. [4]

    Senyu Fei, Siyin Wang, Junhao Shi, Z. G. Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models. ArXiv, abs/2510.13626, 2025. URL https://api.semanticscholar.org/CorpusID:282102298

    Pith/arXiv arXiv 2025

  5. [5]

    Goldberg

    Letian Fu, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, Will Panitch, Fangchen Liu, Hui Li, and Ken- neth Y . Goldberg. Icrt: In-context imitation learning via next-token prediction. 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 5937–5944, 2024. URL https://api.semanticscholar.org/CorpusID: 271974730

    2025

  6. [6]

    Mimicdroid: In-context learning for humanoid robot manipulation from human play videos

    Rutav Shah, Shuijing Liu, Qi Wang, Zhenyu Jiang, Sateesh Kumar, Mingyo Seo, Roberto Mart’in-Mart’in, and Yuke Zhu. Mimicdroid: In-context learning for humanoid robot manipulation from human play videos. ArXiv, abs/2509.09769, 2025. URL https://api.semanticscholar.org/CorpusID:281309736

    Pith/arXiv arXiv 2025

  7. [7]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533

    2019

  8. [8]

    Llama: Open and efficient foundation language models

    Hugo T ouvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023. URL https://api.semanticscholar.org/Co...

    Pith/arXiv arXiv 2023

  9. [9]

    Introducing chatgpt, 2022

    OpenAI. Introducing chatgpt, 2022. URL https://openai.com/blog/chatgpt

    2022

  10. [10]

    T om B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry , Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Li...

    Pith/arXiv arXiv 2005

  11. [11]

    URL https://api.semanticscholar.org/CorpusID:218971783

  12. [12]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://api.semanticscholar.org/CorpusID:255372865

    2022

  13. [13]

    Vuong Dinh An, Minh Nhat Vu, Dong An, and Ian D. Reid. Action tokenizer matters in in-context imitation learning. 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13490–13496, 2025. URL https://api.semanticscholar.org/CorpusID:276742267. 11

    2025

  14. [14]

    Ricl: Adding in-context adaptability to pre-trained vision-language-action models

    Kaustubh Sridhar, Souradeep Dutta, Dinesh Jayaraman, and Insup Lee. Ricl: Adding in-context adaptability to pre-trained vision-language-action models. ArXiv, abs/2508.02062, 2025. URL https://api.semanticscholar. org/CorpusID:280422322

    Pith/arXiv arXiv 2025

  15. [15]

    Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R

    Vidhi Jain, Maria Attarian, Nikhil J. Joshi, Ayzaan Wahid, Danny Driess, Quan Vuong, Pannag R. Sanketi, Pierre Sermanet, Stefan Welker, Christine Chan, Igor Gilitschenski, Yonatan Bisk, and Debidatta Dwibedi. Vid2robot: End- to-end video-conditioned policy learning with cross-attention transformers. ArXiv, abs/2403.12943, 2024. URL https://api.semanticsch...

    Pith/arXiv arXiv 2024

  16. [16]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  17. [17]

    Meta-learning with implicit gradients

    Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019

    2019

  18. [18]

    Rl 2: Fast reinforcement learning via slow reinforcement learning

    Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016

    Pith/arXiv arXiv 2016

  19. [19]

    Varibad: A very good method for bayes-adaptive deep rl via meta-learning

    Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon White- son. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348, 2019

    Pith/arXiv arXiv 1910

  20. [20]

    Recurrent world models facilitate policy evolution

    David R Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Neural Information Processing Systems, 2018. URL https://api.semanticscholar.org/CorpusID:52171619

    2018

  21. [21]

    A path towards autonomous machine intelligence version 0.9.2, 2022-06-27

    Yann LeCun and Courant. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. 2022. URL https://api.semanticscholar.org/CorpusID:251881108

    2022

  22. [22]

    Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, and Yong Li

    Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, J. Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, and Yong Li. Understanding world or predicting future? a comprehensive survey of world models. ACM Computing Surveys, 58:1 – 38, 2024. URL https://api.semanticscholar.org/CorpusID: 274192171

    2024

  23. [23]

    World modeling makes a better planner: Dual preference optimization for embodied task planning

    Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes a better planner: Dual preference optimization for embodied task planning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21518–21537, 2025

    2025

  24. [24]

    World action models: The next frontier in embodied ai

    Siyin Wang, Junhao Shi, Zhaoyang Fu, Xinzhe He, Feihong Liu, Chenchen Yang, Yikang Zhou, Zhaoye Fei, Jingjing Gong, Jinlan Fu, et al. World action models: The next frontier in embodied ai. arXiv preprint arXiv:2605.12090, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chi-Hou Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. ArXiv, abs/2312.13139, 2023. URL https://api.semanticscholar.org/CorpusID:266374724

    Pith/arXiv arXiv 2023

  26. [26]

    Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy.IEEE Robotics and Automation Letters, 10:1912–1919, 2024

    Peiyan Li, Hongtao Wu, Yan Huang, Chi-Hou Cheang, Liang Wang, and Tao Kong. Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy.IEEE Robotics and Automation Letters, 10:1912–1919, 2024. URL https://api.semanticscholar.org/CorpusID:271957548

    1912

  27. [27]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming-Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung- Yi Lin. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pa...

    2025

  28. [28]

    Worldvla: T owards autoregressive action world model

    Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, Deli Zhao, and Hao Chen. Worldvla: T owards autoregressive action world model. ArXiv, abs/2506.21539,

    Pith/arXiv arXiv

  29. [29]

    URL https://api.semanticscholar.org/CorpusID:280010695

  30. [30]

    Lillicrap, Jimmy Ba, and Mohammad Norouzi

    Danijar Hafner, Timothy P . Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. ArXiv, abs/1912.01603, 2019. URL https://api.semanticscholar.org/CorpusID:208547755. 12

    Pith/arXiv arXiv 1912

  31. [31]

    Lillicrap, Mohammad Norouzi, and Jimmy Ba

    Danijar Hafner, Timothy P . Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ArXiv, abs/2010.02193, 2020. URL https://api.semanticscholar.org/CorpusID:222133157

    Pith/arXiv arXiv 2010

  32. [32]

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and P . Abbeel. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, 2022. URL https://api.semanticscholar.org/ CorpusID:250088882

    2022

  33. [33]

    Pašukonis, Jimmy Ba, and Timothy P

    Danijar Hafner, J. Pašukonis, Jimmy Ba, and Timothy P . Lillicrap. Mastering diverse domains through world models. ArXiv, abs/2301.04104, 2023. URL https://api.semanticscholar.org/CorpusID:255569874

    Pith/arXiv arXiv 2023

  34. [34]

    Flare: Robot learning with implicit world modeling

    Ruijie Zheng, Jing Wang, Scott Reed, Johan Bjorck, Yu Fang, Fengyuan Hu, Joel Jang, Kaushil Kundalia, Zongyu Lin, Loic Magne, Avnish Narayan, You Liang Tan, Guanzhi Wang, Qi Wang, Jiannan Xiang, Yinzhen Xu, Seonghyeon Ye, Jan Kautz, Furong Huang, Yuke Zhu, and Linxi Fan. Flare: Robot learning with implicit world modeling. ArXiv, abs/2505.15659, 2025. URL h...

    Pith/arXiv arXiv 2025

  35. [35]

    Tenenbaum, Dale Schuurmans, and P

    Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and P . Abbeel. Learning universal policies via text-guided video generation. ArXiv, abs/2302.00111, 2023. URL https://api.semanticscholar.org/CorpusID:256459809

    Pith/arXiv arXiv 2023

  36. [36]

    Zettlemoyer, Di- eter Fox, Jan Kautz, Scott Reed, Yuke Zhu, and Linxi Fan

    Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, Avnish Narayan, You Liang Tan, Guanzhi Wang, Jing Wang, Qi Wang, Yinzhen Xu, Xi Zeng, Kaiyuan Zheng, Ruijie Zheng, Ming-Yu Liu, Luke S. Zettlemoyer, Di- eter Fox, Jan Kautz, Scott Reed, Yuke Zhu...

    2025

  37. [37]

    Predictive inverse dynamics models are scalable learners for robotic manipulation

    Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation. ArXiv, abs/2412.15109, 2024. URL https: //api.semanticscholar.org/CorpusID:274859727

    Pith/arXiv arXiv 2024

  38. [38]

    Video pretraining (vpt): Learning to act by watching unlabeled online videos

    Bowen Baker, Ilge Akkaya, Peter Zhokhov , Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. ArXiv, abs/2206.11795, 2022. URL https://api.semanticscholar.org/CorpusID:249953673

    Pith/arXiv arXiv 2022

  39. [39]

    Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets

    Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. ArXiv, abs/2504.02792, 2025. URL https://api.semanticscholar.org/CorpusID:277510147

    Pith/arXiv arXiv 2025

  40. [40]

    Unified video action model

    Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. ArXiv, abs/2503.00200, 2025. URL https://api.semanticscholar.org/CorpusID:276741531

    Pith/arXiv arXiv 2025

  41. [41]

    LIBERO: bench- marking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: bench- marking knowledge transfer for lifelong robot learning. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Proce...

    2023

  42. [42]

    Nora: A small open-sourced generalist vision language action model for embodied tasks

    Chia-Yu Hung, Qi Sun, Pengfei Hong, Amir Zadeh, Chuan Li, U-Xuan Tan, Navonil Majumder, and Soujanya Poria. Nora: A small open-sourced generalist vision language action model for embodied tasks. ArXiv, abs/2504.19854,

    Pith/arXiv arXiv

  43. [43]

    URL https://api.semanticscholar.org/CorpusID:278165428

  44. [44]

    FAST : efficient action tokenization for vision-language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST : efficient action tokenization for vision-language-action models. CoRR, abs/2501.09747, 2025. doi: 10.48550/ARXIV.2501.09747. URL https://doi.org/10.48550/arXiv.2501.09747

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.09747 2025

  45. [45]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsc...

    Pith/arXiv arXiv 2025

  46. [46]

    pick up the black bowl next to the plate and place it on the plate

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. Ar...

    Pith/arXiv arXiv 2025

  47. [47]

    Put the toy on the box into the basket

    Spatial Reasoning & Disambiguation: “Put the toy on the box into the basket.” This task requires the agent to understand the vertical spatial relationship between the toy and the box, necessitating precise end- effector positioning to take the toy without disturbing the support surface ia process highly sensitive to viewpoint-induced depth errors 19 Camer...

  48. [48]

    Stack the yellow cup onto the red cup

    Fine-grained Alignment: “Stack the yellow cup onto the red cup.” This serves as a benchmark for high-precision motor control, where the agent must align the principal axes of two objects under novel perspective projections

  49. [49]

    Lift the basket

    Structural Manipulation: “Lift the basket.” This task focuses on handle-centric grasping of large-scale empty containers, testing the model’s ability to ground actions on specific structural af- fordances of an object

  50. [50]

    Pick up the eggplant and place it onto the red plate

    Multi-Object Semantic Grounding: “Pick up the eggplant and place it onto the red plate.” Con- ducted in a cluttered scene with multiple distrac- tor objects, this task assesses the model’s ability to maintain correct object-instruction alignment when viewed from unfamiliar angles that may cause occlusion or visual overlap. For task-specific knowledge, we ...

This paper was first reviewed by grok-4.5 on July 12, 2026.