Pith. sign in

REVIEW 4 major objections 5 minor 138 references

PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A session-level OS layer lifts task success for every robot policy tested.

desk verdict A serious integration proposal whose headline gains rest on a First-vs-Final comparison with no retry-only control; deserves a referee who will demand that control, verifier calibration, and the missing safety numbers. read the letter →

arxiv 2607.16636 v1 pith:7ATJHOM3 submitted 2026-07-18 cs.RO

Yang Liu , Weixing Chen , Xinshuai Song , Tao Pu , Siwen Mo , Yongjie Bai , Zihao Chen , Qianran Sun
show 3 more authors
Liruo Zhong Ying Shen Liang Lin
This is my paper · ORCID
classification cs.RO
keywords EmbodiedAIrobotoperatingsystemvision-language-actionmodelssemanticverificationself-evolutionsessionschedulingState-as-a-Fileepisodicmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PhyAgentOS argues that many failures of embodied policies are recoverable execution errors rather than task misunderstandings, and that a runtime layer can catch and correct them without retraining any neural model. It treats a session, not an action, as the unit of scheduling, verification, and memory, and puts a file-based protocol between cognitive planning and physical execution. A SessionVerifier judges whether a finished run actually satisfied the task, returning success, failure, or replan, and verified outcomes feed persistent memory. Across LIBERO, CALVIN, and RoboCasa365, the layer raises final success for every policy backend tested, up to +9.2 points on RoboCasa365. The claim matters because it makes verification, memory, benchmarking, and safety system services rather than model properties.

What carries the argument

The Session-Centered Runtime makes the session the minimum unit of scheduling, preflight, supervision, evidence collection, and acceptance, supervised by a WatchdogSupervisor that never runs the low-level loop itself. The cognition-physics boundary is a file system: State-as-a-File materializes cross-layer state (sessions, targets, skills, environment, lessons, knowledge) as human-readable Markdown with embedded YAML, giving an append-only, versionable audit trail with no code dependency between Agent and Runtime. The SessionVerifier is the judgment function defined above, and it converts termination into semantic acceptance; epistemic memory then turns verdicts into reusable knowledge (KNOW

What would settle it

Run the RoboCasa365 recovery protocol but replace the verifier's verdicts with random or inverted ones; if the '+9.2' gain persists, the verifier's semantic judgment is not doing the work. Alternatively, have independent human annotators judge the same evidence bundles and compare; if agreement is low, the benchmark gains are not evidence of task-level recovery.

Watch

Extended reading notes

Core claim

The paper's central claim is that a large fraction of embodied-policy failures are not fundamental incapabilities but recoverable execution errors that can be detected by semantic verification of the achieved world state and corrected by controlled continuation from the current physical state. The authors build an operating-system layer that decouples cognition from physics: the Agent emits session contracts, the Runtime executes them under watchdog supervision, and a SessionVerifier distinguishes execution termination from task completion using an evidence bundle (initial and terminal observations, environment snapshot, trace, and acceptance criteria) rendered through V(G,S0,ST,τ,H) → {succ

Load-bearing premise

The whole loop presupposes that SessionVerifier verdicts are correct: the paper reports no calibration of V against human labels, so a verifier that falsely labels failures as successes would inflate both the 'rescued' episodes and the lessons stored in memory.

Editorial extensions

If this is right

  • A single runtime layer can raise final success of many VLA families on both tabletop and household benchmarks without modifying policy weights or task goals.
  • Recovery matters most for the hardest long-horizon cases: CALVIN full chains gain 4.1–6.7 points and RoboCasa365 composite activities gain 5.6–9.4 points.
  • Because benchmarking reuses the deployment session and verification path, reported results are traceable to real execution rather than a separate harness.
  • Self-evolution is achieved at the system level: later sessions are informed by verified outcomes, so capability accumulates without retraining.
  • Layered safety separates safety interventions from task failures in the evidence record, which preserves the meaning of verifier verdicts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SessionVerifier verdicts were calibrated against human judgments on a sample of episodes, the reported gains could be decomposed into true recoveries versus verifier errors; a random-verifier ablation would bound how much of the gain is due to semantic judgment rather than to the retry mechanism alone.
  • The same evidence schema could let world-model predictions serve as virtual evidence, so the verifier could pre-judge proposed actions before physical commitment, reducing costly physical trial-and-error.
  • The paper's own limitation list flags monotonic memory growth and a polling-based protocol; a third caution the authors do not state is that DST-Dojo's benchmark citation is a placeholder reference, so that part of the evaluation rests on unreleased concurrent work, and real-robot results emphasize safety validation over task-completion statistics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. PhyAgentOS is presented as a runtime/operating-system layer for embodied agents that treats a session, rather than an atomic action, as the unit of scheduling, verification, memory, and safety. The paper introduces a State-as-a-File protocol (Markdown/YAML documents) as the cognition-physics boundary, a SessionVerifier that issues success/failure/replan verdicts from evidence bundles, an epistemic-memory loop that consolidates verified outcomes into reusable knowledge, and a layered safety architecture. Validation is progressive: game environments (Optimus-67, StarDojo, DST-Dojo), simulation benchmarks (LIBERO, CALVIN, RoboCasa365), and real-hardware deployment. The central empirical claim is that, without retraining any policy, PhyAgentOS improves final task performance for every evaluated VLA backend, with absolute gains up to +9.2 points on RoboCasa365.

Significance. If the mechanism isolation were clean, this would be a meaningful contribution: a model-agnostic runtime layer that improves embodied-task performance through verification and memory, anchored to external benchmark success criteria. The architecture is ambitious and the progressive-validation strategy is sensible. Strengths include the use of external benchmarks as the success anchor, a deployment-identical benchmarking path, and an open GitHub repository. However, the headline First-vs-Final comparisons conflate the proposed mechanism with the generic effect of additional attempts, and the SessionVerifier—which triggers every recovery and labels every lesson—is never calibrated. The current evidence therefore does not yet isolate the mechanisms that define PhyAgentOS.

major comments (4)
  1. [§5.2 (First/Final protocol), Tables 4–6] The central evidence for the headline claim is the First-vs-Final comparison. In the Final condition, the SessionVerifier may trigger a recovery attempt after a failed first attempt; in the First condition it may not. The paper reports no retry-only control in which the same policy receives the same additional attempts without PhyAgentOS. The reported gains (+0.4 to +1.3 on LIBERO, +0.019 to +0.124 Avg. Len. on CALVIN, +7.2 to +9.2 on RoboCasa365, and 18–23 rescued episodes) therefore conflate verifier/memory benefits with the trivial effect of being allowed to try again. A retry-only baseline with matched attempt counts, plus explicit reporting of the recovery budget per episode, is needed to isolate the proposed mechanism.
  2. [§4.2 (SessionVerifier), §5.2.3 (RoboCasa365)] The SessionVerifier is the trigger for every recovery and the labeler of every lesson, but its accuracy is never measured. The paper reports no false-positive/false-negative rates, no human agreement, and no independent audit of the verdict function V(G,S0,ST,τ,H). In §5.2.3 the verifier 'may either overturn a false-negative completion judgment' even though success is defined by the environment completion signal. Without knowing how many Final successes were relabeled by the uncalibrated verifier versus accepted by the external signal, the 'rescued episode' counts cannot be interpreted. A calibration study is load-bearing for both the benchmark gains and the self-evolution loop.
  3. [§5.1.2–§5.1.3 (Optimus-67, StarDojo)] The game-tier attribution is also verifier-dependent. Table 2 attributes the StarDojo Crafting improvement partly to the SessionVerifier 'accepts partially completed craft sequences that reactive baselines reject,' and §5.1.2 attributes RedStone gains to detecting 'partial circuit constructions that binary completion metrics would count as failures.' If the official benchmark success criterion is full completion, then PhyAgentOS is using altered acceptance criteria, making gains over baselines not directly comparable. The paper should state explicitly whether the reported success uses the official benchmark criterion or a PhyAgentOS-specific acceptance criterion, and report both.
  4. [§4.3, §5.1.5, Table 3] The self-evolution claim—that later sessions are informed by verified outcomes—is not isolated from simple retrying. The game-tier evidence is either qualitative or based on very small samples: Table 3 reports 10 episodes, and the DST-Dojo change in death-by-darkness from 90% to 80% is within the likely noise of that sample. There is no no-memory ablation that removes LESSONS.md/KNOWLEDGE.md retrieval while keeping the same recovery loop. Without such an ablation, the memory-consolidation mechanism is not independently supported.
minor comments (5)
  1. [Table 3] The 'Raw LLM + PhyAgentOS' column is malformed: e.g., '1.02±0.082.10±0.88' lacks a delimiter between baseline and PhyAgentOS values. The row label is also ambiguous.
  2. [Table 6] '23 Final' in the Rescued column appears to be a formatting error; it should likely be '23' aligned with the Final row.
  3. [References / §6.3] Reference [83] for DST-Dojo is a placeholder ('arXiv:placeholder'), and §6.3 refers to it as concurrent work. This must be fixed before publication.
  4. [Throughout] Naming is inconsistent: 'Calvin' appears in §5.2.4 while the benchmark is 'CALVIN'; 'DeepSeek v4 Flash' in §5.1.4 vs. 'deepseek-v4-flash' in §5.1.1. Figure 9 also contains the typo 'MENDATORY'.
  5. [Appendix C] Appendix C promises full YAML schemas, a state-machine transition table, and SessionVerifier verdict rules, but these are not included in the manuscript. If they are part of supplementary material, state so explicitly; otherwise include them.

Circularity Check

2 steps flagged · score 5.0 of 10

Simulation-tier 'improvements' are partly constructed: Final is defined as First plus verifier-gated retries, and in RoboCasa365 the uncalibrated SessionVerifier can overturn the external completion signal, so some headline gains reduce to the verifier labeling itself correct.

  1. self definitional [§5.2.3 (RoboCasa365), Table 6]
    "Success is measured by the environment completion signal. ... The verifier may either overturn a false-negative completion judgment or produce a recovery instruction that allows execution to continue from the current physical state."

    The reported 'Final' success includes 18–23 'rescued' episodes in which the SessionVerifier may have overturned the environment completion signal, i.e., declared success where the external benchmark signal says failure. The SessionVerifier is the very mechanism whose benefit the paper claims to demonstrate, and it is never calibrated (no false-positive/false-negative rates, no human agreement, no independent audit). If those overturned episodes are counted as successes, part of the headline gain (+9.2, +7.2, +8.4) is produced by the system's own acceptance verdict rather than by the external completion signal. The paper does not report how many Final successes were accepted by the external signal versus relabeled by the verifier, so the self-defined portion cannot be separated.

  2. other [§5.2 introduction (dual evaluation protocol), Tables 4–6, §5.2.4]
    "The First setting measures the policy’s original first-attempt performance. The Final setting measures performance after PhyAgentOS intervention, where the verifier is invoked only after failure and may trigger a controlled recovery attempt from the current physical state."

    Final is defined as First plus verifier-triggered recovery attempts. For the same episode, the Final success set is therefore First's successes plus any recovered episodes, so a positive gain is structurally built into the protocol whenever a retry succeeds. No retry-only control is reported, so the First-vs-Final contrast cannot distinguish 'semantic verification and memory helped' from 'extra stochastic rollouts helped.' The conclusion in §5.2.4 that the gains 'support the central premise ... recoverable execution errors that can be identified by semantic verification' attributes to PhyAgentOS's mechanisms an effect that is at least partly an artifact of the protocol's definition.

full rationale

The paper is not circular in the self-citation sense: no load-bearing argument reduces to an unverified citation by the same authors, and most benchmark numbers are anchored by external LIBERO, CALVIN, and RoboCasa365 signals. The circularity is narrower and definitional, and it affects the simulation-tier evidence that carries the central claim. The dual evaluation protocol defines 'Final' as 'First plus verifier-triggered recovery,' so the reported gains are not an independent measurement of verification or memory; they are statistically entangled with the generic effect of allowing additional attempts after failure. In RoboCasa365 this is compounded by the SessionVerifier being permitted to overturn the environment completion signal. Because the verifier is uncalibrated and is both the judge of failure and the trigger for rescue, any rescued episode counted on the verifier's authority is a self-accepted success rather than an externally confirmed one. The paper's own §4.4 says that 'Agent-assisted retry can be reported as a distinct protocol, but it is not conflated with first-attempt policy performance,' yet §5.2's headline comparison is exactly First-vs-Final with retries included. Section 6.2 acknowledges polling latency and real-robot coverage limits, but not the missing retry-only control. These issues make the central mechanistic claim partially self-defined, though not wholly: most episodes are still judged by external signals, so the overall result is not a pure renaming of inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

Central results rest on design choices rather than on mathematical parameters: no explicit numbers are fitted to data, but several hand-chosen, unreported mechanisms (recovery budget, acceptance criteria, memory retrieval heuristic, skill-promotion threshold) determine the reported gains. The axioms are domain assumptions about file-based state and verifier reliability, with no formal proof. No new physical or mathematical entities are introduced; the runtime components and protocol documents are software artifacts that are not independently audited in this manuscript.

free parameters (4)
  • Recovery attempt budget = unreported
    The Final condition allows verifier-triggered recovery after failure; the number of permitted attempts determines most of the gains in Tables 4–6 but is never stated.
  • SessionVerifier acceptance criteria / verifier configuration = per-task, unreported
    Acceptance criteria decide what counts as success/failure/replan; in StarDojo the verifier 'accepts partially completed craft sequences', which can change the effective success metric relative to baselines.
  • Memory retrieval selection heuristic = task-type matching + recency weighting
    ContextBuilder's retrieval rule shapes the self-evolution gains; §6.2 admits it has no theoretical guarantees and no ablation isolates its contribution.
  • Skill promotion success-rate threshold = unreported
    §6.2 says strategies are promoted to SKILL.md based on success-rate thresholds, but thresholds and regression procedures are not specified.
assumptions (4)
  • domain assumption A Markdown/YAML file boundary can carry enough cross-layer state (intent, environment, evidence, lessons) for semantic verification without losing control fidelity.
    Assumed throughout §3.6/§4.1; no formal capacity or latency analysis is provided.
  • domain assumption The SessionVerifier produces correct semantic verdicts across tasks, tiers, and embodiments.
    §4.2 defines the judging interface, but the paper reports no verifier accuracy, inter-rater agreement, or false-positive/false-negative rates.
  • domain assumption External benchmark success signals are valid ground truth in the Final setting despite verifier-assisted continuation.
    §5.2 uses environment completion signals as success, but recovery started from the same physical state may not satisfy the benchmark's intended reset or evaluation conditions.
  • ad hoc to paper Polling-based file access preserves atomicity and timing needed for safe physical execution.
    §6.2 acknowledges polling latency and says event-driven variants must preserve single-writer atomic semantics, but no correctness proof is supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution." pith.science (2026). https://pith.science/paper/7ATJHOM3

@misc{pith2026260716636,
  author       = {Pith},
  title        = {Pith review of: PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ATJHOM3}},
  note         = {Machine review of arXiv:2607.16636}
}
read the original abstract

Vision-language-action models, world models, and agentic planners each advance physical intelligence, yet their composition lacks a common execution abstraction, shared state, semantic verification, and persistent experience across heterogeneous embodiments. We present PhyAgentOS, a runtime foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services. Its Session-Centered Runtime treats a session, not an action, as the minimum unit of scheduling, compatibility preflight, supervised execution, evidence collection, and acceptance. To decouple cognition from physical execution, the cognition-physics boundary is a file system: the State-as-a-File protocol materializes cross-layer state as Markdown with YAML, yielding inspectable, versionable records without code dependencies between Agent and Runtime layers. These views form a unified cognitive state space aligning intent, capabilities, environment, execution, and experience. The SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan. Verified outcomes are consolidated through epistemic memory into reusable knowledge and corrective lessons, closing a trial-and-error loop without retraining. Benchmarking reuses the deployment session and verification path, so results trace to real execution. Layered safety constrains both policy-driven and agent-driven execution: preflight, action bridges, SafetyGuard, heartbeat monitoring, and target-local constraints. Validation is progressive: games test cognitive planning, simulation adds dynamics and control, real robots add hardware noise, with the cognitive layer held constant. PhyAgentOS is benchmarked on Optimus-67, StarDojo, and DST-Dojo, validated on 19+ simulated and physical embodiments, and gains on LIBERO, Calvin, and RoboCasa365 across multiple VLA models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

138 extracted references · 6 canonical work pages

  1. [1]

    34 PhyAgentOS v0.1.6 2026-07-21

    An architecture for autonomy.The International Journal of Robotics Research, 17(4):315–337, 1998. 34 PhyAgentOS v0.1.6 2026-07-21

    1998

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. InConference on Robot Learning, 2022

    2022

  3. [3]

    AutoRT: Embodied foundation models for large scale orchestration of robotic agents.arXiv preprint arXiv:2401.12963, 2024

    Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng ...

    doi:10.48550/arxiv.2401.12963 2024

  4. [4]

    Bridge-WA: Predicting Where and How the World Changes for Robotic Action

    Yongjie Bai, Hanting Wang, Mingtong Dai, Qijun Zhong, Yang Liu, and Liang Lin. Bridge- WA: Predicting where and how the world changes for robotic action.arXiv preprint arXiv:2607.02195, 2026. doi: 10.48550/arXiv.2607.02195. URL https://arxiv.org/abs/ 2607.02195

    work page Pith review arXiv doi:10.48550/arxiv.2607.02195 2026

  5. [5]

    Learning to see and act: Task- aware virtual view exploration for robotic manipulation

    Yongjie Bai, Zhouxia Wang, Yang Liu, Kaijun Luo, Yifan Wen, Mingtong Dai, Weixing Chen, Ziliang Chen, Lingbo Liu, Guanbin Li, and Liang Lin. Learning to see and act: Task- aware virtual view exploration for robotic manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. URL https://arxiv.org/ abs/2508.05186...

    arXiv 2026

  6. [6]

    Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823, 2024

    arXiv 2024

  7. [7]

    Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023

    Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking, 2023

    2023

  8. [8]

    pi0: A vision-language- action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi0: A vision-language- action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    arXiv 2024

Show all 138 references
  1. [9]

    Robocat: A self-improving generalist agent for robotic manipulation.Transactions on Machine Learning Research

    Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Manon Devin, Alex X Lee, Maria Bauza Villalonga, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-improving generalist agent for robotic manipulation.Transactions on Machine Learning Research

  2. [10]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    2016

  3. [11]

    Rt-1: Robotics transformer for real-world control at scale.Robotics: Science and Systems XIX, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.Robotics: Science and Systems XIX, 2023

    2023

  4. [12]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning. 35 PhyAgentO...

    2026

  5. [13]

    Learning to act anywhere with task-centric latent actions

    Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Learning to act anywhere with task-centric latent actions. InProceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2025. doi: 10.15607/RSS.2025.XXI.014....

    2025 doi

  6. [14]

    ROSClaw: An OpenClaw ROS 2 framework for agentic robot control and interaction.arXiv preprint arXiv:2603.26997, 2026

    Irvin Steve Cardenas, Marcus Anthony Arnett, Natalie Catherine Yeo, Lucky Sah, and Jong-Hoon Kim. ROSClaw: An OpenClaw ROS 2 framework for agentic robot control and interaction.arXiv preprint arXiv:2603.26997, 2026

    2026

  7. [15]

    Turner, Eric Undersander, and Tsung-Yen Yang

    Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M....

    2025

  8. [16]

    Closing the sim-to-real loop: Adapting simulation randomization with real world experience

    Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In2019 international conference on robotics and automation (ICRA), pages 8973...

    2019

  9. [17]

    NaVILA: Legged robot vision- language-action model for navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, and Xiaolong Wang. NaVILA: Legged robot vision- language-action model for navigation. InProceedings of Robotics: Science and Systems, Los Angeles, CA, USA, Jun...

    2025 doi

  10. [18]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  11. [19]

    Robonet: Large-scale multi-robot learning

    Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeck- peper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. InConference on Robot Learning, pages 885–897. PMLR, 2020

    2020

  12. [20]

    Causal confusion in imitation learning.Advances in neural information processing systems, 32, 2019

    Pim De Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning.Advances in neural information processing systems, 32, 2019

    2019

  13. [21]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  14. [22]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in neural information processing systems, 36:9156–9172, 2023

    2023

  15. [23]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In2017 IEEE international conference on robotics and automation (ICRA), pages 2786–2793. IEEE, 2017

    2017

  16. [24]

    CaP-X: A framework for benchmarking and improving coding agents for robot manipulation.arXiv preprint arXiv:2603.22435, 2026

    Max Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Gold- berg, and Linxi Jim Fan. CaP-X: A framework for benchmarking and improving coding agents for robot manipulatio...

    2026 arXiv

  17. [25]

    Integrated task and motion planning

    Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    2021

  18. [26]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, Steven Bohez, Konstantinos Bousmalis, Anthony Brohan, et al. Gemini robotics: Bringin...

    arXiv

  19. [27]

    FlowDreamer: A RGB-D world model with flow-based motion representations for robot manipulation.IEEE Robotics and Automation Letters, 11(3):2466–2473, 2026

    Jun Guo, Xiaojian Ma, Yikai Wang, Min Yang, Huaping Liu, and Qing Li. FlowDreamer: A RGB-D world model with flow-based motion representations for robot manipulation.IEEE Robotics and Automation Letters, 11(3):2466–2473, 2026. doi: 10.1109/LRA.2026.3653273. URLhttps://doi.org/1...

    2026

  20. [28]

    Doremi: Grounding language model by detecting and recovering from plan-execution misalignment

    Yanjiang Guo, Yen-Jen Wang, Lihan Zha, and Jianyu Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12124–12131. IEEE, 2024

    2024

  21. [29]

    Dream to con- trol: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to con- trol: Learning behaviors by latent imagination. InInternational Conference on Learning Representations,

  22. [30]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. InInternational Conference on Learning Representations,

  23. [31]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. InInternational conference on machine learning, pages 2555–2565. PMLR, 2019

    2019

  24. [32]

    Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    2025

  25. [33]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. InThe Twelfth International Conference on Learning Representations

  26. [35]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InInternational conference on machine learning, pages 9118–9147. PMLR, 2022

    2022

  27. [36]

    Voxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, pages 540–562. PMLR, 2023. 37 PhyAgentOS v0.1.6 2026-07-21

    2023

  28. [37]

    Inner monologue: Embodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. InConference on Robot Learning, pages 1769–1782. PMLR, 2023

    2023

  29. [38]

    ABot-Claw: A foundation for persistent, cooperative, and self-evolving robotic agents

    Dongjie Huo, Haoyun Liu, Guoqing Liu, Dekang Qi, Zhiming Sun, Maoguo Gao, Jianxin He, Yandan Yang, Xinyuan Chang, Feng Xiong, Xing Wei, Zhiheng Ma, and Mu Xu. ABot-Claw: A foundation for persistent, cooperative, and self-evolving robotic agents. arXiv preprint arXiv:2604.10096, 2026

    2026 arXiv

  30. [39]

    Deliberation for autonomous robots: A survey.Artificial Intelligence, 247:10–44, 2017

    Félix Ingrand and Malik Ghallab. Deliberation for autonomous robots: A survey.Artificial Intelligence, 247:10–44, 2017

    2017

  31. [40]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin Le...

    2025

  32. [41]

    A survey of behavior trees in robotics and ai.Robotics and Autonomous Systems, 154:104096, 2022

    Matteo Iovino, Edvards Scukins, Jonathan Styrud, Petter Ögren, and Christian Smith. A survey of behavior trees in robotics and ai.Robotics and Autonomous Systems, 154:104096, 2022

    2022

  33. [42]

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment, 2019

    2019

  34. [43]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. Inconference on Robot Learning, pages 991–1002. PMLR, 2022

    2022

  35. [44]

    DreamGen: Unlocking generalization in robot learning through video world models

    Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, Loic Magne, Ajay Mandlekar, et al. DreamGen: Unlocking generalization in robot learning through video world models. arXiv preprint arXiv:2505...

    2025 doi arXiv

  36. [45]

    When to trust your model: Model-based policy optimization.Advances in neural information processing systems, 32, 2019

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization.Advances in neural information processing systems, 32, 2019

    2019

  37. [47]

    Rldx-1 technical report.arXiv preprint arXiv:2605.03269, 2026

    Dongyoung Kim, Huiwon Jang, Myungkyu Koo, Suhyeok Jang, Taeyoung Kim, et al. Rldx-1 technical report.arXiv preprint arXiv:2605.03269, 2026. 38 PhyAgentOS v0.1.6 2026-07-21

    2026 arXiv

  38. [48]

    Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Ethan Lynch, Soroush Gopalakrishnan, Adrian Zeng, Jonathan Tompson, et al. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  39. [49]

    Fine-tuning vision-language-action models: Optimizing speed and success

    Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. InRobotics: Science and Systems, 2025. doi: 10.15607/RSS. 2025.XXI.017. URLhttps://arxiv.org/abs/2502.19645

    2025 arXiv

  40. [50]

    Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.Information Fusion, 58:52–68, 2020

    Timothee Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Diaz-Rodriguez. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.Information Fusion, 58:52–68, 2020. doi: 10.1016/j.inffus.2019.12.004

    2020 doi

  41. [51]

    PointVLA: Injecting the 3d world into vision-language-action models.IEEE Robotics and Automation Letters, 11 (3):2506–2513, 2026

    Chengmeng Li, Junjie Wen, Yaxin Peng, Yan Peng, and Yichen Zhu. PointVLA: Injecting the 3d world into vision-language-action models.IEEE Robotics and Automation Letters, 11 (3):2506–2513, 2026. doi: 10.1109/LRA.2026.3653303. URL https://doi.org/10.1109/ LRA.2026.3653303

    2026

  42. [52]

    Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese

    Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation ...

    2021

  43. [53]

    Vla-rft: Vision-language- action reinforcement fine-tuning with verified rewards in world simulators.arXiv preprint arXiv:2510.00406, 2025

    Hengtao Li, Pengxiang Ding, Runze Suo, Yihao Wang, Zirui Ge, Dongyuan Zang, Kexian Yu, Mingyang Sun, Hongyin Zhang, Donglin Wang, et al. Vla-rft: Vision-language- action reinforcement fine-tuning with verified rewards in world simulators.arXiv preprint arXiv:2510.00406, 2025

    2025

  44. [55]

    Embodied agent interface: Benchmarking LLMs for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Fei-Fei Li, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. InAdvances...

    2024 doi

  45. [56]

    Xiang Li, Ya-Li Li, Yuan Wang, Huaqiang Wang, and Shengjin Wang. TCoT: Trajectory chain-of-thoughts for robotic manipulation with failure recovery in vision-language-action model.Proceedings of the AAAI Conference on Artificial Intelligence, 40(8):6486–6494, 2026. doi: 10.1609...

    2026 doi

  46. [57]

    Evaluating real-world robot manipulation policies in simulation

    Xuanlin Li, Kyle Hsu, Jiayuan Gu, et al. Evaluating real-world robot manipulation policies in simulation. InConference on Robot Learning, volume 270 ofProceedings of Machine Learning Research, pages 3705–3728, 2024. URLhttps://arxiv.org/abs/2405.05941

    2024 arXiv

  47. [58]

    Optimus-3: Dual-router aligned mixture-of-experts agent with dual-granularity reasoning-aware policy optimization.arXiv preprint arXiv:2506.10357, 2025

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Weili Guan, Dongmei Jiang, Yaowei Wang, and Liqiang Nie. Optimus-3: Dual-router aligned mixture-of-experts agent with dual-granularity reasoning-aware policy optimization.arXiv preprint arXiv:2506.10357, 2025. 39 PhyAgentOS v0.1....

    2025

  48. [60]

    Genie envisioner: A unified world foundation platform for robotic manipulation, 2025

    Yue Liao, Pengfei Zhou, Siyuan Huang, Donglin Yang, Shengcong Chen, Yuxin Jiang, Yue Hu, Jingbin Cai, Si Liu, Jianlan Luo, Liliang Chen, Shuicheng Yan, Maoqing Yao, and Guanghui Ren. Genie envisioner: A unified world foundation platform for robotic manipulation, 2025

    2025

  49. [61]

    Steve-1: A generative model for text-to-behavior in minecraft.Advances in Neural Information Processing Systems, 2023

    Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. Steve-1: A generative model for text-to-behavior in minecraft.Advances in Neural Information Processing Systems, 2023

    2023

  50. [62]

    Droid: A large-scale in-the-wild robot manipulation dataset

    Joseph Jaewhan Lim. Droid: A large-scale in-the-wild robot manipulation dataset. In Robotics: Science and Systems. RSS Foundation, 2024

    2024

  51. [63]

    LIBERO: Benchmarking knowledge transfer for lifelong robot learning

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. InAdvances in Neural Information Processing Systems, volume 36, pages 44776–44791, 2023. URL https: //arxiv.org/abs/2306.03310

    2023 arXiv

  52. [64]

    RDT-1B: A diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, et al. RDT-1B: A diffusion foundation model for bimanual manipulation. InInternational Conference on Learning Representa- tions, 2025. URL https://openreview.net/forum?id=yAzN4tz7oI. Also available as arXiv:2410.07864

    2025 arXiv

  53. [65]

    Aligning cyber space with physical world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics, 2025

    2025

  54. [66]

    World action verifier: Self-improving world models via forward-inverse asymmetry.arXiv preprint arXiv:2604.01985, 2026

    Yuejiang Liu, Fan Feng, Lingjing Kong, Weifeng Lu, Jinzhou Tang, Kun Zhang, Kevin Murphy, Chelsea Finn, and Yilun Du. World action verifier: Self-improving world models via forward-inverse asymmetry.arXiv preprint arXiv:2604.01985, 2026. doi: 10.48550/arXiv. 2604.01985. URLhtt...

    2026 doi arXiv

  55. [67]

    REFLECT: Summarizing robot experiences for failure explanation and correction

    Zeyi Liu, Arpit Bahety, and Shuran Song. REFLECT: Summarizing robot experiences for failure explanation and correction. InConference on Robot Learning, 2023. URL https: //arxiv.org/abs/2306.15724

    2023 arXiv

  56. [68]

    ASPIRE: Agentic /Skills discovery for robotics.arXiv preprint arXiv:2607.00272, 2026

    Runyu Lu, Yubo Wu, Ethan Kou, Max Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi Fan, and Guanzhi Wang. ASPIRE: Agentic /Skills discovery for robotics.arXiv preprint arXiv:2607.00272, 2026

    2026 arXiv

  57. [69]

    RoVLA: Multi-consistency constraints for robust vision-language-action models.arXiv preprint arXiv:2605.19678, 2026

    Jingzhou Luo, Yifan Wen, Yongjie Bai, Xinshuai Song, Yang Liu, and Liang Lin. RoVLA: Multi-consistency constraints for robust vision-language-action models.arXiv preprint arXiv:2605.19678, 2026. doi: 10.48550/arXiv.2605.19678. URL https://arxiv.org/abs/ 2605.19678

    2026 doi arXiv

  58. [70]

    LIV: Language-image representations and rewards for robotic control

    Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-image representations and rewards for robotic control. InInternational Conference on Machine Learning, 2023. URL https: //arxiv.org/abs/2306.00958. 40 Phy...

    2023 arXiv

  59. [71]

    Eureka: Human-level reward design via coding large language models

    Yecheng Jason Ma, William Liang, Guanzhi Wang, et al. Eureka: Human-level reward design via coding large language models. InInternational Conference on Learning Represen- tations, 2024. URLhttps://arxiv.org/abs/2310.12931

    2024 arXiv

  60. [72]

    A survey on vision-language-action models for embodied ai.IEEE Transactions on Neural Networks and Learning Systems, 37(7):3031–3051, 2026

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai.IEEE Transactions on Neural Networks and Learning Systems, 37(7):3031–3051, 2026. doi: 10.1109/TNNLS.2025.3650584. URL https://arxiv.org/abs/2405.14093

    2026

  61. [73]

    Robot operating system 2: Design, architecture, and uses in the wild.Science Robotics, 7(66): eabm6074, 2022

    Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. Robot operating system 2: Design, architecture, and uses in the wild.Science Robotics, 7(66): eabm6074, 2022. doi: 10.1126/scirobotics.abm6074

    2022 doi

  62. [74]

    Isaac gym: High performance gpu-based physics simulation for robot learning, 2021

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021

    2021

  63. [75]

    Moerland, Joost Broekens, Aske Plaat, and Catholijn M

    Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based reinforcement learning: A survey.Foundations and Trends in Machine Learning, 16(1):1–118,

  64. [76]

    Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    Tongzhou Mu, Zhan Ling, Fanbo Xiang, Jinghong Chen, Xiaoran Su, Tong Yang, Naixuan Zhong, Jiawei Xiong, Siqi Liu, Yuzhe Zhu, et al. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.arXiv preprint arXiv:2107.14483, 2021

    2021 arXiv

  65. [77]

    Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots

    Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. Robocasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations (ICLR), 2026

    2026

  66. [78]

    Gr00t n1: An open foundation model for generalist humanoid robots, 2025

    NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llont...

    2025

  67. [80]

    Sim-to-real transfer of robotic control with dynamics randomization

    Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. InIEEE International Conference on Robotics and Automation, 2018. doi: 10.1109/ICRA.2018.8460528

    2018

  68. [81]

    FAST: Efficient action tokenization for vision- language-action models

    Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision- language-action models. InRobotics: Science and Systems, 2025. doi: 10.15607/RSS.2025. XXI.012

    2025 doi

  69. [82]

    Execution monitoring in robotics: A survey.Robotics and Autonomous Systems, 53(2):73–88, 2005

    Ola Pettersson. Execution monitoring in robotics: A survey.Robotics and Autonomous Systems, 53(2):73–88, 2005. doi: 10.1016/j.robot.2005.09.004. 41 PhyAgentOS v0.1.6 2026-07-21

    2005 doi

  70. [83]

    Dst-dojo: A don’t starve benchmark for long-horizon survival agents.arXiv preprint arXiv:placeholder, 2026

    placeholder. Dst-dojo: A don’t starve benchmark for long-horizon survival agents.arXiv preprint arXiv:placeholder, 2026

    2026

  71. [84]

    SpatialVLA: Exploring spatial represen- tations for visual-language-action models

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Jiayuan Gu, Zhigang Wang, Yan Ding, Bin Zhao, Dong Wang, and Xuelong Li. SpatialVLA: Exploring spatial represen- tations for visual-language-action models. InRobotics: Science and Systems XXI, 2025. doi: 10.15607/RSS.20...

    2025 doi

  72. [85]

    Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. ROS: An open-source robot operating system. InICRA Workshop on Open Source Software, 2009. URL http://www.robotics.stanford.edu/ ~ang/papers/icraoss09-ROS.pdf

    2009

  73. [86]

    A generalist agent.Transactions on Machine Learning Research, 2022

    Scott Reed, Konrad Zolna, Emilio Parisotto, et al. A generalist agent.Transactions on Machine Learning Research, 2022. URLhttps://arxiv.org/abs/2205.06175

    2022 arXiv

  74. [87]

    Ren, Anushri Dixit, Alexandra Bodrova, et al

    Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, et al. Robots that ask for help: Uncer- tainty alignment for large language model planners. InConference on Robot Learning, 2023. URLhttps://arxiv.org/abs/2307.01928

    2023 arXiv

  75. [88]

    Vision-language models are zero-shot reward models for reinforcement learning

    Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In International Conference on Learning Representations, 2024. URL https://arxiv.org/ abs/2310.12921

    2024 arXiv

  76. [89]

    Schoellig

    Ralf Römer, Yi Zhang, Yuming Li, and Angela P . Schoellig. CLARE: Continual learning for vision-language-action models via autonomous adapter routing and expansion.IEEE Robotics and Automation Letters, 11(7):8331–8338, 2026. doi: 10.1109/LRA.2026.3693992. URLhttps://doi.org/10...

    2026

  77. [90]

    Gordon, and J

    Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InInternational Conference on Artificial Intelligence and Statistics, pages 627–635, 2011. URL https://proceedings. mlr.press/v15...

    2011

  78. [91]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, et al. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, 2023. URLhttps://arxiv.org/abs/2302.04761

    2023 arXiv

  79. [92]

    Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588:604–609, 2020

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, et al. Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588:604–609, 2020. doi: 10.1038/s41586-020-03051-4

    2020 doi

  80. [93]

    Robovqa: Multimodal long-horizon reasoning for robotics

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, et al. Robovqa: Multimodal long-horizon reasoning for robotics. InIEEE International Conference on Robotics and Automation, 2024. doi: 10.1109/ICRA57147.2024.10610216. URLhttps://arxiv.org/abs/2311.00899

    2024

  81. [94]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models

    Lucy Xiaoyang Shi, Brian Ichter, Michael Robert Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hie...

    2025

  82. [95]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/ 2303.11366

    2023 arXiv

  83. [96]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. InConference on Robot Learning, volume 164 ofProceedings of Machine Learning Research, pages 894–906, 2022. URLhttps://arxiv.org/abs/2109.12098

    2022 arXiv

  84. [97]

    SmolVLA: A vision-language- action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844, 2025

    Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andrés Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Rémi Cadène. SmolVLA: A vision-language- action model for affor...

    2025 doi arXiv

  85. [98]

    Reid G. Simmons. Structured control for autonomous robots.IEEE Transactions on Robotics and Automation, 10(1):34–43, 1994. doi: 10.1109/70.285583

    1994 doi

  86. [99]

    Progprompt: Generating situated robot task plans using large language models

    Ishika Singh, Valts Blukis, Arsalan Mousavian, et al. Progprompt: Generating situated robot task plans using large language models. InIEEE International Conference on Robotics and Automation, 2023. doi: 10.1109/ICRA48891.2023.10161317. URL https://arxiv. org/abs/2209.11302

    2023

  87. [100]

    ConditionNET: Learning preconditions and effects for execution monitoring.IEEE Robotics and Automation Letters, 10(2):1337–1344, 2025

    Daniel Sliwowski and Dongheui Lee. ConditionNET: Learning preconditions and effects for execution monitoring.IEEE Robotics and Automation Letters, 10(2):1337–1344, 2025. doi: 10.1109/LRA.2024.3520916. URLhttps://doi.org/10.1109/LRA.2024.3520916

    2025

  88. [101]

    Sadler, Wei-Lun Chao, and Yu Su

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner: Few-shot grounded planning for embodied agents with large language models. InIEEE/CVF International Conference on Computer Vision, 2023. URL https://arxiv.org/abs/2212.04088

    2023 arXiv

  89. [102]

    Sontakke, Jesse Zhang, Sebastien M

    Sumedh A. Sontakke, Jesse Zhang, Sebastien M. R. Arnold, et al. Roboclip: One demon- stration is enough to learn robot policies. InAdvances in Neural Information Processing Systems, volume 36, 2023. URLhttps://arxiv.org/abs/2310.07899

    2023 arXiv

  90. [103]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yangyi Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Naoki Yokoyama, et al. Habitat 2.0: Training home assistants to rearrange their habitat. InAdvances in Neural Information Processing Systems, 2021

    2021

  91. [104]

    RoboOS: A hierarchical embodied framework for cross-embodiment and multi-agent collaboration.arXiv preprint arXiv:2505.03673, 2025

    Huajie Tan, Xiaoshuai Hao, Cheng Chi, Minglan Lin, Yaoxu Lyu, Mingyu Cao, Dong Liang, Zhuo Chen, Mengsi Lyu, Cheng Peng, Chenrui He, Yulong Ao, Yonghua Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. RoboOS: A hierarchical embodied framework for cross-embodiment and mu...

    2025 arXiv

  92. [105]

    Stardojo: Benchmarking open-ended behaviors of agentic multimodal llms in production–living simulations with stardew valley.arXiv preprint arXiv:2507.07445, 2025

    Weihao Tan et al. Stardojo: Benchmarking open-ended behaviors of agentic multimodal llms in production–living simulations with stardew valley.arXiv preprint arXiv:2507.07445, 2025

    2025 arXiv

  93. [106]

    Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024. 43 PhyAgentOS v0.1.6 2026-07-21

    2024 arXiv

  94. [107]

    Mitchell

    Sebastian Thrun and Tom M. Mitchell. Lifelong robot learning.Robotics and Autonomous Systems, 15(1–2):25–46, 1995. doi: 10.1016/0921-8890(95)00004-Y

    1995 doi

  95. [108]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. InIEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. doi: 10.1109/IROS...

    2017

  96. [109]

    Voyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023

    2023

  97. [110]

    Gensim: Generating robotic simulation tasks via large language models

    Lirui Wang, Yiyang Ling, Zhecheng Yuan, et al. Gensim: Generating robotic simulation tasks via large language models. InInternational Conference on Learning Representations,

  98. [112]

    VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model.Proceedings of the AAAI Conference on Artificial Intelligence, 40(22):18638–18646, 2026

    Yihao Wang, Pengxiang Ding, Lingxiao Li, Can Cui, Zirui Ge, Xinyang Tong, Wenxuan Song, Han Zhao, Wei Zhao, Pengxu Hou, Siteng Huang, Yifan Tang, Wenhui Wang, Ru Zhang, Jianyi Liu, and Donglin Wang. VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model...

    2026 doi

  99. [113]

    Robogen: Towards unleashing infinite data for automated robot learning via generative simulation, 2023

    Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation, 2023

    2023

  100. [114]

    De- scribe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents

    Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. De- scribe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. InAdvances in Neural Information Processing Systems, 2023. URLhttps://arx...

    2023 arXiv

  101. [115]

    KARMA: Augmenting embodied AI agents with long-and-short term memory systems

    Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Yinhe Han, and Yiming Gan. KARMA: Augmenting embodied AI agents with long-and-short term memory systems. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 1–8,...

    2025

  102. [116]

    DexVLA: Vision-language model with plug-in diffusion expert for general robot control.arXiv preprint arXiv:2502.05855, 2025

    Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, and Feifei Feng. DexVLA: Vision-language model with plug-in diffusion expert for general robot control.arXiv preprint arXiv:2502.05855, 2025. doi: 10.48550/arXiv.2502.05855. URL https://arxiv. org/abs/2502.05855

    2025 doi arXiv

  103. [117]

    DiffusionVLA: Scaling robot foundation models via unified diffusion and autoregression

    Junjie Wen, Yichen Zhu, Minjie Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Xiaoyu Liu, Chaomin Shen, Yaxin Peng, and Feifei Feng. DiffusionVLA: Scaling robot foundation models via unified diffusion and autoregression. InProceedings of the 42nd International Conference on Machi...

  104. [118]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation

    Wen Wen, Hu Li, et al. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. InConference on Robot Learning, 2024

    2024

  105. [119]

    WorldDreamer: Multi-task policy for RoboCasa365

    WorldAgents. WorldDreamer: Multi-task policy for RoboCasa365. https: //huggingface.co/WorldAgents-c/world_dreamer-robocasa365-multi_task ,

  106. [120]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. InConference on Robot Learning,

  107. [121]

    44 PhyAgentOS v0.1.6 2026-07-21

    URLhttps://proceedings.mlr.press/v267/wen25g.html. 44 PhyAgentOS v0.1.6 2026-07-21

    2026

  108. [122]

    Unisim: A neural closed-loop sensor simulator

    Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. InCVPR, 2023

    2023

  109. [123]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, et al. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2210.03629

    2023 arXiv

  110. [124]

    Stable language guidance for vision-language-action models

    Zhihao Zhan, Yuhao Chen, Jiaying Zhou, Qinhan Lyu, Hao Liu, Keze Wang, Liang Lin, and Guangrun Wang. Stable language guidance for vision-language-action models. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

  111. [125]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. InAAAI Conference on Artificial Intelligence,

  112. [126]

    URLhttps://arxiv.org/abs/2206.14176

    arXiv

  113. [128]

    VLMPC: Vision-language model predictive control for robotic manipulation

    Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. VLMPC: Vision-language model predictive control for robotic manipulation. InRobotics: Science and Systems XX, 2024. doi: 10.15607/RSS.2024.XX.106

    2024 doi

  114. [129]

    X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025

    Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, et al. X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025. 45 PhyAgentO...

    2025 arXiv

  115. [130]

    Code-as-monitor: Constraint-aware visual pro- gramming for reactive and proactive robotic failure detection

    Enshen Zhou, Qi Su, Cheng Chi, et al. Code-as-monitor: Constraint-aware visual pro- gramming for reactive and proactive robotic failure detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. doi: 10.1109/CVPR52734.2025.00649. URL https://arxiv.org/a...

    2025

  116. [131]

    URL https://doi.org/10.18653/v1/ 2026.acl-long.190

    doi: 10.18653/v1/2026.acl-long.190. URL https://doi.org/10.18653/v1/ 2026.acl-long.190

    2026 doi

  117. [132]

    Robo- dreamer: Learning compositional world models for robot imagination

    Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robo- dreamer: Learning compositional world models for robot imagination. InInternational Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research,

  118. [133]

    URLhttps://arxiv.org/abs/2308.10144

    doi: 10.1609/aaai.v38i17.29936. URLhttps://arxiv.org/abs/2308.10144

    arXiv

  119. [135]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRobotics: Science and Systems, 2023. doi: 10.15607/RSS.2023.XIX.016. URLhttps://arxiv.org/abs/2304.13705

    2023 arXiv

  120. [136]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–

  121. [139]

    DINO-WM: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research,

  122. [140]

    URLhttps://proceedings.mlr.press/v267/zhou25t.html

  123. [142]

    URLhttps://proceedings.mlr.press/v235/zhou24f.html

  124. [143]

    ChatVLA: Unified multimodal understanding and robot control with vision-language-action model

    Zhongyi Zhou, Yichen Zhu, Minjie Zhu, Junjie Wen, Ning Liu, Zhiyuan Xu, Weibin Meng, Yaxin Peng, Chaomin Shen, Feifei Feng, and Yi Xu. ChatVLA: Unified multimodal understanding and robot control with vision-language-action model. InProceedings of the 2025 Conference on Empiric...

    2025 doi

  125. [144]

    IRASim: A fine-grained world model for robot manipulation

    Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. IRASim: A fine-grained world model for robot manipulation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9834–9844, 2025. doi: 10.1109/ ICCV51701.2025.00917. UR...

    2025

  126. [145]

    robosuite: A modular simulation framework and benchmark for robot learning, 2025

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Kevin Lin, Abhiram Maddukuri, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning, 2025

    2025

  127. [2023]

    URLhttps://arxiv.org/abs/2006.16712

    doi: 10.1561/2200000086. URLhttps://arxiv.org/abs/2006.16712

    2006 arXiv

  128. [2024]

    URLhttps://arxiv.org/abs/2310.01361

    arXiv

  129. [2025]

    URLhttps://arxiv.org/abs/2503.20020

    doi: 10.48550/arXiv.2503.20020. URLhttps://arxiv.org/abs/2503.20020

    doi arXiv

  130. [2026]

    RoboCasa365 leaderboard checkpoint and evaluation code

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.