Pith. sign in

REVIEW 4 major objections 4 minor 59 references

This paper claims that a home-robot policy trained entirely in simulation, without oracle perception, can transfer zero-shot to a physical dual-arm robot and complete long-horizon household tasks with interactive disambiguation at a 78.3% e

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:33 UTC pith:LVXNHL7Q

load-bearing objection A real system paper with a useful benchmark, a clean training story, and a candid failure analysis — but the sim-to-real transfer claim rests on an unproven equivalence between oracle-like sim handlers and messy real-world perception. the 4 major comments →

arxiv 2607.13653 v2 pith:LVXNHL7Q submitted 2026-07-15 cs.CV cs.RO

Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

Boyu Mi , Mengchen Ma , Yifei Yao , Xing Gao , Junting Chen , Yangzi Li , Zihou Zhu , Guohao Li
show 5 more authors
Zhenfei Yin Tai Wang Yao Mu Jiangmiao Pang Hanqing Wang
This is my paper
classification cs.CV cs.RO
keywords embodied agentmobile manipulationsim-to-real transfervision-language modelreinforcement learningactive explorationinteractive disambiguationREAL-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a home-robot policy can be trained entirely in simulation—without privileged object lists or oracle perception—and then transferred to a physical robot with no retraining. To do this it builds an environment whose tool interface is identical in simulation and reality, adds a simulated user who gives ambiguous instructions and clarifies only after the robot reports what it sees, and trains a small vision-language model with supervised fine-tuning followed by reinforcement learning. On its 241-task benchmark the trained agent beats strong commercial vision-language models on interactive disambiguation tasks (56.9% success). On a physical dual-arm mobile robot it claims 78.3% end-to-end success over 60 episodes, which it presents as evidence that the sim-to-real-consistent design bridges the reality gap for long-horizon mobile manipulation.

Core claim

The central claim is that replacing oracle perceptual APIs with a physically realizable toolchain—scanning, category-based detection, gaze, and manipulation primitives—plus a simulated user that withholds information until the robot observes the relevant object, allows a vision-language policy to learn active exploration and interactive disambiguation purely from simulation. Combined with closed-loop sequence-level reinforcement learning, the policy learns to ask the user when objects are visually ambiguous. When the same tool interface is run against a physical robot's navigation, open-vocabulary perception, and VLA manipulation backend, the policy transfers zero-shot and completes househol

What carries the argument

The deployable toolchain: a fixed set of discrete tool calls (navigate, walk around, show objects by category, gaze, pick/place/open/close, ask) exposed to the policy through the same interface in simulation and on the robot, with no global object list or ground-truth pose access. The simulated user with progressive disclosure and the compact memory (a self-updated summary of history plus task-phase analysis) serve as the supporting machinery; group sequence policy optimization with world-graph-grounded rewards is what shapes the policy to explore and ask. The work the machinery does: it converts the task from 'locate and manipulate' into a closed loop of visual exploration, dialogue, and ex

Load-bearing premise

The load-bearing premise is that the simulation's perfect-perception tool handlers (a live object registry and ground-truth segmentation masks) behave equivalently enough to the physical robot's noisy open-vocabulary perception that a policy shaped by perfect perception still works when perception is imperfect.

What would settle it

Run the trained policy on the interactive benchmark while replacing the simulated handler's ground-truth object registry and masks with outputs from a real open-vocabulary segmentation model (with realistic false positives and negatives) applied to rendered views; if success on the interactive split drops well below the reported 56.9%, the robust-transfer claim is falsified. Alternatively, on the physical robot, run episodes where target and distractor objects differ only by subtle texture or color and measure the object-confusion rate; a large drop versus simulation would indicate the transfe

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, training high-level household robot policies entirely in simulation without privileged perception is feasible for long-horizon tasks.
  • Interactive disambiguation—asking the user when ambiguous—can be learned from environmental reward rather than hand-crafted prompts, and transfers to real human users.
  • The same tool-call interface means a policy can be evaluated in simulation and deployed on a robot with different low-level controllers without retraining the high-level reasoning.
  • A compact memory summary (overwriting raw history) suffices to avoid context explosion and temporal hallucination in long horizons.
  • The main remaining bottleneck is fine-grained visual grounding (object confusion), not planning or interaction, for this approach.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 78.3% real-world number likely depends on the low-level VLA primitives being highly reliable (85.3% primitive executability); if those primitives degrade, the high-level transfer claim would be weaker.
  • The perfect-perception simulation handlers may underestimate how much real-world noise the policy must tolerate; real open-vocabulary segmentation errors could make the learned 'ask when ambiguous' behavior even more critical than the benchmark suggests.
  • A testable extension: swapping the simulated walk-around registry for a noisy open-vocabulary detector at evaluation time would directly quantify the sim-to-real perception gap.
  • The policy's tolerance of short real-user replies (27 vs 29 successes with a simulated user) suggests the ask behavior is robust, but underspecified feedback remains a failure driver.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents REAL, a framework for training and evaluating vision-language embodied agents for open-world mobile manipulation. The authors build a simulation environment in GRUtopia/Isaac Sim that removes oracle perception APIs and adds a simulated user that issues ambiguous instructions and answers queries. They generate an SFT dataset using rule-based planners with reasoning traces annotated by gemini-3-pro, then apply GSPO online RL. They introduce REAL-Bench with 241 tasks across four families (FDP, FODP, FDO, SUL). The trained Qwen3-VL-8B agent achieves 56.9% success on the interactive SUL split, slightly above gemini-3-pro-preview (53.8%), and 78.3% end-to-end success in 60 physical trials on an ARX LIFT2 robot. The paper claims zero-shot transfer of the high-level policy.

Significance. The paper makes a valuable contribution by building a closed-loop training environment for exploratory and communicative manipulation, with a benchmark and a real-robot deployment. The code and benchmark are intended to be released, which supports reproducibility. If the claims hold, the work demonstrates that a VLM-based high-level policy can learn to explore, ask for disambiguation, and transfer to a physical robot via a standardized MCP tool interface. However, the central transfer claim depends on the unverified assumption that the simulation's perfect-perception tool handlers are behaviorally equivalent to the real world's noisy perception stack. Additionally, the headline simulation result rests on a statistically fragile two-episode margin, and the training/evaluation pipeline shares the same model family for teacher annotation and simulated-user interaction. These caveats substantially temper the strength of the conclusions.

major comments (4)
  1. [Appendix C.2, Section 5.3, Table 2, Appendix G.2] The 'without oracle perception' claim is undermined by the simulation tool implementations. The simulation walk_around reads a live per-receptacle registry and returns a complete inventory; show_object_by_category uses ground-truth per-instance masks. The real-world counterparts use Grounded-SAM-2 with noisy masks and cross-view deduplication. No quantitative comparison is provided. Since Appendix D.6 shows 94.3% of RL failures are wrong-object picks even with perfect inventories, and the real-world failure breakdown (G.2) reports only 2/60 high-level visual errors, the 78.3% transfer rate could reflect an easier real-world distribution rather than a perception-robust policy. Please measure the sim-to-real perception gap (e.g., detection precision/recall) or acknowledge that the real trials were less perceptually demanding.
  2. [Table 1, SUL column] The headline claim of surpassing commercial VLMs rests on 56.9% vs 53.8% for gemini-3-pro-preview — a difference of about 2 out of 65 episodes. No confidence intervals or significance tests are reported. For a claim of this importance, the paper should either provide additional seeds, more episodes, or a statistical test, or qualify the claim as non-significant.
  3. [Appendix B.5, Appendix C.3, Table 1] The SFT reasoning traces are annotated by gemini-3-pro (the same model family as the strongest zero-shot baseline, labeled 'teacher' in Table 1), and the simulated user used in training and SUL evaluation is gemini-3-flash. This creates a train/eval overlap that could inflate the student's SUL performance. The human-user evaluation in Appendix E.3 mitigates this (27/50 vs 29/50), but the benchmark claim should be interpreted with this caveat, and the paper should discuss the overlap explicitly.
  4. [Appendix C.2, walk_around; Section 3.1] In simulation, the agent receives a perfect inventory string from a live registry, so exploration reduces to selecting which receptacle to visit rather than resolving within-receptacle detection uncertainty. This weakens REAL-Bench's stated focus on 'active exploration' and raises doubts about whether training in this environment imparts perception-robust exploration skills. A noisy-perception variant of the simulation toolchain or an explicit analysis of the policy's sensitivity to inventory errors would strengthen the claims.
minor comments (4)
  1. [Section 4.1 vs Appendix A.1] Section 4.1 states '7 high-fidelity scenes', while Appendix A.1 lists 6 training environments (plus held-out S7). Please clarify the total scene count.
  2. [Abstract and Section 5.3] The abstract's 'zero-shot transferability' should be qualified: the low-level VLA policy was fine-tuned on 980 real teleoperation trajectories (Appendix G.1); only the high-level policy transfers zero-shot.
  3. [Equation (2)] Eq. (2) uses Serialize(A) before A is defined in the text. Please define the action space earlier for clarity.
  4. [Table G.1] In the navigation/gaze repeatability section, the row labeled 'All' would benefit from a brief note on the distribution of target types across the 30 trials.

Circularity Check

1 steps flagged

The 'no oracle perception' design is defined around tool names, while the simulation's walk_around tool returns a complete simulator-registry inventory to the policy.

specific steps
  1. self definitional [Abstract; Section 3.1; Appendix C.2 (walk_around)]
    "REAL establishes sim-to-real-consistent environment APIs without oracle perception. ... The agent is restricted from accessing global object lists. ... The simulation maintains a per-receptacle object registry that is kept synchronized throughout each episode ... When walk_around is invoked, the handler reads this live registry ... returning a natural-language inventory string."

    The central 'oracle-free' claim is defined by what appears in the initial prompt rather than by information content. The walk_around tool delivers a complete, correct object inventory read from simulator state—exactly the global object list the paper says it removes—so benchmark success on active exploration is largely supplied by the tool itself. The real-world replacement (Grounded-SAM-2 over noisy RGB) has no quantified behavioral equivalence, so the zero-shot-transfer conclusion is not implied by the oracle-free design; it is an artifact of defining 'no oracle' around tool names while the tool's return is oracle data by construction.

full rationale

The headline numbers (56.9% SUL in simulation, 78.3% real-world) are empirical measurements rather than quantities derived from fitted parameters, and the use of gemini-3-pro both as SFT annotation teacher and zero-shot baseline is standard distillation rather than circularity. The Gemini-driven simulated user is used symmetrically for the agent and the baselines, and the paper provides a real-human SUL check (27/50 vs. 29/50), so that comparison is not self-referential. However, the abstract's load-bearing claim—'sim-to-real-consistent environment APIs without oracle perception'—is undercut by the paper's own implementation: walk_around reads a live per-receptacle simulator registry and returns a perfect inventory string, and show_object_by_category uses pixel-accurate per-instance segmentation masks from the renderer. The policy therefore receives privileged state information through the tool-return channel that the design declares forbidden, making the 'exploration without oracle' claim true only by definitional fiat. Since the real-world tools are noisy and their equivalence to the registry/ground-truth-mask handlers is never established, the abstract's conclusion that the design 'successfully bridges the reality gap' rests on this definitional sleight rather than on demonstrated equivalence. This is a partial circularity of the central design claim, not of the raw success-rate numbers themselves.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claims rest on four domain assumptions rather than new physical entities: simulator fidelity, equivalence of the deterministic sim tool handlers to real noisy perception, the simulated-user envelope as a proxy for humans, and world-graph match as the success criterion. Free parameters are the hand-set reward coefficients that induce the asking behavior and the RL hyperparameters. No new particles, forces, dimensions, or conserved quantities are introduced; the 'simulated user' and 'world graph' are software constructs with no independent-evidence requirement. The perception-equivalence assumption is the one with the weakest support, and the paper's own failure data (94.3% wrong-object picks in RL rollouts) show exactly what happens when that assumption is violated.

free parameters (3)
  • Reward shaping coefficients and query budget = r_task=+1.5, r_prog=2.0·Δρ, r_phys=+0.3 pick / +0.5 in-transit / +1.0 recovery, r_ask=+1.0, r_safe=−0.5/−0.3, λ_time=0.0
    Hand-chosen constants that shape whether the policy explores, asks, and recovers; the reported ask rate (15.8% of RL trajectories) and exploration behavior depend directly on them.
  • GSPO hyperparameters and LoRA configuration = β=0.04, group size 8, clip radius ε, LoRA rank 32/α=64, LR 5e-6, 1000 steps (Table D.1)
    Standard RL training choices; results are reported for these specific settings and no sensitivity analysis is given beyond the ±3% image-resolution check.
  • Episode horizon and image resolution = H_max=30 decision steps; max_pixels=100,352
    Task difficulty and the visual-grounding ceiling depend on these; a resolution-ablation is reported (±3% success across task families, Appendix D.6).
axioms (5)
  • domain assumption GRUtopia/Isaac Sim physics and rendering are a faithful proxy for real household physics and appearance.
    The entire sim-to-real argument (Sections 3 and 5.3) presumes that training rollouts in GRUtopia transfer to a physical kitchen; this is tested only indirectly by the 60-episode deployment.
  • domain assumption Deterministic simulation tool handlers are behaviorally equivalent to the real perception/navigation stack.
    Appendix C.2: walk_around reads a perfect simulator-maintained registry and show_object_by_category uses ground-truth instance segmentation, while the real implementations use Grounded-SAM-2 with noisy masks. Without this equivalence, policies trained in simulation would not transfer.
  • domain assumption The simulated user's progressive-disclosure rules approximate real human clarification behavior.
    Section 3.1 / Appendix C.3. The user may name the target only after the robot reports seeing it; real humans need not follow this envelope. Partially validated by the 50-episode real-user test (Appendix E.3): 27 vs 29 successes.
  • domain assumption World-graph state match is a valid success criterion for task completion.
    Sections 4.1 and 5.1: success is a state comparison (WG_goal ⊖ WG_init empty), not a human judgment of whether the task intent was satisfied; equivalence of state match to user satisfaction is assumed.
  • standard math GSPO (length-normalized sequence ratio with clipping) is a sound sequence-level RL objective.
    Equations 12–14 are taken from the cited GSPO paper [58]; the authors apply it as published with no new mathematical claim.

pith-pipeline@v1.3.0-alltime-deepseek · 33838 in / 21524 out tokens · 204886 ms · 2026-08-02T04:33:55.801398+00:00 · methodology

0 comments
read the original abstract

Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at https://github.com/InternRobotics/REAL.

Figures

Figures reproduced from arXiv: 2607.13653 by Boyu Mi, Guohao Li, Hanqing Wang, Jiangmiao Pang, Junting Chen, Mengchen Ma, Tai Wang, Xing Gao, Yangzi Li, Yao Mu, Yifei Yao, Zhenfei Yin, Zihou Zhu.

Figure 1
Figure 1. Figure 1: Comparison of embodied agent frameworks. Left: Reliance on oracle perceptual APIs and open-loop execution in prior environments hinders ambiguity resolution and real-world deployment. Right: REAL removes oracle perceptual APIs while retaining physically obtainable scene priors, enabling RGB-based exploration and natural language interaction for ambiguous instructions. Specifically, the first limitation lie… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the environment design. The Embodied Agent receives physically obtainable re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Real-world deployment on the ARX LIFT2 mobile manipulator. REAL transfers the high-level visual-interactive policy zero-shot to a physical backend with the same MCP tool interface. The execution trajectory demonstrates causal planning (e.g., preparing the microwave before fetching food), human-robot disambiguation under multiple candidate objects, and language-to-physical grounding for closed-loop task com… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 18 linked inside Pith

  1. [1]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Ku...

    2023

  2. [2]

    Introducing the model context protocol, Nov 2024,https://www.anthropic

    Anthropic. Introducing the model context protocol, Nov 2024,https://www.anthropic. com/news/model-context-protocol, accessed: June 26, 2026

    2024

  3. [3]

    Claude haiku 4.5 system card

    Anthropic. Claude haiku 4.5 system card. System card, Anthropic, Oct 2025, https: //www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf, accessed: July 1, 2026

    2025

  4. [4]

    Qwen3-VL technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    Pith/arXiv arXiv 2025

  5. [5]

    RT-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. InRSS, 2023

    2023

  6. [6]

    Turner, Eric Undersander, and Tsung-Yen Yang

    Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. PARTNR: A benchmark for planning ...

    2025

  7. [7]

    Era: Transforming vlms into em- bodied agents via embodied prior learning and online reinforcement learning.arXiv preprint arXiv:2510.12693, 2025

    Hanyang Chen, Mark Zhao, Rui Yang, Qinwei Ma, Ke Yang, Jiarui Yao, Kangrui Wang, Hao Bai, Zhenhailong Wang, Rui Pan, Mengchao Zhang, Jose Barreiros, Aykut Onol, ChengXiang Zhai, Heng Ji, Manling Li, Huan Zhang, and Tong Zhang. Era: Transforming vlms into em- bodied agents via embodied prior learning and online reinforcement learning.arXiv preprint arXiv:2...

    arXiv 2025

  8. [8]

    OWMM-Agent: Open world mobile manipulation with multi-modal agentic data synthesis

    Junting Chen, Haotian Liang, Lingxiao Du, Weiyun Wang, Mengkang Hu, Yao Mu, Wenhai Wang, Jifeng Dai, Ping Luo, Wenqi Shao, and Lin Shao. OWMM-Agent: Open world mobile manipulation with multi-modal agentic data synthesis. InNeurIPS, 2025

    2025

  9. [9]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. IJRR, 2024. doi: 10.1177/02783649241273668. 53

    doi:10.1177/02783649241273668 2024

  10. [10]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, longcontext, andnextgenerationagenticcapabilities.arXivpreprintarXiv:2507.06261, 2025

    Gheorghe Comanici, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, longcontext, andnextgenerationagenticcapabilities.arXivpreprintarXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    ManiTaskGen: A comprehensive task generator for benchmarking and improving vision-language agents on embodied decision-making.arXiv preprint arXiv:2505.20726, 2025

    Liu Dai, Haina Wang, Weikang Wan, and Hao Su. ManiTaskGen: A comprehensive task generator for benchmarking and improving vision-language agents on embodied decision-making.arXiv preprint arXiv:2505.20726, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    ProcTHOR: Large-scale embodied AI using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-scale embodied AI using procedural generation. InNeurIPS, 2022

    2022

  13. [13]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. InICML, 2023

    2023

  14. [14]

    Integrated task and motion planning.An- nual Review of Control, Robotics, and Autonomous Systems,4(1), 2021

    Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning.An- nual Review of Control, Robotics, and Autonomous Systems,4(1), 2021. doi: 10.1146/ annurev-control-091420-084139

    2021

  15. [15]

    Gemini.https://deepmind.google/models/gemini/, 2025, accessed: June 26, 2026

    Google DeepMind. Gemini.https://deepmind.google/models/gemini/, 2025, accessed: June 26, 2026

    2025

  16. [16]

    Gemini 3 pro model card

    Google DeepMind. Gemini 3 pro model card. Model card, Google DeepMind, 2026, https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf, model release: November 2025; last updated: May 2026. Used to document the Gemini 3 Pro family corresponding to the evaluated gemini-3-pro-previewAPI identifier. Accessed: July 1, 2026

    2026

  17. [17]

    InICRA, 2024

    Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, AdityaAgarwal, CorbanRivera, WilliamPaul, KirstyEllis, RamaChellappa, etal.ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning. InICRA, 2024

    2024

  18. [18]

    MesaTask: Towards task-driven tabletop scene generation via 3D spatial reasoning

    Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, and Jiangmiao Pang. MesaTask: Towards task-driven tabletop scene generation via 3D spatial reasoning. InNeurIPS, 2025

    2025

  19. [19]

    Language models as zero- shot planners: Extracting actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. InICML, 2022

    2022

  20. [20]

    InCoRL, 2023

    WenlongHuang,FeiXia,TedXiao,HarrisChan,JackyLiang,PeteFlorence,AndyZeng,Jonathan Tompson, IgorMordatch, YevgenChebotar, etal.Innermonologue: Embodiedreasoningthrough planning with language models. InCoRL, 2023

    2023

  21. [21]

    VL-LN bench: Towards long-horizon goal-oriented navigation with active dialogs.arXiv preprint arXiv:2512.22342, 2025

    Wensi Huang, Shaohao Zhu, Meng Wei, Jinming Xu, Xihui Liu, Hanqing Wang, Tai Wang, Feng Zhao, and Jiangmiao Pang. VL-LN bench: Towards long-horizon goal-oriented navigation with active dialogs.arXiv preprint arXiv:2512.22342, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Hierarchical task and motion planning in the now

    Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. InICRA, 2011

    2011

  23. [23]

    OpenVLA: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, RafaelRafailov, EthanFoster, GraceLam, PannagSanketi, QuanVuong, ThomasKollar, Benjamin 54 Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. InCoRL, 2024

    2024

  24. [24]

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi.AI2-THOR:Aninteractive3denvironmentforvisualai.arXivpreprintarXiv:1712.05474, 2017

    Pith/arXiv arXiv 2017

  25. [25]

    EmboMatrix: A scalable training-ground for embodied decision-making.arXiv preprint arXiv:2510.12072, 2025

    Zixing Lei, Sheng Yin, Yichen Xiong, Yuanzhuo Ding, Wenhao Huang, Yuxi Wei, Qingyao Xu, Yiming Li, Weixin Li, Yunhong Wang, and Siheng Chen. EmboMatrix: A scalable training-ground for embodied decision-making.arXiv preprint arXiv:2510.12072, 2025

    arXiv 2025

  26. [26]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InCoRL, 2023

    2023

  27. [27]

    EMMOE: A comprehensive benchmark for embodied mobile manipulation in open environments.arXiv preprint arXiv:2503.08604, 2025

    Dongping Li, Tielong Cai, Tianci Tang, Wenhao Chai, Katherine Rose Driggs-Campbell, and Gaoang Wang. EMMOE: A comprehensive benchmark for embodied mobile manipulation in open environments.arXiv preprint arXiv:2503.08604, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Embodied agent interface: Benchmarking LLMs for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. InNeurIPS, 2024

    2024

  29. [29]

    Code as policies: Language model programs for embodied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. InICRA, 2023

    2023

  30. [30]

    Visualagentbench: Towards large multimodal models as visual foundation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: To...

    2025

  31. [31]

    PIVOT: Iterative visual prompting elicits actionable knowledge for VLMs

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. PIVOT: Iterative visual prompting elicits ac...

    2024

  32. [32]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2026

    OpenAI. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2026

    Pith/arXiv arXiv 2026

  34. [34]

    TEACh: Task-driven embodied agents that chat

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gökhan Tür, and Dilek Hakkani-Tür. TEACh: Task-driven embodied agents that chat. InAAAI, 2022

    2022

  35. [35]

    ADAPT: Actively discovering and adapting to preferences for any task

    Maithili Patel, Xavier Puig, Ruta Desai, Roozbeh Mottaghi, Sonia Chernova, Joanne Truong, and Akshara Rai. ADAPT: Actively discovering and adapting to preferences for any task. InCOLM, 2025

    2025

  36. [36]

    Simulating user agents for embodied conversational-ai.arXiv preprint arXiv:2410.23535, 2024

    Daniel Philipov, Vardhan Dongre, Gökhan Tür, and Dilek Hakkani-Tür. Simulating user agents for embodied conversational-ai.arXiv preprint arXiv:2410.23535, 2024. 55

    Pith/arXiv arXiv 2024

  37. [37]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.𝜋0.5: a vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    VirtualHome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Frieda Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating household activities via programs. InCVPR, 2018

    2018

  39. [39]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh

    Allen Z. Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confident: Efficient exploration for embodied question answering. InRSS, 2024

    2024

  41. [41]

    Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. InCoRL, 2023

    2023

  42. [42]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    ALFRED: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. InCVPR, 2020

    2020

  44. [44]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In ICLR, 2021

    2021

  45. [45]

    Replan: Robotic replanning with perception and language models.arXiv preprint arXiv:2401.04157, 2024

    Marta Skreta, Zihan Zhou, Jia Lin Yuan, Kourosh Darvish, Alán Aspuru-Guzik, and Ani- mesh Garg. Replan: Robotic replanning with perception and language models.arXiv preprint arXiv:2401.04157, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Habitat 2.0: Training home assistants to rearrange their habitat

    Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Von- drus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assis...

    2021

  47. [47]

    From multimodal llms to generalist embodied agents: Methods and lessons

    Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal llms to generalist embodied agents: Methods and lessons. InCVPR, 2025

    2025

  48. [48]

    InICLR, 2024

    Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Ryan Metcalf, Walter Talbott, NatalieMackraz, R.DevonHjelm, andAlexanderToshev.Largelanguagemodelsasgeneralizable policies for embodied tasks. InICLR, 2024

    2024

  49. [49]

    RoboOS: A hierarchical embodied framework for 56 cross-embodiment and multi-agent collaboration.arXiv preprint arXiv:2505.03673, 2025

    Huajie Tan, Xiaoshuai Hao, Cheng Chi, Minglan Lin, Yaoxu Lyu, Mingyu Cao, Dong Liang, Zhuo Chen, Mengsi Lyu, Cheng Peng, Chenrui He, Yulong Ao, Yonghua Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. RoboOS: A hierarchical embodied framework for 56 cross-embodiment and multi-agent collaboration.arXiv preprint arXiv:2505.03673, 2025. doi: 10.48550/...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.03673 2025

  50. [50]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. TMLR, 2024

    2024

  51. [51]

    GRUtopia: Dream general robots in a city at scale.arXiv preprint arXiv:2407.10943, 2024

    Hanqing Wang, Jiahe Chen, Wensi Huang, Qingwei Ben, Tai Wang, Boyu Mi, Tao Huang, Siheng Zhao, Yilun Chen, Sizhe Yang, Peizhou Cao, Wenye Yu, Zichao Ye, Jialun Li, Junfeng Long, Zirui Wang, Huiling Wang, Ying Zhao, Zhongying Tu, Yu Qiao, Dahua Lin, and Jiangmiao Pang. GRUtopia: Dream general robots in a city at scale.arXiv preprint arXiv:2407.10943, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    World modeling makes a better planner: Dual preference optimization for embodied task planning

    Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes a better planner: Dual preference optimization for embodied task planning. InACL, 2025

    2025

  53. [53]

    Communication-efficient desire alignment for embodied agent-human adaptation

    Yuanfei Wang, Xinju Huang, Fangwei Zhong, Yaodong Yang, Yizhou Wang, Yuanpei Chen, and Hao Dong. Communication-efficient desire alignment for embodied agent-human adaptation. arXiv preprint arXiv:2505.22503, 2025

    arXiv 2025

  54. [54]

    Set-of-mark promptingunleashesextraordinaryvisualgroundinginGPT-4V.arXivpreprintarXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark promptingunleashesextraordinaryvisualgroundinginGPT-4V.arXivpreprintarXiv:2310.11441, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision- driven embodied agents

    Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision- driven embodied agents. InICML, 2025

    2025

  56. [56]

    Holodeck: Language guided generation of 3D embodied AI environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Jae Sung Fox, Aniruddha Kembhavi, and Roozbeh Mottaghi. Holodeck: Language guided generation of 3D embodied AI environments. InCVPR, 2024

    2024

  57. [57]

    Being-0: A humanoid robotic agent with vision-language models and modular skills.arXiv preprint arXiv:2503.12533, 2025

    Haoqi Yuan, Yu Bai, Yuhui Fu, Bohan Zhou, Yicheng Feng, Xinrun Xu, Yi Zhan, Börje F Karlsson, and Zongqing Lu. Being-0: A humanoid robotic agent with vision-language models and modular skills.arXiv preprint arXiv:2503.12533, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Group sequence policy opti- mization.arXiv preprint arXiv:2507.18071, 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy opti- mization.arXiv preprint arXiv:2507.18071, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Sanketi, Grecia Salazar, Michael S

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski...

    2023