Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A vision-language planning and monitoring layer, added atop a standard two-level humanoid control stack, lets a 29-DoF robot autonomously complete a multi-step pick-and-place task, achieving 73% success over 40 real-world trials.

desk verdict Real system integration on a Unitree G1 with honest failure reporting, but the paper's headline claim that the VLM planner/monitor adds a benefit is not supported by the numbers: 73% integrated equals the product of the two skill rates (90% x 83%), and no baseline or ablation is reported. read the letter →

arxiv 2506.22827 v3 pith:XAMNDPA7 submitted 2025-06-28 cs.RO

André Schakkal , Ben Zandonati , Zhutian Yang , Navid Azizan This is my paper
classification cs.RO
keywords hierarchicalcontrolhumanoidmanipulationvision-languagemodelsimitationlearningreinforcementskillmonitoringmulti-steppick-and-place
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a humanoid robot can chain several separately learned manipulation skills into a longer, useful task without a human operator in the loop, and that the missing ingredient is a layer of vision-language reasoning on top of standard learning-based control. The central claim is that pretrained vision-language models can do two jobs at once: plan which skills to run in what order, and monitor whether the current skill has actually finished, using only short egocentric video clips. Deployed on a 29-degree-of-freedom Unitree G1 humanoid performing an obstructed bag pick-and-place, the integrated three-layer system completed the full sequence in 73% of 40 real-world trials. If this holds, then long-horizon humanoid manipulation may no longer be blocked by low-level control or by skill training, but by the comparatively cheap challenge of orchestrating and verifying skills, which can be addressed with existing models and natural-language skill descriptions.

What carries the argument

The central mechanism is the planner-monitor loop. The planner is a pretrained vision-language model that receives an initial image, a natural-language task description, and a library of skill descriptions written in a PDDL-like format (each with preconditions, effects, and an example verification question), and it outputs a grounded sequence of skill calls, each with a specific verification question. The monitor is a second, lightweight vision-language model that runs continuously at 1 Hz, sampling short video snippets and producing a binary 'completed / in progress' judgment for the current skill, which drives the skill selector to switch. Underneath, the mid-level imitation policy produces target joint angles for a single skill from egocentric binocular images and proprioception, and the low-level PPO tracking policy converts those targets into joint torque commands through a 200 Hz PD controller. Together the two VLM components replace the human who normally decides when one skill ends and the next begins.

What would settle it

Replace the VLM monitor's decisions with a fixed schedule (e.g., always switch from pick to place after the median pick duration observed in successful trials), keep everything else identical, and run the 40-trial evaluation; if the end-to-end success rate does not fall clearly below 73%, the monitor is not carrying the orchestration load. A second check is to score the monitor's binary judgments against hand-labeled ground truth on held-out video snippets; if the false-positive rate on 'bag not yet in hand' clips is high enough to explain the reported premature transitions, the central claim about reliable visual verification fails.

Watch

Extended reading notes

Core claim

The paper's own claim is that adding a high-level vision-language planning and monitoring layer to the canonical two-layer humanoid stack is sufficient to make a multi-step manipulation task autonomous. The lower two layers are standard: a PPO-trained whole-body tracking controller runs at 200 Hz, and a mid-level imitation transformer (based on action chunking) maps egocentric images and joint states to 50-step chunks of target joint angles for a single skill. The new layer is a closed loop between a VLM planner, which turns a natural-language goal and an initial image into an ordered list of grounded skills, and a VLM skill monitor, which at about 1 Hz looks at 10–15 frames from the past 1.5 seconds and answers a natural-language verification question with a binary done/in-progress signal. That signal triggers the transition to the next skill. Over 40 physical trials the full system succeeded 29 times, with failures seen in all three layers but most often in the imitation skill during grasping. The paper reads this as evidence that the orchestration layer works, and that the main practical cost is writing grounded verification questions for each skill.

Load-bearing premise

The load-bearing premise is that the vision-language monitor's binary 'done or not' verdict, drawn from 10–15 frames of a 1.5-second egocentric video clip at 1 Hz, is accurate enough to trigger skill transitions; the paper itself records optimistic monitor judgments that caused premature transitions, so this is the link most likely to break.

Editorial extensions

If this is right

  • New skills can be added to the robot by training one imitation policy and writing one natural-language skill description; the planner and monitor consume only the skill library, egocentric images, and text, so the orchestration layer does not need to be retrained.
  • Because the monitor closes the loop at 1 Hz with binary decisions, long-horizon tasks no longer require a human to judge skill completion, eliminating the main bottleneck in earlier two-layer humanoid systems.
  • The 1 Hz verification rate and the 1.5-second snippet length set a timescale constraint on the skill library: skills must be slow enough that their success is visually legible in about a second, so the framework suits deliberative manipulation rather than fast reactive recovery.
  • The PDDL-like natural-language skill descriptions make the plans and monitor questions inspectable, so failures can be attributed to one of three layers (planner mis-grounding, monitor misjudgment, or skill policy failure) rather than debugging an end-to-end network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An ablation that replaces the VLM monitor with a fixed time budget would clarify how much of the 73% comes from real-time verification versus the underlying skill policies; the paper does not perform that ablation, so the monitor's marginal contribution remains untested.
  • The same planner-monitor pattern should transfer to skills that change scene state beyond bag manipulation (opening doors, pushing objects, tool use), as long as the verification question isolates a visually legible effect within a 1.5-second window; this suggests the bottleneck for broader deployment is question design, not model capability.
  • Because the monitor's binary judgment is made from short RGB video, skills whose completion cues are subtle (e.g., precise alignment or depth relations) will be prone to the optimistic false positives the paper reports; writing questions that force the VLM to check relative positions, not just object presence, is a cheap extension that could reduce the most frequent monitor failure.
  • The framework implies a concrete design rule: pick the timescale of skill abstraction to match the monitor's refresh rate, so a robot using a faster monitor (or event-based vision) could support shorter, more dynamic skills, while the current 1 Hz monitor effectively forces skills to last several seconds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a three-layer hierarchical system for multi-step humanoid manipulation: a low-level RL-based whole-body tracking controller, a mid-level imitation-learned skill policy module (for picking and placing), and a high-level vision-language planning and monitoring module comprising a GPT-4o-based planner and a Gemini-Flash-Lite-based skill monitor. The system is evaluated on a Unitree G1 humanoid in a real-world pick-and-place task. The reported results are 90% success (27/30) for the pick skill, 83% (25/30) for the place skill, and 73% (29/40) for the integrated pick-and-place sequence. The paper also provides a qualitative failure analysis and discusses limitations such as monitor optimism and the 1 Hz verification rate.

Significance. If the authors' claims were fully supported, this would be a valuable demonstration of closed-loop VLM orchestration over learned whole-body skills on a humanoid platform, with attractive modularity and interpretability. The real-robot evaluation is a legitimate strength, and the authors are honest about failure modes, including premature monitor transitions. However, the experimental design does not currently isolate the contribution of the high-level VLM layer: the headline 73% success is statistically indistinguishable from the product of the two skill success rates (90% x 83% = 75%), and no baseline or ablation is provided. The paper's central claim that the results highlight the benefits of VLM-based planning and monitoring is therefore not supported by the data as presented. The planning challenge in the evaluated task is also minimal, since only two skills in a fixed order are needed.

major comments (3)
  1. [Abstract; Section IV-D, Table I] The central claim that the results "highlight the benefits of VLM-based skill planning and monitoring" is not supported by the experimental design. The full-system success rate of 29/40 (73%) is consistent with the product of the two independently measured skill success rates (0.90 x 0.83 = 0.75); the 95% Wilson confidence interval for 29/40 is approximately [56%, 85%], which fully contains 75%. Without a baseline that executes the two IL skills sequentially without the VLM planner/monitor, or an ablation that replaces the monitor with time-triggered transitions, the data cannot distinguish the hierarchical system from naive serial composition of the two skills. This is a load-bearing gap because the paper's stated contribution is the high-level VLM layer, and the current experiment does not establish that this layer contributes positively to task success.
  2. [Section III-C.5; Section IV-D] The VLM skill monitor is the only component that actively decides when to transition between skills, but its reliability is characterized only qualitatively. The paper lists "optimistic monitor judgments" as a failure mode that caused premature transitions, yet provides no quantitative measure of monitor accuracy (e.g., precision, recall, or at least the number of trials in each failure category). Because the monitor's binary decisions directly determine whether the pick and place skills are chained correctly, the authors should provide per-trial monitor decisions or an ablation comparing the monitor against a fixed-time transition rule or an oracle (human-labeled) transition signal. Without such data, the reader cannot assess whether the monitor is net beneficial or harmful, and the failure categorization remains anecdotal.
  3. [Section IV-B; Appendix C; Figure 1] The evaluated task involves only two skills in a fixed causal order (pick, then place), and the VLM planner's output is therefore a deterministic two-step sequence. This does not exercise the claimed capabilities of dynamic sequencing, handling of alternative orderings, obstacle avoidance (despite the push-aside skill shown in Fig. 1), or re-planning after a failed skill. The paper's contributions claim "dynamically sequences and verifies" skills and "autonomous execution of extended sequences," but the experiment provides no evidence of planning flexibility beyond a fixed script. The authors should either add a task variant that requires selecting among multiple skills or recovering from a failed execution, or explicitly temper the planning claims to reflect the current task's simplicity.
minor comments (5)
  1. [Section III-A.2; Appendix A.1; Table III] The number of parallel environments used for PPO training is inconsistent: Section III-A.2 states 4096, Appendix A.1 states 6144, and Table III lists 4096. Please correct this inconsistency.
  2. [Abstract; Section I; Section IV-B] The task is described as "non-prehensile pick-and-place" in the abstract and introduction, but Section IV-B defines picking as "grasp and lift the bag," which is prehensile. Please clarify the intended meaning or correct the terminology.
  3. [Appendix A.2] The reported mean keypoint position error of 13.88 cm seems large for whole-body tracking and could affect the downstream imitation-learning skills. Please discuss this value in the main text and relate it to the robot's morphology and task demands.
  4. [Section IV-D] The failure analysis lists categories "in order of frequency" but does not provide the number of trials in each category. Adding raw counts would make the failure analysis reproducible and more informative.
  5. [Section III-C.5] Please clarify whether the 10-15 frames used by the VLM monitor are sampled uniformly from the 1.5-second window, and report how many monitor evaluations were performed across the 40 trials.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the integrated system success rate is a direct real-world measurement, and no fitted parameter or self-citation chain is used to derive the headline claim.

full rationale

The paper's central numerical claim (73% success over 40 real-world trials) is an empirical measurement of the integrated system, not the output of a derivation or of a parameter fitted to a subset of the same data. The VLM planner and monitor are pretrained external models (GPT-4o and Gemini-2.0-Flash-Lite), and the mid-level IL skills are trained on teleoperated demonstrations; none of these components is fitted to the 29/40 headline result, and the full-system success condition is defined by the human experiment design (the robot correctly picks up the bag and places it on the target surface), not by the VLM monitor's own binary outputs. The observation that 29/40 is close to the product of the two per-skill success rates (0.90 x 0.83 = 0.747) is a legitimate concern about whether the VLM layer has an incremental causal benefit, but it is a missing-baseline and statistical-power issue, not circularity: the paper never claims to derive the full-system rate from the per-skill rates. Author self-citations ([23], [26], [29]) appear in motivation and future-work contexts and are not used to justify the main result, and no uniqueness theorem or ansatz is imported from the authors' prior work. The central claim therefore rests on direct measurements and on externally supported components rather than on a self-referential reduction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new physical entities or free parameters are introduced. The system is built from existing components (PPO tracker, HIT imitation policy, GPT-4o, Gemini-2.0-Flash-Lite). The main unverified assumptions are about the reliability of the VLM monitor, sim-to-real transfer, and the sufficiency of the demonstration data.

assumptions (4)
  • domain assumption The VLM skill monitor's binary completion signal at 1 Hz is sufficiently accurate and timely for orchestrating skill transitions.
    The system depends on monitor decisions to switch skills; the paper reports optimistic monitor judgments as a failure mode but provides no quantitative accuracy measure for the monitor.
  • domain assumption The low-level RL tracking policy trained in Isaac Gym transfers to the physical G1 robot without a sim-to-real gap that breaks the task.
    Section III-A and Appendix A report simulated tracking errors (3.4 degrees, 13.88 cm keypoint error) but provide no quantitative real-robot tracking evaluation.
  • domain assumption The retargeted human teleoperation demonstrations and the 30 successful demonstrations per skill are sufficient for the imitation policies to generalize to the test-time object and camera setup.
    Section III-B describes training on 30 successful demonstrations segmented by skill, but provides no analysis of demonstration diversity or policy generalization.
  • domain assumption AMASS motion data plus the custom teleoperation data cover the robot's operational range for the pick and place tasks.
    Section III-A states a curated AMASS subset plus custom data were used, but no details on the curation or coverage are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation." pith.science (2026). https://pith.science/paper/XAMNDPA7

@misc{pith2026250622827,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Vision-Language Planning for Multi-Step Humanoid Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAMNDPA7}},
  note         = {Machine review of arXiv:2506.22827}
}
read the original abstract

Enabling humanoid robots to reliably execute complex multi-step manipulation tasks is crucial for their effective deployment in industrial and household environments. This paper presents a hierarchical planning and control framework designed to achieve reliable multi-step humanoid manipulation. The proposed system comprises three layers: (1) a low-level RL-based controller responsible for tracking whole-body motion targets; (2) a mid-level set of skill policies trained via imitation learning that produce motion targets for different steps of a task; and (3) a high-level vision-language planning module that determines which skills should be executed and also monitors their completion in real-time using pretrained vision-language models (VLMs). Experimental validation is performed on a Unitree G1 humanoid robot executing a non-prehensile pick-and-place task. Over 40 real-world trials, the hierarchical system achieved a 73% success rate in completing the full manipulation sequence. These experiments confirm the feasibility of the proposed hierarchical system, highlighting the benefits of VLM-based skill planning and monitoring for multi-step manipulation scenarios. See https://vlp-humanoid.github.io/ for video demonstrations of the policy rollout.

Figures

Figures reproduced from arXiv: 2506.22827 by the authors.

Figure 1
Figure 1. Our hierarchical humanoid manipulation system autonomously executes a multi-step rearrangement task. The robot first pushes aside an obstacle, then picks up a bag and places it onto another surface. The system uses pretrained VLMs to orchestrate manipulation policies that output whole-body joint positions, which serve as tracking targets for a low-level RL policy. Abstract—Enabling humanoid robots to reliably execut… view at source ↗
Figure 2
Figure 2. Overview of the proposed hierarchical framework for autonomous multi-step humanoid manipulation. The system [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example structured skill description provided to the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Real-time teleoperation on the physical Unitree G1 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Overview of the Teleoperation Module pipeline. Human [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Example of human pose detection by HybrIK, illustrat [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Example system prompt for VLM skill planning. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Example generated prompt for VLM skill planning. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Example VLM skill planning results [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Closing the Loop in Humanoid VLA: Persistent 3D Object Tokens for Verifiable Loco-Manipulation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Persistent role-indexed 3D object tokens that condition both action generation and geometric verification improved a GR00T-N1.7 humanoid's loco-manipulation success from 39/80 to 71/80 across eight real-world task families.

Reference graph

Works this paper leans on

40 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pddl| the planning domain definition language

    Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. Pddl| the planning domain definition language. Technical Report, Tech. Rep., 1998

    1998

  2. [2]

    RT-H: Action Hierarchies Using Language

    Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, De- bidatta Dwibedi, and Dorsa Sadigh. RT-H: Action Hierarchies Using Language. arXiv, June 2024. doi: 10.48550/arXiv.2403.01823. arXiv:2403.01823 [cs]

    arXiv doi:10.48550/arxiv.2403.01823 2024

  3. [3]

    Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y . Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mo- hith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy...

    arXiv doi:10.48550/arxiv.2504.16054 2025

  4. [4]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Flo- rence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashn...

    arXiv doi:10.48550/arxiv.2307.15818 2023

  5. [5]

    NaVILA: Legged Robot Vision-Language-Action Model for Navigation

    An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. NaVILA: Legged Robot Vision-Language-Action Model for Navigation. arXiv, February 2025. doi: 10.48550/arXiv.2412.04453. arXiv:2412.04453 [cs]

    arXiv doi:10.48550/arxiv.2412.04453 2025

  6. [6]

    Expressive Whole-Body Control for Humanoid Robots

    Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive Whole-Body Control for Humanoid Robots. arXiv, March 2024. doi: 10.48550/arXiv.2402.16796. arXiv:2402.16796 [cs]

    arXiv doi:10.48550/arxiv.2402.16796 2024

  7. [7]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. page 02783649241273668. SAGE Publications Sage UK: London, England, 2023

    work page 2023

  8. [8]

    Local Policies Enable Zero-shot Long-horizon Manipu- lation

    Murtaza Dalal, Min Liu, Walter Talbott, Chen Chen, Deepak Pathak, Jian Zhang, and Ruslan Salakhutdinov. Local Policies Enable Zero-shot Long-horizon Manipu- lation. arXiv, March 2025. doi: 10.48550/arXiv.2410. 22332. arXiv:2410.22332 [cs]

    arXiv doi:10.48550/arxiv.2410.22332 2025

Show all 40 references
  1. [9]

    Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling

    Xiaolin Fang, Bo-Ruei Huang, Jiayuan Mao, Jasmine Shone, Joshua B. Tenenbaum, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Keypoint Abstraction using Large Models for Object-Relative Imitation Learning. arXiv, October 2024. doi: 10.48550/arXiv.2410.23254. arXiv:2410.23254 [cs]

    2024 doi arXiv

  2. [10]

    HumanPlus: Humanoid Shadowing and Imitation from Humans

    Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. HumanPlus: Humanoid Shadowing and Imitation from Humans. arXiv, June 2024. doi: 10.48550/arXiv.2406.10454. arXiv:2406.10454 [cs]

    2024 doi arXiv

  3. [11]

    Karen Liu, Abder- rahmane Kheddar, Xue Bin Peng, Yuke Zhu, Guanya Shi, Quan Nguyen, Gordon Cheng, Huijun Gao, and Ye Zhao

    Zhaoyuan Gu, Junheng Li, Wenlan Shen, Wenhao Yu, Zhaoming Xie, Stephen McCrory, Xianyi Cheng, Ab- dulaziz Shamsah, Robert Griffin, C. Karen Liu, Abder- rahmane Kheddar, Xue Bin Peng, Yuke Zhu, Guanya Shi, Quan Nguyen, Gordon Cheng, Huijun Gao, and Ye Zhao. Humanoid Locomotion ...

    2025 doi arXiv

  4. [12]

    OmniH2O: Universal and Dexter- ous Human-to-Humanoid Whole-Body Teleoperation and Learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. OmniH2O: Universal and Dexter- ous Human-to-Humanoid Whole-Body Teleoperation and Learning. arXiv, June 2024. doi: 10.48550/arXiv.2406. 08858. arXiv:2406.08858 [cs]

    2024 doi arXiv

  5. [13]

    Learning human- to-humanoid real-time whole-body teleoperation

    Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human- to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8944–8951. IEEE, 2024

    2024

  6. [14]

    ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills

    Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang, Jiashun Wang, Zhengyi Luo, Guanqi He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, Linxi "Jim" Fan, Yuke Zhu, Changliu Liu, and Guanya Shi. ASAP: Aligning Simulation and Real-World P...

    2025 doi arXiv

  7. [15]

    Learning visual quadrupedal loco-manipulation from demonstrations

    Zhengmao He, Kun Lei, Yanjie Ze, Koushil Sreenath, Zhongyu Li, and Huazhe Xu. Learning visual quadrupedal loco-manipulation from demonstrations. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 9102–9109. IEEE, 2024

    2024

  8. [16]

    ExBody2: Advanced Expressive Humanoid Whole-Body Control

    Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, and Xiaolong Wang. ExBody2: Advanced Expressive Humanoid Whole-Body Control. arXiv, March 2025. doi: 10.48550/arXiv.2412.13196. arXiv:2412.13196 [cs]

    2025 doi arXiv

  9. [17]

    OpenVLA: An Open-Source Vision-Language-Action Model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA...

    2024 doi arXiv

  10. [18]

    Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation

    Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, and Cewu Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3383– 3393, 2021

    2021

  11. [19]

    HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation

    Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation. arXiv, February 2025. doi: 10.48550/arXiv.2502....

    2025 doi arXiv

  12. [20]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019

    2019

  13. [21]

    Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. arXiv, August 2021. doi: 10.4...

    2021 doi arXiv

  14. [22]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A. J. Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, et al. GPT-4o System Card. arXiv, October 2024. doi: 10.48550/arXiv.2410...

    2024 doi arXiv

  15. [23]

    Quantifying representation reliability in self- supervised learning models

    Young-Jin Park, Hao Wang, Shervin Ardeshir, and Navid Azizan. Quantifying representation reliability in self- supervised learning models. In Uncertainty in Artificial Intelligence, pages 2835–2860. PMLR, 2024

    2024

  16. [24]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learning...

    2021

  17. [25]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv, August 2017. doi: 10.48550/arXiv. 1707.06347. arXiv:1707.06347 [cs]

    2017 doi arXiv

  18. [26]

    Sketching curvature for efficient out-of-distribution de- tection for deep neural networks

    Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of-distribution de- tection for deep neural networks. In Uncertainty in artificial intelligence, pages 1958–1967. PMLR, 2021

    1958

  19. [27]

    Hi Robot: Open-Ended Instruction Follow- ing with Hierarchical Vision-Language-Action Models

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liy- iming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li- Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi Robot: Open-Ended Instruction Follow- ing with Hiera...

    2025 doi arXiv

  20. [28]

    Gemini robotics: Bringing ai into the physical world

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:250...

    2025 arXiv

  21. [29]

    Guiding Long- Horizon Task and Motion Planning with Vision Language Models

    Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Guiding Long- Horizon Task and Motion Planning with Vision Language Models. In ICRA. IEEE, 2025

    2025

  22. [30]

    RoboPoint: A Vision- Language Model for Spatial Affordance Prediction for Robotics

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Ar- salan Mousavian, and Dieter Fox. RoboPoint: A Vision- Language Model for Spatial Affordance Prediction for Robotics. arXiv, June 2024. doi: 10.48550/arXiv.2406. 10721. arXiv:24...

    2024 doi arXiv

  23. [31]

    Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn

    Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipu- lation with Low-Cost Hardware. arXiv, April 2023. doi: 10.48550/arXiv.2304.13705. arXiv:2304.13705 [cs]. APPENDIX A. Low-Level Tracking Policy In this section, we describe addit...

    2023 doi arXiv

  24. [32]

    Each hidden layer contains 256 units and ReLU activations

    Policy design and training procedure: The policy fol- lows an actor-critic paradigm with a three-layer perception (MLP) backbone. Each hidden layer contains 256 units and ReLU activations. The system is trained by PPO in Isaac gym using a 0.02 integration timestep and 6144 par...

  25. [33]

    Example real-time teleoperation is shown in Fig

    Deployment evaluation: Following successful validation of the tracking policy in simulation, we deploy the policy on the physical Unitree G1 robot. Example real-time teleoperation is shown in Fig. 4. Quantitatively, we evaluate the tracking performance of our trained policy in...

  26. [34]

    Teleoperation Architecture: An overall diagram for the RGB teleoperation architecture is shown in Fig. 5. The joint retargeting procedure is outlined in Algorithm 1 while an example of the human pose estimation used for teleoperation is shown in Fig. 6. Algorithm 1 Retargeting...

  27. [35]

    Imitation Learning Policy Details: Hyperparameters for the Humanoid Imitation Transformer (HIT) [10] implementa- tion are outlined in Tab. VI. RGB Teleoperation Human Pose Estimation Motion Retargeting Keypoint Generation RGB Image Estimated Human Skeleton Target Joint Angles ...

  28. [36]

    , πk} based on visual inputs and task prompts

    The VLM Planner Πplan vlm generates a sequence of skills {π1, π2, . . . , πk} based on visual inputs and task prompts. In this example, the sequence would be pick, followed by place

  29. [37]

    The robot begins executing the first skill in the sequence, here the pick skill, governed by the trained imitation learning policy πpick

  30. [38]

    In this example, the monitor verifies whether the object (a) Human operator (b) HybrIK pose detection result Fig

    Concurrently, the VLM Skill Monitor actively observes the robot’s real-time visual feedback to assess whether the current skill (i.e., picking the object) has been completed. In this example, the monitor verifies whether the object (a) Human operator (b) HybrIK pose detection ...

  31. [39]

    Once the monitor confidently determines the com- pletion of this skill, it signals the transition to the next planned skill

    If the Skill Monitor indicates the skill remains in- progress, the robot continues executing the current pick skill. Once the monitor confidently determines the com- pletion of this skill, it signals the transition to the next planned skill

  32. [40]

    skill_name

    Upon receiving a completed indication from the Monitor, the robot initiates the subsequent skill in the planned sequence—in this case, the place skill, governed by the corresponding policy πplace. D. Example VLM prompts Fig. 7 shows the system prompt used in OpenAI API, Fig. 8...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.