Pith. sign in

REVIEW 4 major objections 4 minor 70 references

SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SafeMimic lets a robot learn multi-step manipulation from a single human video, safely and autonomously.

desk verdict Strong systems paper for learning mobile manipulation from a single human video, but the safety-filter validation is the load-bearing weakness and needs more evidence. read the letter →

arxiv 2506.15847 v1 pith:KRTY3NFQ submitted 2025-06-18 cs.RO cs.AI

Arpit Bahety , Arnav Balaji , Ben Abbatematteo , Roberto Martín-Martín This is my paper
classification cs.ROcs.AI
keywords learningfromdemonstrationmobilemanipulationsafetyQ-functionssim-to-realtransferhumanvideoparsingsafeexplorationbacktrackingpolicymemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents SafeMimic, a framework that lets a mobile manipulator learn a multi-step task—like fetching and refrigerating an item—by watching a single third-person video of a human doing it, without a human supervisor. It claims this works across different demonstrators and environments, and that the robot both succeeds and stays safe during the learning process, reporting an unsafe-action rate of 0.6%. The result matters because it removes two bottlenecks in learning from human video: costly teleoperated data collection and constant human monitoring of robot exploration. A robot could therefore be taught a useful home skill from one demonstration and then refine that skill on its own.

What carries the argument

The load-bearing object is the ensemble of safety Q-functions, each predicting the probability that a state-action pair leads to a specific failure mode such as collision, joint-limit violation, excessive force, grasp loss, or dropping. Trained in the OmniGibson simulator on pointcloud-plus-proprioception inputs with domain randomization, the ensemble is applied zero-shot to the real robot and filters sampled actions in a receding-horizon fashion. Together with a one-step inverse-action backtracking routine and an optional grasp-mode switch, this machinery turns a parsed human plan into a safe search over nearby actions that adapts the demonstration to the robot's own body.

What would settle it

Deploy SafeMimic in scenes containing objects and layouts not represented in the simulator's domain-randomized set, such as deformable materials, mirrored surfaces, or cluttered counters with transparent containers, and measure the unsafe-action rate during learning. If the rate climbs well above the reported 0.6% or the robot consistently flags all candidate actions as unsafe and stalls, the claim that the safety critic generalizes zero-shot is contradicted.

Watch

Extended reading notes

Core claim

SafeMimic's central claim is that a multi-step mobile manipulation skill can be acquired from one human video by factorizing the demonstration into single semantic-change segments, translating the human motion to the robot's egocentric frame, and then refining each segment through safe, autonomous trial and error. Safety comes from an ensemble of safety Q-functions pretrained in a domain-randomized simulator that score candidate actions before execution; when no sampled action is predicted safe, the robot backtracks to earlier states and tries alternative trajectories or grasp modes. The paper reports at least 40% final success across seven tasks, outperforming direct execution, exploration without safety filtering, and imitation-learning baselines, with the safety filter keeping the unsafe-action rate near zero.

Load-bearing premise

The load-bearing premise is that safety Q-functions trained in a simulator transfer zero-shot to the real world accurately enough to keep unsafe actions near zero while still allowing the robot to explore; if real scenes differ too much from the simulated training scenarios, safe exploration breaks down.

Editorial extensions

If this is right

  • A robot can acquire a new multi-step mobile manipulation skill from a single human video without a human in the loop for monitoring or resets.
  • The same safety Q-function ensemble transfers across tasks, so new tasks do not require task-specific safety data collection.
  • Backtracking plus grasp-mode exploration lets the robot overcome embodiment differences, for example switching from a human's top-down grasp to a side grasp that fits the robot's arm.
  • The policy memory module carries successful strategies forward, reducing exploration on later attempts by up to 67% for the tasks tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the safety critic consumes pointclouds and proprioception rather than task-specific features, the same pretrained ensemble could plausibly serve other mobile manipulators, though the paper does not test cross-embodiment transfer.
  • The vision-language parsing step is a likely bottleneck: a mislabeled semantic goal would send the safe-exploration stage to the wrong target, so the pipeline's robustness depends on the parser's reported 86-100% segment accuracy.
  • A natural testable extension, which the paper itself suggests, is updating the safety critics from real-world failures after deployment; measuring whether a small number of real corrections lowers the already low false-negative rate would sharpen the safety claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. SafeMimic proposes a framework for learning multi-step mobile manipulation tasks from a single third-person human video. The pipeline parses the video into semantic segments using human pose tracking and a vision-language model, translates the demonstrated motions to the robot's egocentric frame, and then adapts the behavior through real-world exploration. Exploration is gated by an ensemble of safety Q-functions pretrained in OmniGibson, with backtracking and grasp-mode switching when safe forward progress is not possible. A policy memory stores successful strategies to reduce exploration in later attempts. The paper reports experiments on seven real-robot tasks with five trials each, comparing against direct execution, exploration without safety Q-functions, and imitation-learning baselines, and reports a 0.6% unsafe-action rate, robustness across users and environments, and reduced exploration with policy memory.

Significance. If the safety and success claims hold, SafeMimic would be a meaningful advance toward autonomous robot learning from single human demonstrations in real settings. The paper's strengths are its real-robot evaluation across seven multi-step tasks, the explicit ablations with and without safety Q-functions, the inclusion of grasp-mode adaptation as a first-class mechanism, and a candid limitations section that identifies the sim-to-real coverage requirement, the limited failure modes, and the irreversibility of some failures. The framework is well motivated and the components are clearly described. However, the central safety claim currently rests on an unvalidated decision threshold for the safety critic, and several empirical claims use small samples without statistical support; these issues need to be addressed before the abstract's claims can be taken at face value.

major comments (4)
  1. [III-B, Table I, Q2, Section V] The safety claim depends on the operating point ε=0.7 in Table I applied to the safety Q-function defined in Section III-B as the probability that action a_t in state s_t leads to failure. With this definition, treating Q_safe ≤ ε as safe permits actions with an estimated failure probability as high as 70%. The paper provides no ROC curve, reliability diagram, false-negative rate, or held-out real-world evaluation of the SQF ensemble under sim-to-real shift, and Section V explicitly concedes that the simulated training data must cover scenarios encountered during real-world exploration. The reported 0.6% unsafe-action rate (Q2) is a useful system-level metric, but it cannot distinguish a well-calibrated filter from a conservative critic that was adequate on these seven tasks by chance. Please re-justify the threshold, report calibration statistics on real data, and give per-failure-mode counts with confidence intervals.
  2. [IV, Q1, Fig. 4] The central empirical claim of 'significantly outperforming' all baselines is based on five trials per task with no error bars, confidence intervals, or statistical tests. In Fig. 4, several comparisons are 100% versus 0% on five trials, but others involve small margins or overlapping curves, and the text does not report per-trial data or a pre-specified test. The same issue applies to Q3 and Q4, which report three trials. Please provide per-trial results, exact binomial confidence intervals, and either a statistical test or a careful softening of the significance claims.
  3. [IV, IL baselines] The imitation-learning baselines are not evaluated on the full seven-task problem: the text states they were evaluated only on 'Place', 'Open', and 'Close' segments and were provided successful navigation and picking from SafeMimic's trials. This is disclosed, but it means the abstract's claim of improvements 'across seven tasks' over these baselines is not a full-task comparison. Please clarify in the results and abstract that the IL comparison is segment-level, or evaluate the IL baselines on the full task under the same initial conditions.
  4. [III-A, III-B, Algorithm 1] Success verification uses the same vision-language model (GPT-4o) that generated the semantic segmentation and labels. If the VLM mislabels a semantic change, it can both create the segment goal and certify that the goal has been achieved, potentially inflating reported success rates. The paper does not report human verification of success or an agreement rate between VLM success detection and human judgment. Please provide a validation of the success-detection mechanism, for example by auditing a subset of trials with human labels.
minor comments (4)
  1. [Table I vs. Fig. 10] Table I reports the SQF dataset size as 9,000 while Fig. 10 states that 43,200 state-action pairs were collected; please clarify whether these numbers refer to different stages (e.g., raw pairs versus filtered/labeled pairs) or whether one of them is a typo.
  2. [Algorithm 1, line 40] The filtering condition in the pseudocode, which keeps trajectories when Q^k_safe(s_t,a_0) < ε, appears inconsistent with the main text in Section III-B, where actions with Q_safe > ε are treated as unsafe and trigger backtracking. Please fix the pseudocode so that the safe/unsafe condition matches the described behavior.
  3. [III-C, Q3] The Q3 text refers to the policy memory as 'Sec. III.D' but the module is described in Section III-C; please correct the cross-reference.
  4. [Q2, unsafe action rate] The unsafe-action rate is defined as a pooled ratio of unsafe actions to total actions across all trials; since episode lengths vary, please also report per-episode rates or confidence intervals so that the metric is not dominated by longer episodes.

Circularity Check

1 steps flagged · score 3.0 of 10

The headline success metric is partly self-referential because the same VLM that parses semantic goals also confirms their achievement, though the safety-Q-function and exploration claims remain independently grounded.

  1. self definitional [Section III-A (semantic parsing) and Section IV Q1 (success verification)]
    "Each segment is then annotated with its semantic goal, again obtained with a VLM query. ... success is confirmed by prompting a VLM with the observation and semantic goal."

    The success signal is not anchored to an independent ground-truth check. The semantic goal that defines a successfully completed segment is produced by VLM queries during parsing, and the same class of VLM is later asked whether that goal is achieved. If the VLM systematically mislabels a goal or is self-consistent in its interpretation, the robot can be scored as successful when the semantic change it verifies is the VLM's own prior output. The reported per-stage success rates therefore partly measure self-consistency of the VLM with its own labels rather than an externally verified completion of the demonstrated behavior.

full rationale

SafeMimic's derivation chain is mostly self-contained. The safety Q-functions are pretrained in OmniGibson on domain-randomized failure data and are not fitted to the real-world success metric; the unsafe-action rate is labeled by a human monitor, so the main safety comparison is externally grounded. The policy-memory reduction is by design rather than circular, and the self-citation to ScrewMimic appears in related work without being load-bearing. The one identifiable circular element is evaluational: the same VLM supplies semantic goals during parsing and confirms success during execution, making the headline success metric partially self-referential. This does not collapse the central claim, because the robot still executes real motor commands and the safety/failure claims rest on independent labels, but it prevents the reported success rates from being treated as fully independent evidence of task completion.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on empirical assumptions about sim-to-real transfer of learned safety models, the reliability of VLM-based goal detection, and initial pose correspondence. The hyperparameters listed are not fitted to the success metric but are chosen by the authors and affect safety and exploration. No new physical or conceptual entities are introduced beyond the modules described.

free parameters (5)
  • SQF threshold epsilon = 0.7
    Actions with Q_safe > epsilon are treated as unsafe; chosen by hand, affects the safety-exploration tradeoff. Section III-B and Table I.
  • Exploration standard deviation sigma = 0.05
    Noise added to the human-demonstrated trajectory when sampling candidate actions. Table I.
  • Number of action samples m = not specified in text
    Used in Algorithm 1 to sample candidate trajectories; value not reported, so reproducibility is limited.
  • Max actions before grasp switching = 50
    After 50 actions without progress, SafeMimic backtracks to a grasping segment and tries a new grasp mode. Section III-B.
  • Number of grasp modes explored = 3
    The robot explores among a discrete set of 3 grasp options from the grasp generator. Section III-B.
assumptions (6)
  • domain assumption Safety Q-functions trained in simulation with domain randomization transfer zero-shot to real-world pointclouds and proprioception.
    Used throughout Section III-B and IV; if this transfer fails, safe exploration collapses. The authors acknowledge in Section V that training data must cover similar scenarios.
  • domain assumption The robot starts close enough to the human's initial position that relative motion commands produce similar trajectories without calibration.
    Section III-A: 'SAFEMIMIC assumes that in navigation segments, the robot starts from a location close enough to the human initial location (no calibration needed!)'.
  • domain assumption The VLM correctly segments the demo and correctly verifies segment completion during exploration.
    The VLM is used for both parsing (what?) and success detection; errors could cause wrong learning signals. Sections III-A, III-B, and V.
  • domain assumption Backtracking by executing the inverse action restores the environment to the previous state.
    Algorithm 1 and Section III-B; the authors note in Section V that this fails for irreversible events like dropping objects.
  • domain assumption The grasp generator provides a discrete set of feasible grasps that includes one compatible with the robot's morphology for the task.
    SafeMimic relies on grasp pose generation [64] and explores only among those candidates. Section III-B and V.
  • standard math MDP and Q-function definitions as in Srinivasan et al. [45].
    Used to define the safety Q-function and failure indicators in Section III-B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation." pith.science (2026). https://pith.science/paper/KRTY3NFQ

@misc{pith2026250615847,
  author       = {Pith},
  title        = {Pith review of: SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KRTY3NFQ}},
  note         = {Machine review of arXiv:2506.15847}
}
read the original abstract

For robots to become efficient helpers in the home, they must learn to perform new mobile manipulation tasks simply by watching humans perform them. Learning from a single video demonstration from a human is challenging as the robot needs to first extract from the demo what needs to be done and how, translate the strategy from a third to a first-person perspective, and then adapt it to be successful with its own morphology. Furthermore, to mitigate the dependency on costly human monitoring, this learning process should be performed in a safe and autonomous manner. We present SafeMimic, a framework to learn new mobile manipulation skills safely and autonomously from a single third-person human video. Given an initial human video demonstration of a multi-step mobile manipulation task, SafeMimic first parses the video into segments, inferring both the semantic changes caused and the motions the human executed to achieve them and translating them to an egocentric reference. Then, it adapts the behavior to the robot's own morphology by sampling candidate actions around the human ones, and verifying them for safety before execution in a receding horizon fashion using an ensemble of safety Q-functions trained in simulation. When safe forward progression is not possible, SafeMimic backtracks to previous states and attempts a different sequence of actions, adapting both the trajectory and the grasping modes when required for its morphology. As a result, SafeMimic yields a strategy that succeeds in the demonstrated behavior and learns task-specific actions that reduce exploration in future attempts. Our experiments show that our method allows robots to safely and efficiently learn multi-step mobile manipulation behaviors from a single human demonstration, from different users, and in different environments, with improvements over state-of-the-art baselines across seven tasks

Figures

Figures reproduced from arXiv: 2506.15847 by the authors.

Figure 1
Figure 1. Robot imitating a single video of a human-demonstrated mobile [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of SAFEMIMIC. From an RGB-D video of a human performing a multi-step mobile manipulation task acquired by the robot, SAFEMIMIC uses a combination of human pose tracking models [4, 5] and VLM prompting to perform coarse-to-fine segmentation obtaining semantic changes –“what?”– and human action trajectories –“how?”–, and translating them to the robot’s point of view (left, Sec. III-A). SAFEMIMIC then refines … view at source ↗
Figure 3
Figure 3. Multi-Step Mobile Manipulation Evaluation Tasks. SAFEMIMIC is evaluated on seven complex multi-step tasks combining navigation and manipulation. From left to right: boxing an item, shelving an item, store_in_drawer, erase_whiteboard, refrigerating an item, fill_pot, load_oven. The tasks involve risky contact-rich phases, grasping steps and the manipulation of constrained mechanisms. SAFEMIMIC is able to adapt action… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Accumulated Success on Multi-Step Tasks. Accumulated success rate at each stage of each of the seven evaluated multi-step mobile manipulation tasks, indicating the percentage of the five trials each method completed up to and including that segment. We compare SAFEMIMI…
Figure 5
Figure 5. Figure 5: Grasping mode adaptation. Two examples (top and bottom rows) of SAFEMIMIC’s grasping mode adaptation. Left column: human demonstrated grasp. Middle column: robot failing when attempting the task by matching the human grasp. Right column: robot succeeding in the task th…
Figure 7
Figure 7. Figure 7: Exploration Reduction with Policy Memory. Number of actions explored by SAFEMIMIC with (right) and without policy memory (left). Successful attempts from an initial exploration are recorded and use to train the policy memory. Object poses are varied relative to the rob…
Figure 9
Figure 9. Figure 9: Snapshots of three different human demonstrations of the same task [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 8
Figure 8. Figure 8: Human video parsing by SAFEMIMIC. An initial RGB-D video demonstration is processed by SAFEMIMIC using a body tracking solution to obtain segments where the human is either navigating or performing stationary manipulation. Initial stationary manipulation segments are f…
Figure 10
Figure 10. Figure 10: Simulation training domains for the safety Q-function functions of [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Backtracking actions in SAFEMIMIC. Example sequence of back￾tracking behavior in Shelving task. When no safe samples are available, or when a segment exploration ends without success, SAFEMIMIC backtracks to previous states by undoing the last actions, possibly steppi…
Figure 12
Figure 12. Figure 12: Snapshots of the videos we evaluate SAFEMIMIC’s video parsing module on in C. We test the module on videos with different camera angles (left column), different levels of clutter (middle column), and different lighting conditions (right column). Our results indicate t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 45 canonical work pages

  1. [1]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Ab- hishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024

    2024

  2. [2]

    Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- win Balakrishna, Sudeep Dasari, Siddharth Karam- cheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset.arXiv preprint arXiv:2403.12945, 2024

    arXiv 2024

  3. [3]

    Bridge data: Boosting generalization of robotic skills with cross- domain datasets.arXiv preprint arXiv:2109.13396, 2021

    Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross- domain datasets.arXiv preprint arXiv:2109.13396, 2021

    arXiv 2021

  4. [4]

    Decoupling human and camera motion from videos in the wild

    Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2023

    work page 2023

  5. [5]

    Reconstructing hands in 3D with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. InCVPR, 2024

    2024

  6. [6]

    Understanding human hands in contact at internet scale

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    work page 2020

  7. [7]

    Frankmo- cap: A monocular 3d whole-body pose estimation sys- tem via regression and integration

    Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmo- cap: A monocular 3d whole-body pose estimation sys- tem via regression and integration. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749–1759, 2021

    work page 2021

  8. [8]

    Human-to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. 2022

    work page 2022

Show all 70 references
  1. [9]

    Deft: Dexterous fine- tuning for hand policies

    Aditya Kannan, Kenneth Shaw, Shikhar Bahl, Pragna Mannam, and Deepak Pathak. Deft: Dexterous fine- tuning for hand policies. InConference on Robot Learning, pages 928–942, 2023

    2023

  2. [10]

    Videodex: Learning dexterity from internet videos

    Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654–665, 2023

    2023

  3. [11]

    ScrewMimic: Bimanual Im- itation from Human Videos with Screw Space Projection

    Arpit Bahety, Priyanka Mandikal, Ben Abbatematteo, and Roberto Mart´ın-Mart´ın. ScrewMimic: Bimanual Im- itation from Human Videos with Screw Space Projection. InRobotics: Science and Systems. Robotics: Science and Systems Foundation, 2024

    2024

  4. [12]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jack- son Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  5. [13]

    Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube

    Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: Learning a robotic hand imitator by watching humans on youtube. InRobotics: Science and Systems, 2022

    2022

  6. [14]

    Learn- ing by watching: Physical imitation of manipulation skills from human videos

    Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, and Animesh Garg. Learn- ing by watching: Physical imitation of manipulation skills from human videos. In2021 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 7827–78...

    2021

  7. [15]

    Learn- ing generalizable robotic reward functions from” in-the- wild” human videos.arXiv preprint arXiv:2103.16817, 2021

    Annie S Chen, Suraj Nair, and Chelsea Finn. Learn- ing generalizable robotic reward functions from” in-the- wild” human videos.arXiv preprint arXiv:2103.16817, 2021

    2021 arXiv

  8. [16]

    Concept2robot: Learning manipu- lation concepts from instructions and human demonstra- tions.The International Journal of Robotics Research, 40(12-14):1419–1434, 2021

    Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning manipu- lation concepts from instructions and human demonstra- tions.The International Journal of Robotics Research, 40(12-14):1419–1434, 2021

    2021

  9. [17]

    Liv: Language-image representations and rewards for robotic control.arXiv preprint arXiv:2306.00958, 2023

    Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayara- man. Liv: Language-image representations and rewards for robotic control.arXiv preprint arXiv:2306.00958, 2023

    2023 arXiv

  10. [18]

    Vip: Towards universal visual reward and represen- tation via value-implicit pre-training.arXiv preprint arXiv:2210.00030, 2022

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayara- man, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and represen- tation via value-implicit pre-training.arXiv preprint arXiv:2210.00030, 2022

    arXiv

  11. [19]

    R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation.arXiv preprint arXiv:2203.12601, 2022

    2022 arXiv

  12. [20]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416–426. PMLR, 2023

    2023

  13. [21]

    Zero-shot robot manipulation from passive human videos, 2023

    Homanga Bharadhwaj, Abhinav Gupta, Shubham Tul- siani, and Vikash Kumar. Zero-shot robot manipulation from passive human videos, 2023

    2023

  14. [22]

    Structured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. 2023

    2023

  15. [23]

    Okami: Teaching humanoid robots manipulation skills through single video imitation

    Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, and Yuke Zhu. Okami: Teaching humanoid robots manipulation skills through single video imitation. InConference on Robot Learning, 2024

    2024

  16. [24]

    Expressive whole- body control for humanoid robots.arXiv preprint arXiv:2402.16796, 2024

    Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole- body control for humanoid robots.arXiv preprint arXiv:2402.16796, 2024

    2024 arXiv

  17. [25]

    Nonpara- metric motion retargeting for humanoid robots on shared latent space

    Sungjoon Choi, Matt Pan, and Joohyung Kim. Nonpara- metric motion retargeting for humanoid robots on shared latent space. In16th Robotics: Science and Systems, RSS

  18. [26]

    Learning human- to-humanoid real-time whole-body teleoperation.arXiv preprint arXiv:2403.04436, 2024

    Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human- to-humanoid real-time whole-body teleoperation.arXiv preprint arXiv:2403.04436, 2024

    2024 arXiv

  19. [27]

    The ingredients of real-world robotic re- inforcement learning.arXiv preprint arXiv:2004.12570, 2020

    Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic re- inforcement learning.arXiv preprint arXiv:2004.12570, 2020

    2004 arXiv

  20. [28]

    Continuously improving mobile manipulation with autonomous real- world rl

    Russell Mendonca and Deepak Pathak. Continuously improving mobile manipulation with autonomous real- world rl. InRSS 2024 Workshop: Data Generation for Robotics

    2024

  21. [29]

    Self-improving robots: End-to-end au- tonomous visuomotor reinforcement learning

    Archit Sharma, Ahmed M Ahmed, Rehaan Ahmad, and Chelsea Finn. Self-improving robots: End-to-end au- tonomous visuomotor reinforcement learning. InCon- ference on Robot Learning. PMLR, 2023

    2023

  22. [30]

    Provable reset- free reinforcement learning by no-regret reduction

    Hoai-An Nguyen and Ching-An Cheng. Provable reset- free reinforcement learning by no-regret reduction. In International Conference on Machine Learning, pages 25939–25955. PMLR, 2023

    2023

  23. [31]

    Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning

    Homer Rich Walke, Jonathan Heewon Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning. InConference on Robot Learning, pages 1652–1662. PMLR, 2023

    2023

  24. [32]

    Deep reinforcement learning for robotics: A survey of real- world successes.Annual Review of Control, Robotics, and Autonomous Systems, 8, 2024

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chan- dra, Roberto Mart ´ın-Mart´ın, and Peter Stone. Deep reinforcement learning for robotics: A survey of real- world successes.Annual Review of Control, Robotics, and Autonomous Systems, 8, 2024

    2024

  25. [33]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St ´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelli- gence and statistics, pages 627–635. JMLR Workshop and ...

    2011

  26. [34]

    Dart: Noise injection for robust imitation learning

    Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. InConference on robot learning, pages 143–156. PMLR, 2017

    2017

  27. [35]

    Safe-gil: Safety guided imitation learning.arXiv preprint arXiv:2404.05249, 2024

    Yusuf Umut Ciftci, Zeyuan Feng, and Somil Bansal. Safe-gil: Safety guided imitation learning.arXiv preprint arXiv:2404.05249, 2024

    2024 arXiv

  28. [36]

    Model-based runtime monitoring with inter- active imitation learning

    Huihan Liu, Shivin Dass, Roberto Mart ´ın-Mart´ın, and Yuke Zhu. Model-based runtime monitoring with inter- active imitation learning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4154–4161. IEEE, 2024

    2024

  29. [37]

    Uncertainty-aware fail- ure detection for imitation learning robot policies

    Chen Xu, Tony Khuong Nguyen, Patrick Miller, Robert Lee, Paarth Shah, Rares Andrei Ambrus, Haruki Nishimura, and Masha Itkina. Uncertainty-aware fail- ure detection for imitation learning robot policies. In CoRL Workshop on Safe and Robust Robot Learning for Operation in the R...

  30. [38]

    Asking for help: Failure prediction in behavioral cloning through value approximation

    Cem Gokmen, Daniel Ho, and Mohi Khansari. Asking for help: Failure prediction in behavioral cloning through value approximation. In2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 5821–

  31. [39]

    Back to the manifold: Recovering from out-of-distribution states

    Alfredo Reichlin, Giovanni Luca Marchetti, Hang Yin, Ali Ghadirzadeh, and Danica Kragic. Back to the manifold: Recovering from out-of-distribution states. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022

    2022

  32. [40]

    Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems.IEEE Control Systems Magazine, 43(5):137–177, 2023

    Kim P Wabersich, Andrew J Taylor, Jason J Choi, Koushil Sreenath, Claire J Tomlin, Aaron D Ames, and Melanie N Zeilinger. Data-driven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems.IEEE Control Systems Maga...

    2023

  33. [41]

    The safety filter: A unified view of safety-critical control in autonomous systems.Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023

    Kai-Chieh Hsu, Haimin Hu, and Jaime F Fisac. The safety filter: A unified view of safety-critical control in autonomous systems.Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023

    2023

  34. [42]

    Enhancing safety in learning from demonstration algorithms via control barrier function shielding

    Yue Yang, Letian Chen, Zulfiqar Zaidi, Sanne van Wav- eren, Arjun Krishna, and Matthew Gombolay. Enhancing safety in learning from demonstration algorithms via control barrier function shielding. InProceedings of the 2024 ACM/IEEE International Conference on Human- Robot Inter...

    2024

  35. [43]

    Error-aware imitation learning from teleoperation data for mobile manipulation

    Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Man- dlekar, Li Fei-Fei, Silvio Savarese, and Roberto Mart ´ın- Mart´ın. Error-aware imitation learning from teleoperation data for mobile manipulation. InConference on Robot Learning, pages 1367–1378. PMLR, 2022

    2022

  36. [44]

    Safe reinforcement learning via shielding

    Mohammed Alshiekh, Roderick Bloem, R ¨udiger Ehlers, Bettina K¨onighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  37. [45]

    Learning to be safe: Deep rl with a safety critic.arXiv preprint arXiv:2010.14603, 2020

    Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with a safety critic.arXiv preprint arXiv:2010.14603, 2020

    2010 arXiv

  38. [46]

    Recovery rl: Safe reinforcement learning with learned recovery zones.IEEE Robotics and Automation Letters, 6(3):4915–4922, 2021

    Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones.IEEE Robotics and Automation Letters, 6(3):4...

    2021

  39. [47]

    Safe reinforcement learning for legged locomotion

    Tsung-Yen Yang, Tingnan Zhang, Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu. Safe reinforcement learning for legged locomotion. In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2454–2461. IEEE, 2022

    2022

  40. [48]

    Robot reinforcement learning on the constraint manifold

    Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. InConference on Robot Learning, pages 1357–

  41. [49]

    POLICEd RL: Learning Closed-Loop Robot Control Policies with Provable Satisfaction of Hard Constraints

    Jean-Baptiste Bouvier, Kartik Nagpal, and Negar Mehr. POLICEd RL: Learning Closed-Loop Robot Control Policies with Provable Satisfaction of Hard Constraints. InRobotics: Science and Systems. Robotics: Science and Systems Foundation, 2024

    2024

  42. [50]

    Safe exploration in continuous action spaces.ArXiv, abs/1801.08757, 2018

    Gal Dalal, Krishnamurthy Dj Dvijotham, Matej Ve- cer´ık, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces.ArXiv, abs/1801.08757, 2018

    2018 arXiv

  43. [51]

    Reachability constrained reinforcement learning

    Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International conference on machine learning, pages 25636–25655. PMLR, 2022

    2022

  44. [52]

    Auditing robot learning for safety and compliance during deployment

    Homanga Bharadhwaj. Auditing robot learning for safety and compliance during deployment. InConference on Robot Learning, pages 1801–1806. PMLR, 2022

    2022

  45. [53]

    Robot execution failure prediction using incomplete data

    Bhekisipho Twala. Robot execution failure prediction using incomplete data. In2009 IEEE International Con- ference on Robotics and Biomimetics (ROBIO), pages 1518–1523. IEEE, 2009

    2009

  46. [54]

    Robot failure mode prediction with explainable machine learning

    Aneseh Alvanpour, Sumit Kumar Das, Christopher Kevin Robinson, Olfa Nasraoui, and Dan Popa. Robot failure mode prediction with explainable machine learning. In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE), pages 61–66. IEEE, 2020

    2020

  47. [55]

    A causal-based approach to explain, predict and prevent failures in robotic tasks.Robotics and Autonomous Systems, 162:104376, 2023

    Maximilian Diehl and Karinne Ramirez-Amaro. A causal-based approach to explain, predict and prevent failures in robotic tasks.Robotics and Autonomous Systems, 162:104376, 2023. Publisher: Elsevier

    2023

  48. [56]

    Reachability-based safe learning with gaussian processes

    Anayo K Akametalu, Jaime F Fisac, Jeremy H Gillula, Shahab Kaynama, Melanie N Zeilinger, and Claire J Tomlin. Reachability-based safe learning with gaussian processes. In53rd IEEE conference on decision and control, pages 1424–1431. IEEE, 2014

    2014

  49. [57]

    A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games.IEEE Transactions on automatic control, 50(7):947–957, 2005

    Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games.IEEE Transactions on automatic control, 50(7):947–957, 2005

    2005

  50. [58]

    Control barrier function based quadratic programs for safety critical systems.IEEE Transactions on Automatic Control, 62(8):3861–3876, 2016

    Aaron D Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems.IEEE Transactions on Automatic Control, 62(8):3861–3876, 2016

    2016

  51. [59]

    Failure prediction with statistical guarantees for vision-based robot control

    Alec Farid, David Snyder, Allen Z Ren, and Anirudha Majumdar. Failure prediction with statistical guarantees for vision-based robot control. InRobotics: Science and Systems (RSS), 2022

    2022

  52. [60]

    Sample-efficient safety assurances us- ing conformal prediction.The International Journal of Robotics Research, 43(9):1409–1424, 2024

    Rachel Luo, Shengjia Zhao, Jonathan Kuck, Boris Ivanovic, Silvio Savarese, Edward Schmerling, and Marco Pavone. Sample-efficient safety assurances us- ing conformal prediction.The International Journal of Robotics Research, 43(9):1409–1424, 2024

    2024

  53. [61]

    Motion planning.IEEE Robotics & Automation Magazine, 18(2):108–118, 2011

    Steven M La Valle. Motion planning.IEEE Robotics & Automation Magazine, 18(2):108–118, 2011

    2011

  54. [62]

    Moveit![ros topics].IEEE robotics & automation magazine, 19(1):18–19, 2012

    Sachin Chitta, Ioan Sucan, and Steve Cousins. Moveit![ros topics].IEEE robotics & automation magazine, 19(1):18–19, 2012

    2012

  55. [63]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  56. [64]

    Grasp pose detection in point clouds.The International Journal of Robotics Research, 36(13-14): 1455–1473, 2017

    Andreas Ten Pas, Marcus Gualtieri, Kate Saenko, and Robert Platt. Grasp pose detection in point clouds.The International Journal of Robotics Research, 36(13-14): 1455–1473, 2017

    2017

  57. [65]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gok- men, Sanjana Srivastava, Roberto Mart ´ın-Mart´ın, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conferenc...

    2023

  58. [66]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. InProceedings of the IEEE conference on computer vision and pattern recog- nition, pages 652–660, 2017

    2017

  59. [67]

    Sentence-bert: Sen- tence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sen- tence embeddings using siamese bert-networks. InPro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Com- putational Linguistics, 11 2019

    2019

  60. [68]

    Trac-ik: An open- source library for improved solving of generic inverse kinematics

    Patrick Beeson and Barrett Ames. Trac-ik: An open- source library for improved solving of generic inverse kinematics. In2015 IEEE-RAS 15th International Con- ference on Humanoid Robots (Humanoids), pages 928–

  61. [935]

    navigate to shelf

    IEEE, 2015. APPENDIX A. Model Details The architecture of the SQFs is a PointNet++ [66] encoder for processing point cloud input, 4-layer MLP encoder for the action and 4-layer MLP encoder for the proprioception (end- effector pose and the FT value). We categorize the continuo...

    2015

  62. [2020]

    MIT Press Journals, 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.