OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:QE6DM3NNrecord.jsonopen to challenge →
read the original abstract
We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
WristMimic: Full-Body Humanoid Control with Wrist-Guided Manipulation
WristMimic achieves comparable or superior object manipulation retargeting by supervising wrist kinematics while letting finger behavior emerge from object and contact dynamics.
-
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
WatchAct is a new benchmark of 3000 instances across 14 tasks in four cognitive domains for evaluating video-grounded robot manipulation, with current systems achieving at most 16.3% success.
-
Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
DO AS I DO reconstructs and retargets hand-object interactions from in-the-wild monocular RGB videos to produce dexterous robot manipulation trajectories, outperforming prior methods on ground-truth and online video datasets.
-
EgoExo-WM: Unlocking Exo Video for Ego World Models
Method converts exocentric videos to egocentric format via body-pose extraction and kinematics to improve egocentric world-model prediction and planning.
-
EgoExo-WM: Unlocking Exo Video for Ego World Models
Converting exocentric video to egocentric format via body-pose extraction and kinematics prior enables training of action-conditioned egocentric world models that improve prediction quality and goal-directed planning.
-
X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
X-Diffusion adapts Ambient Diffusion to selectively train on noised human actions for cross-embodiment robot policies, yielding 16% higher average success rates than naive co-training or manual filtering across five r...
-
Learning Versatile Humanoid Manipulation with Touch Dreaming
HTD, a multimodal transformer policy trained with behavioral cloning and touch dreaming to predict future tactile latents, achieves a 90.9% relative success rate improvement over baselines on five real-world contact-r...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.