TWIST: Teleoperated Whole-Body Imitation System
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:NKZNQTGUrecord.jsonopen to challenge →
read the original abstract
Teleoperating humanoid robots in a whole-body manner marks a fundamental step toward developing general-purpose robotic intelligence, with human motion providing an ideal interface for controlling all degrees of freedom. Yet, most current humanoid teleoperation systems fall short of enabling coordinated whole-body behavior, typically limiting themselves to isolated locomotion or manipulation tasks. We present the Teleoperated Whole-Body Imitation System (TWIST), a system for humanoid teleoperation through whole-body motion imitation. We first generate reference motion clips by retargeting human motion capture data to the humanoid robot. We then develop a robust, adaptive, and responsive whole-body controller using a combination of reinforcement learning and behavior cloning (RL+BC). Through systematic analysis, we demonstrate how incorporating privileged future motion frames and real-world motion capture (MoCap) data improves tracking accuracy. TWIST enables real-world humanoid robots to achieve unprecedented, versatile, and coordinated whole-body motor skills--spanning whole-body manipulation, legged manipulation, locomotion, and expressive movement--using a single unified neural network controller. Our project website: https://humanoid-teleop.github.io
This paper has not been read by Pith yet.
Forward citations
Cited by 30 Pith papers
-
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
MPC-Injection biases off-policy RL locomotion policies toward controller-induced behavior basins by injecting MPC transitions into the replay buffer.
-
HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning
HumanoidArena is a new benchmark of 7 leg-critical HOI/HSI tasks that evaluates egocentric hierarchical whole-body policies in humanoids and finds performance is strongly conditioned on the low-level GMT used.
-
BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion
BeyondMimic combines compact motion tracking with a unified guided latent diffusion model to master diverse agile behaviors from human demos and solve unseen downstream tasks via test-time classifier guidance.
-
ThorArena: Benchmarking Humanoid Physical Interaction with Human Motion-Force Demonstrations
A force-aware humanoid benchmark pairs synchronized human motion-force data with simulation-based force replay to evaluate whole-body control policies under realistic physical disturbances.
-
CWI: Composite Humanoid Whole-Body Imitation System for Loco-manipulation
CWI decouples MoCap data for upper-body manipulation and lower-body locomotion, using dual discriminators and multi-critic training plus distillation to produce a policy that works from hand poses and velocity commands alone.
-
SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction
SceneBot conditions a humanoid tracking policy on motion references and contact labels, using reconstructed scene-interaction data to unify free-space locomotion with contact-rich manipulation and terrain tasks.
-
OmniContact: Chaining Meta-Skills via Contact Flow for Generalizable Humanoid Loco-Manipulation
OmniContact introduces contact flow as a shared representation of body trajectories and contact signals to learn and chain loco-manipulation meta-skills, reporting 98.7% success on box carrying and 76.5% on push-stack tasks.
-
CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation
CoorDex distills privileged body and hand motion teachers into proprioceptive latent priors and composes them via shared-context residual RL heads to enable continuous high-DoF dexterous loco-manipulation.
-
OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation
OpenHLM is an empirical recipe yielding a whole-body humanoid VLA model that outperforms GR00T N1.6 and Ψ0 baselines on long-horizon tasks using less than half the demonstration time.
-
Stubborn: A Streamlined and Unified Reinforcement Learning Framework for Robust Motion Tracking and Fall Recovery for Humanoids
Stubborn introduces a unified RL framework with yaw-aligned representation, Bernoulli probabilistic termination, and adaptive sampling for robust humanoid motion tracking and fall recovery.
-
X-OP: Cross-Morphology Whole-Body Teleoperation via MPC Retargeting
MPC-based retargeting framework enables cross-morphology whole-body teleoperation from a single XR device via dynamic feasibility optimization, state synchronization, and SLAM feedback, with reported gains in simulati...
-
LIMMT: Less is More for Motion Tracking
A data-centric approach shows that less than 3% of AMASS motion data, filtered by physics feasibility, diversity, and complexity, yields better humanoid tracking policies than the full dataset.
-
LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World
LEGS shows synthetic data from a 3DGS-mesh hybrid simulator trains VLA policies for humanoid pick-and-place that match or exceed human teleoperation performance across multiple backbones and tasks while enabling low-c...
-
Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking
Any2Any transfers pretrained humanoid whole-body tracking policies to new embodiments with 1% of original training cost via kinematic alignment and parameter-efficient fine-tuning.
-
SixthSense: Task-Agnostic Proprioception-Only Whole-Body Wrench Estimation for Humanoids
SixthSense infers whole-body contact events and wrenches in humanoids from proprioception and IMU data alone by tokenizing histories and estimating a sparse contact-event flow with conditional flow matching.
-
HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model
HAIC enables robust humanoid interactions with underactuated objects by predicting their dynamics from proprioceptive history and using a world model for adaptive control.
-
HUSKY: Humanoid Skateboarding System via Physics-Aware Whole-Body Control
HUSKY combines humanoid-skateboard dynamics modeling with adversarial motion priors and physics-guided lean-to-steer strategies to achieve real-world stable skateboarding on a humanoid robot.
-
Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary
Humanoid-LLA converts unconstrained natural language commands into stable whole-body motions for humanoid robots using a unified motion vocabulary and two-stage supervised-plus-reinforcement fine-tuning.
-
SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control
Scaling motion tracking models along size, data volume, and compute produces a foundation model for natural, robust humanoid whole-body control with downstream uses in kinematic planning and vision-language-action models.
-
HEFT: Heavy-Payload Full-size Humanoid Teleoperation with Privileged Motion Guidance and Windowed Payload Curriculum
HEFT enables tracking of human motions including locomotion and squats on a 175cm 65kg humanoid under up to 24kg payloads by combining Privileged Motion Guidance from noisy VR data with a Windowed Payload Curriculum.
-
Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration
CHORD uses object-centric contact wrench guidance to improve RL scalability for long-horizon dexterous manipulation, reporting 82.12% average success on 1,831 of 4,739 benchmark tasks with real-world transfer.
-
HALOMI: Learning Humanoid Loco-Manipulation with Active Perception from Human Demonstrations
HALOMI extends UMI with egocentric sensing and a manifold-constrained controller plus alignment adaptations to learn loco-manipulation on humanoids from human demos, reporting 85% average success on three real-world tasks.
-
OMG: Omni-Modal Motion Generation for Generalist Humanoid Control
OMG is a diffusion model for omni-modal whole-body humanoid motion generation that uses language, audio, and reference motions after large-scale data curation to achieve state-of-the-art performance and adaptation.
-
OASIS: From Simulation Data Collection to Real-World Humanoid Loco-Manipulation
OASIS generates scalable simulation data for humanoid loco-manipulation via 3D generative asset reconstruction and domain randomization, yielding a policy with higher zero-shot real-world success than real-robot teleo...
-
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Humanoid-GPT is a causal Transformer pre-trained on a unified billion-scale motion dataset that tracks dynamic behaviors with zero-shot generalization to unseen motions and tasks.
-
Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking
Any2Any transfers humanoid whole-body tracking models across embodiments via kinematic alignment followed by targeted PEFT, matching full-training performance with 1% of the data and compute on tested platforms.
-
Switch: Learning Agile Skills Switching for Humanoid Robots
Switch enables humanoid robots to perform agile, seamless transitions between locomotion skills via a kinematic skill graph, DRL tracking policy, and real-time graph-search scheduler.
-
Learning Versatile Humanoid Manipulation with Touch Dreaming
HTD, a multimodal transformer policy trained with behavioral cloning and touch dreaming to predict future tactile latents, achieves a 90.9% relative success rate improvement over baselines on five real-world contact-r...
-
Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies
Describes an integrated pipeline for curating motion data, adapting real-to-sim models, applying AMP-based RL, and deploying locomotion policies on Booster T1 and K1 humanoid robots.
-
Natural Human Motion Recovery by Aligning High-Order Temporal Dynamics from Monocular Videos
HTD-Refine uses a temporal transformer (PVA-Net) to predict high-order dynamics and refines HMR outputs via optimization for more natural motion.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.