An autoregressive diffusion model with a hybrid explicit-root/latent-body representation generates real-time, controllable 3D human motion from text and spatial constraints.
Available: https://arxiv.org/abs/2310.04582
11 Pith papers cite this work. Polarity classification is still indexing.
11
Pith papers citing it
abstract
We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers.
representative citing papers
Wrist-guided whole-body RL retargets human–object interactions without finger pose supervision, matching supervised methods and generalizing across hand morphologies in simulation.
A data-centric approach shows that less than 3% of AMASS motion data, filtered by physics feasibility, diversity, and complexity, yields better humanoid tracking policies than the full dataset.
A multi-condition latent diffusion model transfers human motion styles to diverse humanoid robot contents with physics regularizations, achieving 96% success in real-robot trials on Unitree G1.
A weightlessness mechanism enables humanoid robots to dynamically relax joints for stable, contact-rich motions across diverse environments without task-specific tuning.
A flow-matching model derives manipulation strategies from object affordance, adds an adversarial interaction prior, and uses stability simulation to generate natural, effective human-human co-manipulation motions.
SynAgent enables generalizable cooperative humanoid manipulation by transferring skills from solo human-object interactions to multi-agent scenarios via interaction-preserving retargeting, single-agent pretraining with multi-agent PPO, and a conditional VAE generative policy.
A progressive neural network automatically specializes experts on varying difficulty levels to enable stable physics-based tracking of contact-rich character interactions.
Fatigue-PINN applies physics-informed neural networks to simulate fatigue effects on human motion using a three-compartment muscle model for joint torque modulation in motion synthesis.
Humanoid-GPT is a causal Transformer pre-trained on a unified billion-scale motion dataset that tracks dynamic behaviors with zero-shot generalization to unseen motions and tasks.
Reinforcement learning with a constellation-based reward enables direct, efficient humanoid locomotion to short-range SE(2) targets, outperforming velocity-tracking baselines in simulation and transferring to hardware.
citing papers explorer
-
ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation
An autoregressive diffusion model with a hybrid explicit-root/latent-body representation generates real-time, controllable 3D human motion from text and spatial constraints.
-
WristMimic: Full-Body Humanoid Control with Wrist-Guided Manipulation
Wrist-guided whole-body RL retargets human–object interactions without finger pose supervision, matching supervised methods and generalizing across hand morphologies in simulation.
-
LIMMT: Less is More for Motion Tracking
A data-centric approach shows that less than 3% of AMASS motion data, filtered by physics feasibility, diversity, and complexity, yields better humanoid tracking policies than the full dataset.
-
Bionic Human-Motion Style Transfer for Physically Executable Whole-Body Control of Humanoid Robots
A multi-condition latent diffusion model transfers human motion styles to diverse humanoid robot contents with physics regularizations, achieving 96% success in real-robot trials on Unitree G1.
-
Learn Weightlessness: Imitate Non-Self-Stabilizing Motions on Humanoid Robot
A weightlessness mechanism enables humanoid robots to dynamically relax joints for stable, contact-rich motions across diverse environments without task-specific tuning.
-
Stability-Driven Motion Generation for Object-Guided Human-Human Co-Manipulation
A flow-matching model derives manipulation strategies from object affordance, adds an adversarial interaction prior, and uses stability simulation to generate natural, effective human-human co-manipulation motions.
-
SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
SynAgent enables generalizable cooperative humanoid manipulation by transferring skills from solo human-object interactions to multi-agent scenarios via interaction-preserving retargeting, single-agent pretraining with multi-agent PPO, and a conditional VAE generative policy.
-
Physics-Based Motion Tracking of Contact-Rich Interacting Characters
A progressive neural network automatically specializes experts on varying difficulty levels to enable stable physics-based tracking of contact-rich character interactions.
-
Fatigue-PINN: Physics-Informed Fatigue-Driven Motion Modulation and Synthesis
Fatigue-PINN applies physics-informed neural networks to simulate fatigue effects on human motion using a three-compartment muscle model for joint torque modulation in motion synthesis.
-
Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking
Humanoid-GPT is a causal Transformer pre-trained on a unified billion-scale motion dataset that tracks dynamic behaviors with zero-shot generalization to unseen motions and tasks.
-
No More Marching: Learning Humanoid Locomotion for Short-Range SE(2) Targets
Reinforcement learning with a constellation-based reward enables direct, efficient humanoid locomotion to short-range SE(2) targets, outperforming velocity-tracking baselines in simulation and transferring to hardware.