CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:MYFHID2Srecord.jsonopen to challenge →
read the original abstract
Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/
This paper has not been read by Pith yet.
Forward citations
Cited by 13 Pith papers
-
Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics
A framework called Policy-as-Data generates task-oriented synthetic HOI data via RL policies in physics simulators, retargets it, and trains diffusion models that generalize to unseen objects and long horizons.
-
ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
ViBES introduces a speech-language-behavior model using modality-specific transformer experts that jointly generates dialogue and 3D body actions, showing gains over separate co-speech and text-to-motion baselines on ...
-
WristMimic: Full-Body Humanoid Control with Wrist-Guided Manipulation
WristMimic achieves comparable or superior object manipulation retargeting by supervising wrist kinematics while letting finger behavior emerge from object and contact dynamics.
-
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
ICMPG combines LLM-based candidate generation with MPC-style physical simulation and semantic scoring to produce text-driven human motions that are both plausible and faithful.
-
MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Control
MIND introduces a multi-scale intent diffusion framework that bridges text commands and low-level actions via holistic and immediate intent predictors for physics-based humanoid control.
-
SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control
A new diffusion transformer policy with joint attention over actions, states, and text plus RL post-training outperforms prior methods on language alignment and motion quality for humanoid control.
-
SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control
SCRIPT presents a scalable diffusion policy with JAST-DiT architecture, nonlinear history conditioning, and RLHR post-training that claims to outperform prior methods on text alignment, motion quality, and physical re...
-
Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos
HA-HOI produces physically plausible 4D HOI animations from monocular videos by anchoring object reconstruction to human motion and refining the result in a physics-based humanoid-object simulator.
-
IAM: Identity-Aware Human Motion and Shape Joint Generation
IAM jointly synthesizes motion sequences and body shape parameters conditioned on multimodal identity signals to achieve more realistic and identity-consistent human motions.
-
Stability-Driven Motion Generation for Object-Guided Human-Human Co-Manipulation
A flow-matching model derives manipulation strategies from object affordance, adds an adversarial interaction prior, and uses stability simulation to generate natural, effective human-human co-manipulation motions.
-
DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion
DreamPolicy integrates an autoregressive diffusion world model with policy learning to produce a single scalable policy that generalizes to unseen composite terrains for humanoid locomotion.
-
GPC: Large-Scale Generative Pretraining for Transferable Motor Control
GPC learns a motion vocabulary via Finite Scalar Quantization and end-to-end RL, then trains an autoregressive transformer for next-token control generation, achieving 99.98% motion reproduction success with emergent ...
-
UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars
UMo presents a sparse MoE-based unified model for real-time co-speech avatar animation that claims superior quality under latency constraints via keyframe-centric design and multi-stage audio-augmented training.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.