Pith

open record

sign in

arxiv: 2412.14172 · v1 · pith:D6A6RHHS · submitted 2024-12-18 · cs.RO · cs.AI· cs.CL· cs.CV

Learning from Massive Human Videos for Universal Humanoid Pose Control

Jiageng Mao , Siheng Zhao , Siqi Song , Tianheng Shi , Junjie Ye , Mingtong Zhang , Haoran Geng , Jitendra Malik
show 2 more authors
Vitor Guizilini Yue Wang
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:D6A6RHHSrecord.jsonopen to challenge →

classification cs.RO cs.AIcs.CLcs.CV
keywords humanoidcontrollearningrobotshumanoid-xmotionreal-worldcorresponding
0
0 comments X
read the original abstract

Scalable learning of humanoid robots is crucial for their deployment in real-world applications. While traditional approaches primarily rely on reinforcement learning or teleoperation to achieve whole-body control, they are often limited by the diversity of simulated environments and the high costs of demonstration collection. In contrast, human videos are ubiquitous and present an untapped source of semantic and motion information that could significantly enhance the generalization capabilities of humanoid robots. This paper introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot poses with corresponding text-based motion descriptions, designed to leverage this abundant data. Humanoid-X is curated through a comprehensive pipeline: data mining from the Internet, video caption generation, motion retargeting of humans to humanoid robots, and policy learning for real-world deployment. With Humanoid-X, we further train a large humanoid model, UH-1, which takes text instructions as input and outputs corresponding actions to control a humanoid robot. Extensive simulated and real-world experiments validate that our scalable training approach leads to superior generalization in text-based humanoid control, marking a significant step toward adaptable, real-world-ready humanoid robots.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ThorArena: Benchmarking Humanoid Physical Interaction with Human Motion-Force Demonstrations

    cs.RO 2026-07 conditional novelty 6.0

    A force-aware humanoid benchmark pairs synchronized human motion-force data with simulation-based force replay to evaluate whole-body control policies under realistic physical disturbances.

  2. LIMMT: Less is More for Motion Tracking

    cs.RO 2026-06 unverdicted novelty 6.0

    A data-centric approach shows that less than 3% of AMASS motion data, filtered by physics feasibility, diversity, and complexity, yields better humanoid tracking policies than the full dataset.

  3. An LLM-Driven Closed-Loop Autonomous Learning Framework for Robots Facing Uncovered Tasks in Open Environments

    cs.RO 2026-04 unverdicted novelty 6.0

    Robots autonomously convert LLM-guided experiences into a reusable local method library, reducing average execution time from 7.7772s to 6.7779s and LLM calls per task from 1.0 to 0.2 in repeated-task experiments.

  4. Re$^2$MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement

    cs.CV 2026-04 unverdicted novelty 6.0

    Re²MoGen generates open-vocabulary motions via MCTS-enhanced LLM keyframe planning, pose-prior optimization with dynamic temporal matching fine-tuning, and physics-aware RL post-training, claiming SOTA performance.

  5. Humanoid-GPT: Scaling Data and Structure for Zero-Shot Motion Tracking

    cs.RO 2026-06 unverdicted novelty 5.0

    Humanoid-GPT is a causal Transformer pre-trained on a unified billion-scale motion dataset that tracks dynamic behaviors with zero-shot generalization to unseen motions and tasks.