REVIEW 12 cited by
ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this work, we aim to learn a unified vision-based policy for multi-fingered robot hands to manipulate a variety of objects in diverse poses. Though prior work has shown benefits of using human videos for policy learning, performance gains have been limited by the noise in estimated trajectories. Moreover, reliance on privileged object information such as ground-truth object states further limits the applicability in realistic scenarios. To address these limitations, we propose a new framework ViViDex to improve vision-based policy learning from human videos. It first uses reinforcement learning with trajectory guided rewards to train state-based policies for each video, obtaining both visually natural and physically plausible trajectories from the video. We then rollout successful episodes from state-based policies and train a unified visual policy without using any privileged information. We propose coordinate transformation to further enhance the visual point cloud representation, and compare behavior cloning and diffusion policy for the visual policy training. Experiments both in simulation and on the real robot demonstrate that ViViDex outperforms state-of-the-art approaches on three dexterous manipulation tasks.
Forward citations
Cited by 12 Pith papers
-
C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video
C2Dex converts monocular human videos into executable dexterous robot manipulation trajectories by using stable object-side contacts as a shared representation for reconstruction and retargeting, achieving 57.78% and ...
-
Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy
A two-stage optimization pipeline produces 9.5 million validated grasps across 31 GRASP taxonomy types and enables a type-conditional generative model to grasp novel objects from single-view point clouds with 82.3% re...
-
OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies
A policy-agnostic two-stage real-world RL method learns tactile residual corrections on frozen visual policies, lifting contact-rich task success from 5–40% to 85–100% in under 80 minutes.
-
Scaling Cross-Embodiment World Models for Dexterous Manipulation
A single particle-based world model trained on many simulated robot hands and real human hands can plan dexterous manipulation on robot hands it never trained on.
-
Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos
A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.
-
Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
A two-stage pipeline trains a robot policy that accepts a human demonstration video as a prompt and generalizes beyond its robot training tasks, with success rates of up to 79 percent on known task variations and unde...
-
Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
PDC trains a single egocentric-vision policy that lets a simulated humanoid search for, grasp, and place objects and open drawers without privileged state information.
-
DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References
A neural controller combining RL and imitation learning on iteratively mined demonstrations tracks human kinematic references for dexterous manipulation, yielding over 10% higher success rates than prior baselines.
-
Reinforcement Learning from Wild Animal Videos
A quadruped robot acquires walking, jumping, running-like, and standing skills using only the output of a video classifier trained on wild-animal videos as its reinforcement learning reward.
-
DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation
A dual-phase diffusion planner with dynamics-consistency and LLM-written guidance achieves strong success on goal-adaptive dexterous manipulation in simulation.
-
Imitation Learning for Adaptive Control of a Virtual Soft Exoglove
A virtual tendon-driven exoglove controller, trained with imitation learning and reinforcement learning, restores about 90.5% of a simulated weakened hand's original manipulation performance on three object-grasping tasks.
-
DexFlow: A Unified Approach for Dexterous Hand Pose Retargeting and Interaction
DexFlow retargets human MANO hand motions to ShadowHand and Allegro robots with contact-aware, temporally smoothed finger optimization and contributes a 292K-frame simulated grasp dataset.
Discussion (0). Continue with ORCID to comment.