Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Dexplore learns dexterous manipulation from MoCap in a single reinforcement-learning loop, using demonstrations as soft guidance inside adaptive spatial scopes, and reports sharply higher success than staged retarget-then-track baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:44 UTC pith:JSQ5XRN5

load-bearing objection Promising unified MoCap-to-policy idea, but the adaptive curriculum equation is reversed as written—a load-bearing flaw that undermines the paper's central novelty. the 4 major comments →

arxiv 2509.09671 v1 pith:JSQ5XRN5 submitted 2025-09-11 cs.RO cs.CV

Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration

Sirui Xu , Yu-Wei Chao , Liuyu Bian , Arsalan Mousavian , Yu-Xiong Wang , Liang-Yan Gui , Wei Yang This is my paper
classification cs.RO cs.CV
keywords dexterous manipulationmotion-capture demonstrationsreinforcement learningretargetingreference-scoped explorationvision-based generative controlsim-to-real transfersoft guidance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that human motion-capture demonstrations can train a dexterous robot hand without the standard three-stage pipeline of retargeting, tracking, and residual correction. Instead, Dexplore runs one reinforcement-learning loop that keeps the robot inside adaptive spatial 'scopes' around the demonstration while letting it discover its own embodiment-specific motions. Treating demonstrations as soft guidance rather than ground truth makes the method robust to noise and to the kinematic gap between human and robot hands. On two simulated hands it reports substantially higher success rates than a staged baseline, and the learned policy distills into a vision-based controller that runs on a real robot with only depth and proprioception.

Core claim

The central claim is that retargeting and tracking need not be separate stages: a unified RL objective with reference-scoped termination envelopes can both preserve the intent of a human demonstration and allow the robot to find physically feasible strategies that deviate from the reference. The scope is tightened adaptively based on failure rates, and the matching reward is downweighted when the hand is far from the object, letting the robot explore while still being drawn toward the demonstration. This yields, in simulation, 87.7% success on the Inspire hand versus 7.4% for a three-stage baseline, and 78.7% versus 45.9% on the Allegro hand. The same policy can then be distilled into a visi

What carries the argument

Reference-Scoped Exploration (RSE): instead of rigidly tracking a retargeted trajectory, the policy is trained within adaptive spatial scopes around the reference, with early-termination thresholds per criterion tightened as κ = κ_init · (N_fail / N_total), and with kinematic matching weights scaled by hand–object surface distance so that exploration is freer when the demonstration is out of reach. A secondary mechanism is the vision-based generative controller, which encodes manipulation skills in a latent space and matches privileged and partial-state encodings to remove the need for motion-capture references at test time.

Load-bearing premise

The load-bearing premise is that the hand–object surface distance and the coarse key-joint mapping (five fingertips for the Inspire hand) encode enough of the demonstrator's intent that staying inside the adaptive scopes guarantees a meaningful grasp; if that mapping misses an affordance such as wrist orientation or thumb opposition, the scopes can reward divergence without preserving intent.

What would settle it

Retrain Dexplore with the key-joint mapping M replaced by a random correspondence, or test on a task where wrist orientation is decisive (e.g., grasping a knife): if success remains high even when the scopes are satisfied but the functional grasp fails, the intent-preservation claim collapses; if success drops sharply, M is load-bearing. A concrete check is to compare success rates on precision grasps of thin objects while tracking fingertip error—if fingertip error is low yet the object still drops, the scopes are not preserving the relevant intent.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The three-stage transfer pipeline—retargeting, tracking, residual correction—can collapse into one optimization, so errors no longer compound across stages.
  • Because demonstrations are soft references, the policy can exploit robot-specific strategies (such as moving the slower thumb first) that strict retargeting would forbid.
  • Noisy or imperfect MoCap data becomes a usable training signal rather than a liability, since the scopes absorb demonstration inaccuracy.
  • The distilled vision-based controller requires only single-view depth and proprioception at runtime, removing the need for mocap references, pose estimators, or force sensors during deployment.
  • The same pipeline transfers across different robot hands with different degrees of freedom and size mismatches, suggesting it generalizes across embodiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the reported gains rest on a particular GRAB subset and simulation setup; the open question is whether the scopes preserve intent for affordances not visible in fingertip deltas, such as wrist orientation during a knife grasp—a policy that satisfies the scope could still miss the functional grasp.
  • Inference: the method motivates scaling to larger MoCap corpora, but the current training set is modest; a natural extension is to apply RSE to large recent datasets and test whether success increases monotonically with data volume.
  • Inference: the adaptive tightening schedule (κ proportional to failure ratio) acts as a curriculum; a testable extension would replace the hand-crafted schedule with an automatically tuned budget or measure sensitivity to the initial κ values.
  • Inference: because RSE abandons strict tracking, it could plausibly be combined with goal-conditioned reward shaping targeting functional outcomes (e.g., the object reaching a goal pose) rather than reference proximity, yielding a more task-aware objective.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Dexplore proposes a two-stage pipeline for learning dexterous manipulation from human MoCap. In stage I, a state-based policy is trained with PPO to track human references without explicit retargeting, using a soft guidance reward and adaptive early-termination "reference scopes". In stage II, this tracker is distilled into a vision-based, skill-conditioned generative policy that latently encodes manipulation intent and runs from single-view depth and proprioception. The paper reports strong simulation results on the GRAB dataset for the Inspire and Allegro hands (e.g., 87.7% vs. 7.4% success on Inspire against an adapted DexTrack baseline), ablates the RSE mechanism, and presents a real-robot deployment with an XArm-7 and Inspire hand.

Significance. The central idea is valuable: replacing the three-stage retargeting/tracking/residual-correction pipeline with a single RL loop that treats demonstrations as soft guidance could materially simplify learning from MoCap and better exploit imperfect human data. The paper is also careful to evaluate on external data (GRAB, TACO) and on two different hand embodiments, and it includes a real-world deployment with a depth-only policy. If the method is reproducible and the reported gains hold under multiple seeds, this would be a meaningful contribution to dexterous manipulation. The authors are candid about limitations (small/thin objects, single-hand tasks, modest dataset size). However, the central curriculum mechanism is mis-specified as written, and the quantitative evidence package needs strengthening.

major comments (4)
  1. [Sec. 3.1, Learning with Reference Scope] The adaptive-termination rule κ = κ_init · N_fail / N_total contradicts the described wide-to-tight curriculum. The termination conditions are of the form "terminate if R < κ" (e.g., R_h_J < κ_h_J), so a larger κ is a stricter, narrower scope and a smaller κ is a looser scope. As the policy improves, N_fail/N_total decreases, so κ decreases and the scope becomes wider, not tighter. Thus the algorithm as written starts with narrow envelopes and ends with wide ones, the reverse of the claimed exploration-to-precision schedule. This is the paper's core novelty, so the reported success rates are not explained by the stated update rule. Please correct the formula or the prose, and ideally release code or an explicit worked example of the schedule evolution.
  2. [Sec. 4, Table 1 and Baselines] The headline quantitative claims rest on evidence that is currently under-specified. (i) PPO is stochastic, yet no seeds, error bars, or run-to-run variability are reported for any entry in Table 1 or Table B. (ii) DexTrack is described as an "adapted" baseline because the official implementation is unavailable, but the adaptation is not described; the reader cannot tell whether the comparison is favorable to Dexplore or to DexTrack. Please report mean±std over at least 3 seeds, describe the DexTrack adaptation in enough detail to reproduce it, or make the evaluation code available.
  3. [Sec. 3.1, Reward weighting] The paper states that w(D)=min(1, D/0.20 m) is a deterministic function of the reference trajectory and therefore does not bias learning. However, D in the state definition is the current vector from hand joints to the nearest object surface (D_t), a state-dependent quantity. If w uses the current D, the reward weights are state-dependent and the policy can influence them; the stationarity argument in the text is then incorrect as written. If w instead uses a reference distance, the notation must distinguish D from D_hat and the claim should be re-evaluated. Please clarify and justify this reward-design choice.
  4. [Sec. 3.1, State / mapping M] The method depends on a fixed key-joint correspondence M adopted from AnyTeleop [33]. For the Inspire hand this reduces to the five fingertips. Since the paper argues that explicit retargeting is unnecessary, the sensitivity of the method to M should be tested: for example, compare the full M against alternative mappings (fingertips only, fingertips plus wrist, or a learned correspondence) and report success rates. Without such an ablation, the reader cannot tell whether Dexplore's gains come from the RSE curriculum or from the particular choice of correspondence.
minor comments (5)
  1. [General] The spelling of the method is inconsistent: "Dexplore" in the title/abstract and running text, "DEXPLORE" elsewhere. Please standardize.
  2. [Fig. 5 / Sec. C] "Fremto Bolt" appears to be a typo for "Femto Bolt".
  3. [Appendix C] The real-world deployment uses several system-level choices beyond the learned policy (RRT wrist motion, thumb-first actuation, masking/filtering via forward kinematics). The paper says no hand-crafted control is used in the core grasp, but the boundary between policy and system-level control should be drawn more explicitly so readers understand exactly what the learned policy contributes.
  4. [Table B / Sec. 4] Table B reports a single success-rate number for the vision policy; as noted above, no error bars or seeds are provided. Also, the cross-reference "Table B evaluates our vision-based policy" appears in a paragraph that is otherwise about Table 1; please clarify numbering.
  5. [Appendix B] The sentence about scaling object size by 1.5 and weight by 1.5^3 should be typeset correctly (e.g., 1.5^3 or 3.375×), and the table/figure references (Figure A, Figure B, Figure C, Figure D) should be made self-contained in the text.

Circularity Check

0 steps flagged

No significant circularity: Dexplore's derivation is self-contained and validated against external baselines; self-citations are adopted prior components, and the apparent RSE formula/prose inconsistency is a correctness/reproducibility issue rather than a circular reduction.

full rationale

I walked the claimed derivation chain: raw MoCap → mapping M → reference-scoped rewards → PPO → vision distillation. Each step is constructive. The mapping M is explicitly adopted from AnyTeleop [33] and the reward terms from InterMimic [51]; these are prior published system components with overlapping authors, but they are inputs, not conclusions, and neither is a uniqueness claim that forces Dexplore's output. No fitted parameter is renamed as a prediction: the adaptive threshold κ=κ_init·N_fail/N_total is a curriculum driven by rollout statistics, and the reported success rates are measured against baseline methods and real-world deployment, not algebraically implied by κ. I specifically checked the possible self-definitional risk that 'success' is just the training reward thresholded: the paper's success metric requires task completion without dropping the object, and the comparison against DexTrack/AnyTeleop provides external content, so this is not a by-construction equivalence. One flagged issue: the prose says training begins wide and tightens, while the formula κ=κ_init·N_fail/N_total would start strict and loosen as N_fail/N_total decreases; this is an internal inconsistency that affects reproducibility, but it is a correctness risk, not circularity. Hence score 1 for minor self-citation presence with no circular steps.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on hand-picked reward weights, unspecified termination thresholds, and a domain assumption that a coarse fingertip mapping conveys task intent. There are no invented physical entities; the latent skill code is a learned model component rather than a postulated entity.

free parameters (6)
  • Kinematic and dynamic reward weights (lambda_h_R, lambda_h_J, lambda_D, lambda_C)
    Hand-selected weighting hyperparameters for the matching rewards in Sec. 3.1; values are not reported, yet they determine the balance between tracking and energy cost.
  • Reference distance scale 0.20 m in w(D) = 0.20 m
    Chosen constant in w(D)=min(1,D/0.20m) that controls when kinematic matching weights saturate.
  • Termination threshold initial values kappa_init = not reported
    Adaptive schedule kappa=kappa_init*Nfail/Ntotal starts from unspecified large thresholds; per-criterion values are not given.
  • Goal horizon K (reference indices) = not reported
    K defines how many future reference frames are in the goal observation; no value is listed.
  • KL weight beta schedule in distillation = not reported
    Beta is gradually increased during generative-control training following beta-VAE practice; exact schedule omitted.
  • Domain randomization ranges = Table C ranges
    Object friction, restitution, density, shape, and point-cloud noise ranges are chosen by hand for sim-to-real transfer; not tied to a calibration procedure.
axioms (5)
  • domain assumption The MANO-to-robot correspondence map M from AnyTeleop [33] is sufficient to express task intent in state and reward differences.
    Invoked in Sec. 3.1 State; for the Inspire hand it reduces to five fingertips, which may not capture wrist and palm affordances.
  • ad hoc to paper Reward weights that vary with hand-object distance are stationary and do not bias learning because they are deterministic functions of the reference trajectory.
    Stated in Sec. 3.1 Learning with Reference Scope; this is a paper-specific argument and is not the usual MDP stationarity argument.
  • ad hoc to paper Adaptively tightening termination thresholds using the global failed-rollout ratio kappa=kappa_init*Nfail/Ntotal produces a useful curriculum.
    Introduced in Sec. 3.1; assumes failure ratios reflect per-frame difficulty and that thresholds should be shared across all tasks.
  • domain assumption Isaac Gym convex-decomposition physics with 20 convex hulls and domain randomization transfers to the real Inspire hand.
    Appendix B and Table C; sim-to-real gap is managed by randomization rather than measured systematically.
  • domain assumption Human MoCap reference trajectories encode task success even when the robot deviates spatially from them.
    Core soft-guidance premise of Sec. 3.1; if the reference is misaligned with object affordance, being in-scope might not imply task completion.

pith-pipeline@v1.3.0-alltime-deepseek · 13185 in / 11349 out tokens · 123198 ms · 2026-08-04T18:44:07.699476+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration." pith.science (2026). https://pith.science/paper/JSQ5XRN5

@misc{pith2026250909671,
  author       = {Pith},
  title        = {Pith review of: Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSQ5XRN5}},
  note         = {Machine review of arXiv:2509.09671}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Hand-object motion-capture (MoCap) repositories offer large-scale, contact-rich demonstrations and hold promise for scaling dexterous robotic manipulation. Yet demonstration inaccuracies and embodiment gaps between human and robot hands limit the straightforward use of these data. Existing methods adopt a three-stage workflow, including retargeting, tracking, and residual correction, which often leaves demonstrations underused and compound errors across stages. We introduce Dexplore, a unified single-loop optimization that jointly performs retargeting and tracking to learn robot control policies directly from MoCap at scale. Rather than treating demonstrations as ground truth, we use them as soft guidance. From raw trajectories, we derive adaptive spatial scopes, and train with reinforcement learning to keep the policy in-scope while minimizing control effort and accomplishing the task. This unified formulation preserves demonstration intent, enables robot-specific strategies to emerge, improves robustness to noise, and scales to large demonstration corpora. We distill the scaled tracking policy into a vision-based, skill-conditioned generative controller that encodes diverse manipulation skills in a rich latent representation, supporting generalization across objects and real-world deployment. Taken together, these contributions position Dexplore as a principled bridge that transforms imperfect demonstrations into effective training signals for dexterous manipulation.

Figures

Figures reproduced from arXiv: 2509.09671 by Arsalan Mousavian, Liang-Yan Gui, Liuyu Bian, Sirui Xu, Wei Yang, Yu-Wei Chao, Yu-Xiong Wang.

Figure 1
Figure 1. Figure 1: We propose DEXPLORE, a unified control policy that tracks diverse hand-object MoCap references on a dexterous robotic hand [1]. Left: By using human demonstrations as soft refer￾ences, we train a robot hand to discover motions that align with its physical form and a given task’s intent through large-scale reinforcement learning. Right: We transfer the learned policy into a skill￾conditioned generative cont… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DEXPLORE. (I) We first train a state-based policy that acquires scalable dexterous manipulation skills across diverse objects from raw MoCap data, without relying on ex￾ternal retargeting. Rather than rigidly imitating demonstration trajectories, the robot is encouraged to explore within reference-scoped regions, allowing embodiment-specific strategies to emerge. (II) We then distill these skil… view at source ↗
Figure 3
Figure 3. Figure 3: Left: Comparison with AnyTeleop [33] (top) and retargeted MoCap data (bottom), both without task awareness. For hands with limited DoFs, retargeting often yields unnatural poses, unlike our tracking results (middle). Right: Comparison with DexTrack [9] (top). For objects with small grasp regions, the baseline fails to grasp reliably, while our method (bottom) succeeds. efficient motion. The reward formulat… view at source ↗
Figure 4
Figure 4. Figure 4: Our imitation control framework is also suitable for Allegro hands [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The vision-based policy deployed on an XArm-7 robotic arm [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

    cs.RO 2026-06 unverdicted novelty 6.0

    DO AS I DO reconstructs and retargets hand-object interactions from in-the-wild monocular RGB videos to produce dexterous robot manipulation trajectories, outperforming prior methods on ground-truth and online video datasets.

  2. ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

    cs.RO 2026-06 unverdicted novelty 5.0

    ConTrack introduces a constrained RL method with online dual-variable adaptation and adaptive resets for improved long-horizon hand tracking in simulation and on real robots.

Reference graph

Works this paper leans on

69 extracted references · 16 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Smaller and higher-precision motion control experts.https:// inspire-robots.store/

    Inspire-robots. Smaller and higher-precision motion control experts.https:// inspire-robots.store/

  2. [2]

    Jiang, Y

    Z. Jiang, Y . Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. Fan, and Y . Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025

    2025

  3. [3]

    Jiang, Y

    H. Jiang, Y . Wang, H. Zhou, and D. Seita. Learning to Singulate Objects in Packed Environ- ments using a Dexterous Hand. InInternational Symposium on Robotics Research (ISRR), 2024

    2024

  4. [4]

    Bhirangi, A

    R. Bhirangi, A. DeFranco, J. Adkins, C. Majidi, A. Gupta, T. Hellebrekers, and V . Kumar. All the feels: A dexterous hand with large-area tactile sensing.IEEE Robotics and Automation Letters, 8(12):8311–8318, 2023

    2023

  5. [5]

    Billard and D

    A. Billard and D. Kragic. Trends and challenges in robot manipulation.Science, 364(6446): eaat8414, 2019

    2019

  6. [6]

    Kroemer, S

    O. Kroemer, S. Niekum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms.Journal of machine learning research, 22(30): 1–82, 2021

    2021

  7. [7]

    C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.arXiv preprint arXiv:2403.07788, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    K. Li, P. Li, T. Liu, Y . Li, and S. Huang. ManipTrans: Efficient dexterous bimanual manipula- tion transfer via residual learning.arXiv preprint arXiv:2503.21860, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    X. Liu, J. Adalibieke, Q. Han, Y . Qin, and L. Yi. DexTrack: Towards generalizable neural tracking control for dexterous manipulation from human references. InICLR, 2025

    2025

  10. [10]

    Mandlekar, Y

    A. Mandlekar, Y . Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. InCoRL, 2018. 9

    2018

  11. [11]

    Mandlekar, D

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation.arXiv preprint arXiv:2108.03298, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto. Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation. InICRA, 2023

    2023

  13. [13]

    Z. Q. Chen, K. Van Wyk, Y .-W. Chao, W. Yang, A. Mousavian, A. Gupta, and D. Fox. Dex- transfer: Real world multi-fingered dexterous grasping with minimal human demonstrations. arXiv preprint arXiv:2209.14284, 2022

    Pith/arXiv arXiv 2022

  14. [14]

    B. Zhou, H. Yuan, Y . Fu, and Z. Lu. Learning diverse bimanual dexterous manipulation skills from human demonstrations.arXiv preprint arXiv:2410.02477, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    S. Li, Z. Huang, T. Chen, T. Du, H. Su, J. B. Tenenbaum, and C. Gan. Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics. In ICLR, 2023

    2023

  16. [16]

    C. Wang, L. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y . Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. InCoRL, 2023

    2023

  17. [17]

    S. An, Z. Meng, C. Tang, Y . Zhou, T. Liu, F. Ding, S. Zhang, Y . Mu, R. Song, W. Zhang, et al. Dexterous manipulation through imitation learning: A survey.arXiv preprint arXiv:2504.03515, 2025

    arXiv 2025

  18. [18]

    J. Ye, K. Wang, C. Yuan, R. Yang, Y . Li, J. Zhu, Y . Qin, X. Zou, and X. Wang. Dex1b: Learning with 1b demonstrations for dexterous manipulation. InRSS, 2025

    2025

  19. [19]

    J. Ye, J. Wang, B. Huang, Y . Qin, and X. Wang. Learning continuous grasping function with a dexterous hand from human demonstrations.IEEE Robotics and Automation Letters, 8(5): 2882–2889, 2023

    2023

  20. [20]

    Y . Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation.IEEE Robotics and Automation Letters, 7(4): 10873–10881, 2022

    2022

  21. [21]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. InRSS, 2023

    2023

  22. [23]

    H. Luo, Y . Feng, W. Zhang, S. Zheng, Y . Wang, H. Yuan, J. Liu, C. Xu, Q. Jin, and Z. Lu. Being-h0: Vision-language-action pretraining from large-scale human videos.arXiv preprint arXiv:2507.15597, 2025

    Pith/arXiv arXiv 2025

  23. [24]

    R. Yang, Q. Yu, Y . Wu, R. Yan, B. Li, A.-C. Cheng, X. Zou, Y . Fang, H. Yin, S. Liu, et al. Egovla: Learning vision-language-action models from egocentric human videos.arXiv preprint arXiv:2507.12440, 2025

    Pith/arXiv arXiv 2025

  24. [25]

    Qin, Y .-H

    Y . Qin, Y .-H. Wu, S. Liu, H. Jiang, R. Yang, Y . Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. InECCV, 2022

    2022

  25. [26]

    Handa, K

    A. Handa, K. Van Wyk, W. Yang, J. Liang, Y .-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system. InICRA, 2020

    2020

  26. [27]

    S. Chen, C. Wang, K. Nguyen, L. Fei-Fei, and C. K. Liu. Arcap: Collecting high-quality human demonstrations for robot learning with augmented reality feedback.arXiv preprint arXiv:2410.08464, 2024. 10

    Pith/arXiv arXiv 2024

  27. [28]

    Cheng, J

    X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-television: Teleoperation with immer- sive active visual feedback. InCoRL, 2024

    2024

  28. [29]

    R. Ding, Y . Qin, J. Zhu, C. Jia, S. Yang, R. Yang, X. Qi, and X. Wang. Bunny- visionpro: Real-time bimanual dexterous teleoperation for imitation learning.arXiv preprint arXiv:2407.03162, 2024

    Pith/arXiv arXiv 2024

  29. [30]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRSS, 2023

    2023

  30. [31]

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators. InIROS, 2024

    2024

  31. [32]

    T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. InIROS, 2024

    2024

  32. [33]

    Y . Qin, W. Yang, B. Huang, K. Van Wyk, H. Su, X. Wang, Y .-W. Chao, and D. Fox. AnyTeleop: A general vision-based dexterous robot arm-hand teleoperation system. InRSS, 2023

    2023

  33. [34]

    Rajeswaran, V

    A. Rajeswaran, V . Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. InRSS, 2018

    2018

  34. [35]

    A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. InICRA, 2018

    2018

  35. [36]

    Vecerik, T

    M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Roth¨orl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.arXiv preprint arXiv:1707.08817, 2017

    Pith/arXiv arXiv 2017

  36. [37]

    Akkaya, M

    I. Akkaya, M. Andrychowicz, M. Chociej, and et al. Solving rubik’s cube with a robot hand. arXiv:1910.07113, 2019

    Pith/arXiv arXiv 1910

  37. [38]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. InNeurIPS, 2017

    2017

  38. [39]

    Tobin, R

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. InIROS, 2017

    2017

  39. [40]

    Dasari, A

    S. Dasari, A. Gupta, and V . Kumar. Learning dexterous manipulation from exemplar object trajectories and pre-grasps. InICRA, 2023

    2023

  40. [41]

    T. Wu, M. Wu, J. Zhang, Y . Gan, and H. Dong. Learning score-based grasping primitive for human-assisting dexterous grasping. InNeurIPS, 2023

    2023

  41. [42]

    H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V . Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. InICRA, 2019

    2019

  42. [43]

    Khandate, S

    G. Khandate, S. Shang, E. T. Chang, T. L. Saidi, Y . Liu, S. M. Dennis, J. Adams, and M. Ciocar- lie. Sampling-based exploration for reinforcement learning of dexterous manipulation.arXiv preprint arXiv:2303.03486, 2023

    Pith/arXiv arXiv 2023

  43. [44]

    T. G. W. Lum, M. Matak, V . Makoviychuk, A. Handa, A. Allshire, T. Hermans, N. D. Ratliff, and K. Van Wyk. Dextrah-g: Pixels-to-action dexterous arm-hand grasping with geometric fabrics.arXiv preprint arXiv:2407.02274, 2024

    Pith/arXiv arXiv 2024

  44. [45]

    Singh, A

    R. Singh, A. Allshire, A. Handa, N. Ratliff, and K. Van Wyk. Dextrah-rgb: Visuomotor policies to grasp anything with dexterous hands.arXiv preprint arXiv:2412.01791, 2024. 11

    Pith/arXiv arXiv 2024

  45. [46]

    W. Wan, H. Geng, Y . Liu, Z. Shan, Y . Yang, L. Yi, and H. Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist- specialist learning. InICCV, 2023

    2023

  46. [47]

    Y . Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y . Weng, J. Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. InCVPR, 2023

    2023

  47. [48]

    Zhang, Z

    H. Zhang, Z. Wu, L. Huang, S. Christen, and J. Song. Robustdexgrasp: Robust dexterous grasping of general objects.arXiv preprint arXiv:2504.05287, 2025

    Pith/arXiv arXiv 2025

  48. [49]

    Huang, H

    L. Huang, H. Zhang, Z. Wu, S. Christen, and J. Song. Fungrasp: functional grasping for diverse dexterous hands.IEEE Robotics and Automation Letters, 2025

    2025

  49. [50]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  50. [51]

    S. Xu, H. Y . Ling, Y .-X. Wang, and L.-Y . Gui. Intermimic: Towards universal whole-body control for physics-based human-object interactions. InCVPR, 2025

    2025

  51. [52]

    Christen, M

    S. Christen, M. Kocabas, E. Aksan, J. Hwangbo, J. Song, and O. Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. InCVPR, 2022

    2022

  52. [53]

    R. S. Dahiya, G. Metta, M. Valle, and G. Sandini. Tactile sensing—from humans to humanoids. IEEE transactions on robotics, 26(1):1–20, 2009

    2009

  53. [54]

    X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne. Deepmimic: Example-guided deep re- inforcement learning of physics-based character skills.ACM Transactions On Graphics (TOG), 37(4):1–14, 2018

    2018

  54. [55]

    Romero, D

    J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics, 36(6), 2017

    2017

  55. [56]

    Orbbec femto bolt high-performance depth camera.https://www.orbbec.com/ products/tof-camera/femto-bolt/

    ORBBEC. Orbbec femto bolt high-performance depth camera.https://www.orbbec.com/ products/tof-camera/femto-bolt/

  56. [57]

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. InNeurIPS, 2017

    2017

  57. [58]

    Tessler, Y

    C. Tessler, Y . Guo, O. Nabati, G. Chechik, and X. B. Peng. Maskedmimic: Unified physics- based character control through masked motion inpainting.ACM Transactions on Graphics (TOG), 43(6):1–21, 2024

    2024

  58. [59]

    Allegro hand.https://www.allegrohand.com

    Wonic-robotics. Allegro hand.https://www.allegrohand.com

  59. [60]

    Taheri, N

    O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. GRAB: A dataset of whole-body human grasping of objects. InECCV, 2020

    2020

  60. [61]

    H. Yao, Z. Song, B. Chen, and L. Liu. Controlvae: Model-based learning of generative con- trollers for physics-based characters.ACM Transactions on Graphics (TOG), 41(6):1–16, 2022

    2022

  61. [62]

    Higgins, L

    I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017

    2017

  62. [63]

    Makoviychuk, L

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. InNeurIPS, 2021. 12

    2021

  63. [64]

    Y . Liu, H. Yang, X. Si, L. Liu, Z. Li, Y . Zhang, Y . Liu, and L. Yi. Taco: Benchmarking generalizable bimanual tool-action-object understanding. InCVPR, 2024

    2024

  64. [65]

    Ufactory x-arm 7.https://www.ufactory.us/product/ ufactory-xarm-7

    UFACTORY . Ufactory x-arm 7.https://www.ufactory.us/product/ ufactory-xarm-7. 13 DEXPLORE: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration Appendix In this appendix, we provide additional experimental setups: 1.Demo Video.A demonstration video including the real world experiment is provided, as described in Sec. A. 2.S...

  65. [66]

    We demonstrate that our vision-based policy successfully transfers to real-world dexterous grasping tasks

  66. [67]

    Our state-based policy consistently outperforms baseline methods, such as DexTrack [9] and AnyTeleop [33], demonstrating superior task execution and generalization capabilities, such as to novel objects

  67. [68]

    Our holistic method integrates retargeting and skill learning into a unified, cohesive frame- work, enhancing overall adaptability and performance

  68. [69]

    Our framework exhibits significant versatility, successfully adapting to different robotic embodiments, including the Allegro [59] hand larger than hand hands, and Inspire [1] hand with unactuated joints

  69. [70]

    Our vision-based policy, which operates without privileged information, reliably accom- plishes tasks with high robustness, emphasizing the practicality and resilience of our ap- proach in real-world scenarios. B Simulation Setup We use MANO models [55] to represent human hand reference data, and objects are represented using convex decomposition into20co...