Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CordViP reports 85-95% success on four real dexterous tasks and 80%/60% on two harder ones, beating the comparison baselines.

desk verdict A sensible, new integration of pose-based point clouds and contact/coordination pretraining, but the headline success-rate margins are within sampling noise at N=20 and need stronger evidence. read the letter →

arxiv 2502.08449 v2 pith:523F5QHI submitted 2025-02-12 cs.RO cs.AI

Yankai Fu , Qiuxuan Feng , Ning Chen , Zichen Zhou , Mengzhen Liu , Mingdong Wu , Tianxing Chen , Shanyu Rong
show 3 more authors
Jiaming Liu Hao Dong Shanghang Zhang
This is my paper · ORCID
classification cs.ROcs.AI
keywords correspondencelearningdexterousmanipulationimitationinteraction-awarepointclouds6Dposeestimationcontactmapsdiffusionpolicyvisuomotor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CordViP is a framework for learning dexterous hand-arm manipulation from demonstrations, built around a new observation representation called interaction-aware point clouds. Instead of feeding the policy raw camera point clouds, the system reconstructs a clean object point cloud from a single image, tracks it with a 6D pose estimator, and generates the hand point cloud from the robot's own joint angles through forward kinematics. The paper claims this keeps hand-object contact visible even when the camera view is occluded, and that pre-training the encoder on contact-map prediction and hand-arm coordination lets a downstream diffusion policy learn from fewer demonstrations. On six real-world tasks with a four-finger hand, CordViP reports 85-95% completion on the four base tasks and 80% and 60% on FlipCap and LongHoriManip, surpassing all comparison baselines tested by a sizable margin. The authors position the result as evidence that correspondence, not raw point-cloud quality, is what makes dexterous imitation learning work in real environments.

What carries the argument

The central object is the interaction-aware point cloud: a pair of aligned 1024-point clouds, one for the object and one for the hand, expressed in a common coordinate frame. The object cloud comes from a single-image 3D reconstruction tracked by a 6D pose estimator; the hand cloud comes from a forward-kinematics model that maps the robot's joint configuration to the surface points of the hand links. A cross-attention transformer aligns the two point-cloud feature sets, and a pre-training objective composed of contact-map prediction (normalized distances between object surface points and hand surface points) plus hand-arm action reconstruction forces the encoder to retain spatial contact and temporal coordination information. The diffusion policy conditions its denoising on these fused features, and the whole pipeline is designed so the observation stays complete even when the camera cannot see the contact region.

What would settle it

Reposition the camera in the FlipCup task so that the dexterous hand fully occludes the cup for several seconds during the wrist rotation, and compare success rates with the original camera placement; if the policy collapses, the result depends on continuous 6D pose rather than on the learned policy. A complementary test is to inject controlled rotation or translation noise into the pose estimate and record how quickly task success drops.

Watch

Extended reading notes

Core claim

The central claim is that a dexterous manipulation policy does not need the full noisy camera point cloud; it needs a point cloud that preserves the spatial correspondence between hand and object. CordViP constructs this by combining a digital twin of the object (generated from a single image), a 6D pose tracker that keeps the object's point cloud aligned in a world coordinate frame, and the hand's point cloud computed from joint states via a forward-kinematics model. The two point clouds are fused through cross-attention encoders, and the encoder is pre-trained to predict an object-centric contact map and to reconstruct arm actions from hand states and hand actions from arm states. That pre-training is what lets the downstream diffusion policy extract contact and coordination features from visual observations alone. The paper's experiments aim to show this representation is robust to viewpoint, lighting, unseen objects, and cluttered scenes, and that it yields higher success rates than the comparison baselines, including image-based policies, point-cloud variants of existing methods, and a state-conditioned diffusion policy.

Load-bearing premise

The load-bearing premise is that the object's 6D pose stays accurate through the whole episode; the paper's own Limitations section concedes that full hand occlusion can break the pose tracker and that deformable objects are out of scope.

Editorial extensions

If this is right

  • With only 10 expert demonstrations, CordViP sustains high success on PickPlace and FlipCup, indicating the correspondence-based observation reduces the data burden of imitation learning.
  • Removing the contact-map pre-training drops FlipCup success from 90% to 75%, so contact prediction is load-bearing for contact-rich tasks.
  • Removing the hand-arm coordination pre-training drops Assembly from 90% to 80% and ArtiManip from 95% to 85%, so coordination prediction matters most for continuous-motion tasks.
  • CordViP transfers to a transformer-based action backbone with comparable success, so the gains are not tied to the diffusion backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the hand point cloud is generated from joint angles rather than from camera pixels, the same representation should transfer to a different robot hand or arm without recollecting visual data, as long as the object pose tracker still works.
  • Because the paper's own limitations point to full hand occlusion as the failure mode, testing a visuotactile pose tracker on that exact scenario would directly probe whether 6D pose is the bottleneck.
  • An added diagnostic could correlate contact-map prediction error with downstream success rate across tasks; the paper's ablations hint contact matters most for contact-rich tasks but do not report that correlation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CordViP, a correspondence-based visuomotor policy for real-world dexterous manipulation. The method constructs interaction-aware point clouds by combining a TripoSR-generated digital twin, FoundationPose 6D pose tracking, and a forward-kinematics point cloud of the Leap Hand from robot proprioception. The observation encoder is pretrained with auxiliary tasks: predicting an object-centric contact map and reconstructing hand/arm coordination information. A diffusion policy is then trained on expert demonstrations conditioned on the pretrained features. The paper reports experiments on four base tasks (PickPlace, FlipCup, Assembly, ArtiManip) and two advanced tasks (FlipCap, LongHoriManip), claiming state-of-the-art performance over six or more baselines, along with efficiency, generalization, and ablation studies.

Significance. If the reported performance is robust, the paper makes a useful empirical contribution to dexterous manipulation: the interaction-aware point cloud construction is a concrete remedy for occlusion-degraded partial point clouds, and the contact/coordination pretraining is a sensible self-supervision scheme for a high-DoF hand-arm system. The paper is also commendable for testing in real-world tasks, for comparing against several strong baselines, for analyzing failure cases, and for candidly stating its main limitations. The central weakness is statistical: the headline claims rest on 20 trials per base task and 10 trials per advanced task with no error bars, confidence intervals, or multiple seeds, and the reported margins over the strongest baseline are within binomial sampling noise. In addition, the encoder ablation shows a striking dependence on PointNet, and an important recent baseline (RISE) is evaluated but excluded from the main comparison. These issues need to be resolved before the state-of-the-art claim can be accepted.

major comments (5)
  1. [§IV-B, Tables I and II] The abstract and Section V claim 'state-of-the-art performance' and 'surpassing other baselines by a large margin,' but the supporting numbers are single point estimates from 20 trials per base task and 10 trials per advanced task with no error bars, confidence intervals, or multiple seeds. Against the strongest baseline G3Flow, the gaps are 85% vs 65% (PickPlace), 90% vs 65% (FlipCup), 90% vs 80% (Assembly), and 95% vs 85% (ArtiManip); Fisher's exact tests yield p>0.05 in each case (e.g., PickPlace p≈0.27, Assembly p≈0.66). At N=10, FlipCap 80% vs 60% and LongHoriManip 60% vs 30% change by 10 percentage points with a single trial. The same fragility applies to the generalization and ablation tables, so the central empirical claim is not statistically supported. The authors should report more trials or confidence intervals, and should temper the 'large margin' claim accordingly.
  2. [§IV-E, Table VIII] The point cloud encoder ablation is a red flag for the robustness of the method: PointNet achieves an average of 90%, PointNet++ 26%, and PointNeXt 0% on all four tasks. A 0% result across every task suggests a configuration, tuning, or pipeline issue rather than a mere architectural preference, and it indicates that the reported success may depend heavily on an encoder the pipeline was implicitly tuned around. Since encoder choice is not part of the proposed contribution, the authors should report the training settings used for each encoder, verify that PointNeXt and PointNet++ are trained under comparable conditions, and discuss why one modern encoder completely fails. Otherwise the claim of robust and generalizable visuomotor learning is not supported.
  3. [Appendix C] RISE is a relevant recent 3D-based imitation learning baseline, and the paper states that it was evaluated in the authors' setting, but it is excluded from the main comparison tables and only described qualitatively as producing 'excessively abrupt movements.' This is problematic for the state-of-the-art claim: the reader cannot verify that CordViP outperforms RISE, nor can the reader assess the magnitude of the purported failure. The authors should either include quantitative RISE results in the main tables or clearly restrict the state-of-the-art claim to the baselines actually reported.
  4. [§III-C and §IV-A] The pretraining phase is described as using 'large-scale play data' to learn contact maps and hand-arm coordination, and this pretraining is a load-bearing contribution of the paper. However, the manuscript never specifies the size, collection protocol, diversity, or task coverage of the play data, nor how it relates to the 50 expert demonstrations per task described in §IV-A. Without these details the pretraining claim is not reproducible. The authors should give the play dataset statistics, collection procedure, and usage details.
  5. [§V, Limitations and §III-B] The interaction-aware point cloud pipeline depends on a digital twin and FoundationPose tracking, and the Limitations section honestly concedes that FoundationPose may fail if the dexterous hand completely occludes the object, that digital twin quality affects tracking, and that deformable objects cannot be handled. This is a genuine scope restriction for the paper's claim of addressing occlusion during dexterous manipulation. The authors should state more precisely under what occlusion conditions the current pipeline works, and ideally quantify the fraction of evaluation episodes in which tracking was maintained, so readers can judge the practical coverage of the method.
minor comments (5)
  1. [Eq. (3)] There is a typo: 'Sigmod' should be 'Sigmoid'.
  2. [§IV-C] The phrase 'To access the learning efficiency' should be 'To assess the learning efficiency,' and the legend in Figure 7 uses 'CoVP' while the paper uses 'CordViP'; please unify notation.
  3. [§IV-A vs Appendix Table X] The main text says episodes are limited to a maximum of 500 steps, but Table X lists Max Steps of 400, 300, 500, 600, 400, and 800 for the six tasks; this inconsistency should be corrected.
  4. [References] References [65] and [66] are the same paper and should be merged.
  5. [Appendix C] The sentence about DP3's 'axis-wise actions' is difficult to parse; please clarify whether the demonstration action representation differs from the natural action representation and whether this affects all compared methods equally.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CordViP's contributions are empirical and its success metrics are externally defined.

full rationale

CordViP is an empirical imitation-learning system, not a formal derivation, and its central claims are evaluated against externally defined task-completion criteria. The interaction-aware point clouds are generated from an object digital twin transformed by FoundationPose pose estimates and a forward-kinematics hand model (Eq. 1), which are external components rather than outputs of the policy being claimed. The contact-map and coordination pretraining losses (Eqs. 2-5) are auxiliary self-supervision computed from the same point-cloud observations that later condition the diffusion policy, but the paper does not present these losses as the predicted outcome of interest; task success is measured physically and is not defined in terms of the fitted contact map or coordination targets. Self-citations to prior work by the same group appear in related work and in components such as point-cloud forward kinematics or digital-twin pipelines, but no load-bearing claim is justified solely by a self-citation, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the design. Reported statistical fragility at N=20 or N=10 is a correctness and evidence-quality concern, not a circularity of definition or derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method rests on external learned components (TripoSR, FoundationPose, PointNet) and on hand-set scaling factors for the contact map and loss weighting. No new physical entities are introduced. The central empirical claim depends on the accuracy and availability of these components.

free parameters (5)
  • theta (contact map scaling) = 10
    Controls the sharpness of the object-centric contact map in Eq. 3; chosen empirically.
  • gamma (distance scaling) = 1
    Scaling factor in the aligned distance D(O,H), Eq. 2; set empirically.
  • lambda (loss weight) = 1
    Balances contact and coordination pretraining losses in Eq. 5.
  • point cloud size = 1024 per cloud
    Downsampled size of hand and object point clouds, set in practice.
  • crop range = x in [-0.4,0.1], y in [-0.7,-0.4], z in [0.1,0.51]
    Manual workspace cropping for point cloud observations.
assumptions (5)
  • domain assumption FoundationPose provides accurate 6D pose estimates and tracking for rigid objects throughout manipulation.
    The interaction-aware point cloud is generated by transforming the digital twin with this pose; Section III-B and Limitations.
  • domain assumption TripoSR reconstructs a digital twin from a single image that is geometrically consistent with the real object.
    Initial object point cloud is sampled from this reconstruction; Section III-B.
  • domain assumption The object-centric contact map computed as normalized distance between object and hand point clouds is a task-relevant learning signal.
    Used as pretraining target; Eq. 2-3; no empirical validation that this matches true physical contact.
  • domain assumption The point cloud forward kinematics model accurately represents the hand surface from joint states.
    Hand point cloud in Eq. 1; follows Wei et al. [59].
  • ad hoc to paper Pretraining on play data transfers to downstream expert-demonstration tasks.
    Supported by ablations in Table VII, but the play data collection and its distribution are not described in detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World." pith.science (2026). https://pith.science/paper/523F5QHI

@misc{pith2026250208449,
  author       = {Pith},
  title        = {Pith review of: CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/523F5QHI}},
  note         = {Machine review of arXiv:2502.08449}
}
read the original abstract

Achieving human-level dexterity in robots is a key objective in the field of robotic manipulation. Recent advancements in 3D-based imitation learning have shown promising results, providing an effective pathway to achieve this goal. However, obtaining high-quality 3D representations presents two key problems: (1) the quality of point clouds captured by a single-view camera is significantly affected by factors such as camera resolution, positioning, and occlusions caused by the dexterous hand; (2) the global point clouds lack crucial contact information and spatial correspondences, which are necessary for fine-grained dexterous manipulation tasks. To eliminate these limitations, we propose CordViP, a novel framework that constructs and learns correspondences by leveraging the robust 6D pose estimation of objects and robot proprioception. Specifically, we first introduce the interaction-aware point clouds, which establish correspondences between the object and the hand. These point clouds are then used for our pre-training policy, where we also incorporate object-centric contact maps and hand-arm coordination information, effectively capturing both spatial and temporal dynamics. Our method demonstrates exceptional dexterous manipulation capabilities, achieving state-of-the-art performance in six real-world tasks, surpassing other baselines by a large margin. Experimental results also highlight the superior generalization and robustness of CordViP to different objects, viewpoints, and scenarios. Code and videos are available on https://aureleopku.github.io/CordViP.

Figures

Figures reproduced from arXiv: 2502.08449 by the authors.

Figure 1
Figure 1. We propose CordViP, a correspondence-based visuomotor policy for dexterous manipulation in the real world. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview Framework (a) We first employ TripoSR to generate the initial object point cloud and FoundationPose to estimate the 6D pose of the object. In parallel, the hand point cloud is generated based on the robot’s state. They are combined to construct interaction-aware point clouds, which demonstrate robustness to viewpoint variations. (b) During the pre-training phase, the generated point cloud data, combined wit… view at source ↗
Figure 3
Figure 3. Point Clouds Comparison. We present point clouds of two methods under three different viewpoints. Notably, for better visualization, we have applied color information to the point clouds. However, color information is not used in the policy learning. at 8Hz, a frequency that is sufficiently high for our experimental setup, where inference is conducted at 5Hz. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Real robot system. Our system consists of a Leap Hand and a UR5 Arm, with a fixed Realsense L515 camera employed to capture visual observation. The Realsense D435 camera is only used for data collection during teleoperation and is not involved in the policy learning. A…
Figure 5
Figure 5. Figure 5: Visualization of six dexterous manipulation tasks, with the right side showing the end state. (a) PickPlace (b) FlipCup (c) Assembly (d) ArtiManip [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Randomization of Object Positions. The red rectangles mark the range of positions of manipulated objects. For PickPlace and FlipCup, both the toy chicken and the cup are randomly rotated within a certain range. tasks such as FlipCap is limited due to the impact of 3D n…
Figure 7
Figure 7. Figure 7: Experimental results of efficiency. We train ACT, DP, DP3, and CordViP on the PickPlace and FlipCup tasks with an increasing number of demonstrations. C. Efficiency The number of expert demonstrations plays a crucial role in the performance of imitation learning. To ac…
Figure 8
Figure 8. Figure 8: Failure case. (a) Case 1 is a failure case from the Flip Cup task; (b) Case2 is an example from the Assembly task, where after a failure to insert the cup, the policy made adjustments, corrected the orientation of the cup, and successfully completed the task. TABLE VII…
Figure 10
Figure 10. Figure 10: 3D digital asset generation from a single view. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 9
Figure 9. Figure 9: Visualization of Point Cloud Processing. The point cloud is synthesized from RGBD data. The point cloud is then cropped and processed using farthest point sampling (FPS) to generate 1024 points. We collect both the robot’s state and actions using joint angles in radian…
Figure 11
Figure 11. Figure 11: Comparison of Motion Patterns. DP3 Uses Axis￾Wise Actions. RISE. RISE [56] is a recently proposed end-to-end baseline for real-world imitation learning, which predicts continuous ac￾tions directly from single-view point clouds. It takes voxelized point clouds as input…
Figure 12
Figure 12. Figure 12: Visualization of Point Clouds During the Task Process. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A visuo-tactile imitation policy pre-trained on simulated point-cloud reconstruction achieves 83% average success manipulating transparent objects on a real robot.

  2. DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization

    cs.RO 2025-11 conditional novelty 5.0 of 10

    Fusing RGB and point-cloud inputs with training-time modality dropout plus cross-attention makes a diffusion visuomotor policy markedly more robust to visual and spatial shifts than unimodal or naively fused baselines.

Reference graph

Works this paper leans on

74 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Dexterous manipulation using both palm and fingers

    Yunfei Bai and C Karen Liu. Dexterous manipulation using both palm and fingers. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 1560–1565. IEEE, 2014

    work page 2014

  2. [2]

    All the feels: A dexterous hand with large-area tactile sensing

    Raunaq Bhirangi, Abigail DeFranco, Jacob Adkins, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, and Vikash Kumar. All the feels: A dexterous hand with large-area tactile sensing. IEEE Robotics and Automation Letters, 2023

    2023

  3. [3]

    A system for general in-hand object re-orientation

    Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning, pages 297–307. PMLR, 2022

    2022

  4. [4]

    G3flow: Generative 3d semantic flow for pose-aware and generalizable object manipulation

    Tianxing Chen, Yao Mu, Zhixuan Liang, Zanxin Chen, Shijia Peng, Qiangyu Chen, Mingkun Xu, Ruizhen Hu, Hongyuan Zhang, Xuelong Li, et al. G3flow: Generative 3d semantic flow for pose-aware and generalizable object manipulation. arXiv preprint arXiv:2411.18369 , 2024

    arXiv 2024

  5. [5]

    Towards human-level bimanual dexterous manipulation with re- inforcement learning

    Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with re- inforcement learning. Advances in Neural Information Processing Systems, 35:5150–5163, 2022

    work page 2022

  6. [6]

    Sequential dexterity: Chaining dexterous policies for long- horizon manipulation

    Yuanpei Chen, Chen Wang, Li Fei-Fei, and C Karen Liu. Sequential dexterity: Chaining dexterous policies for long- horizon manipulation. arXiv preprint arXiv:2309.00987 , 2023

    arXiv 2023

  7. [7]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  8. [8]

    Cover and P

    T. Cover and P. Hart. Nearest neighbor pattern classifica- tion. IEEE Transactions on Information Theory , 13(1): 21–27, 1967. doi: 10.1109/TIT.1967.1053964

    arXiv 1967

Show all 74 references
  1. [9]

    Preafford: Universal affordance-based pre-grasping for diverse objects and environments

    Kairui Ding, Boyuan Chen, Ruihai Wu, Yuyang Li, Zongzheng Zhang, Huan-ang Gao, Siqi Li, Guyue Zhou, Yixin Zhu, Hao Dong, et al. Preafford: Universal affordance-based pre-grasping for diverse objects and environments. In 2024 IEEE/RSJ International Confer- ence on Intelligent R...

    2024

  2. [10]

    Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation

    Ria Doshi, Homer Walke, Oier Mees, Sudeep Dasari, and Sergey Levine. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. arXiv preprint arXiv:2408.11812 , 2024

    2024 arXiv

  3. [11]

    Flowbot3d: Learning 3d articulation flow to manipulate articulated objects

    Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.04382 , 2022

    2022 arXiv

  4. [12]

    Deep se (3)-equivariant geometric reasoning for precise placement tasks

    Ben Eisner, Yi Yang, Todor Davchev, Mel Vecerik, Jonathan Scholz, and David Held. Deep se (3)-equivariant geometric reasoning for precise placement tasks. arXiv preprint arXiv:2404.13478, 2024

    2024 arXiv

  5. [13]

    Act3d: Infinite resolution action detection transformer for robotic manipulation

    Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: Infinite resolution action detection transformer for robotic manipulation. arXiv preprint arXiv:2306.17817, 2023

    2023 arXiv

  6. [14]

    Rvt: Robotic view transformer for 3d object manipulation

    Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  7. [15]

    Reset-free reinforcement learning via multi- task learning: Learning dexterous manipulation behaviors without human intervention

    Abhishek Gupta, Justin Yu, Tony Z Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi- task learning: Learning dexterous manipulation behaviors without human intervention. In 2021 IEEE International Confe...

    2021

  8. [16]

    Towards robust and domain agnostic reinforcement learning competitions: Minerl

    William Hebgen Guss, Stephanie Milani, Nicholay Topin, Brandon Houghton, Sharada Mohanty, Andrew Mel- nik, Augustin Harter, Benoit Buschmaas, Bjarne Jaster, Christoph Berganski, et al. Towards robust and domain agnostic reinforcement learning competitions: Minerl

  9. [17]

    Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play

    Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play. arXiv preprint arXiv:2303.12076, 2023

    2023 arXiv

  10. [18]

    See to touch: Learning tactile dexterity through visual incentives

    Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 13825–13832. IEEE, 2024

    2024

  11. [19]

    Teach a robot to fish: Versatile imitation from one minute of demonstrations

    Siddhant Haldar, Jyothish Pari, Anant Rai, and Ler- rel Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023

    2023 arXiv

  12. [20]

    Dextreme: Transfer of agile in-hand manipulation from simulation to reality

    Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE Inte...

    2023

  13. [21]

    Sur- femb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings

    Rasmus Laurvig Haugaard and Anders Glent Buch. Sur- femb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6749–6758, 2022

    2022

  14. [22]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851, 2020

    2020

  15. [23]

    Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation, 2024

    Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, and Shanghang Zhang. Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation, 2024. URL https://arxi...

    2024 arXiv

  16. [24]

    Hand-object contact consistency reasoning for hu- man grasps generation

    Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong 11 Wang. Hand-object contact consistency reasoning for hu- man grasps generation. In Proceedings of the International Conference on Computer Vision , 2021

    2021

  17. [25]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023

    2023 arXiv

  18. [26]

    Real-time behaviour synthesis for dynamic hand- manipulation

    Vikash Kumar, Yuval Tassa, Tom Erez, and Emanuel Todorov. Real-time behaviour synthesis for dynamic hand- manipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 6808–6815. IEEE, 2014

    2014

  19. [27]

    The functional correspondence problem

    Zihang Lai, Senthil Purushwalkam, and Abhinav Gupta. The functional correspondence problem. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15772–15781, 2021

    2021

  20. [28]

    Dexhanddiff: Interaction-aware diffusion planning for adaptive dexterous manipulation,

    Zhixuan Liang, Yao Mu, Yixiao Wang, Tianxing Chen, Wenqi Shao, Wei Zhan, Masayoshi Tomizuka, Ping Luo, and Mingyu Ding. Dexhanddiff: Interaction-aware diffusion planning for adaptive dexterous manipulation,

  21. [29]

    Limitations of autore- gressive models and their alternatives

    Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R Gormley, and Jason Eisner. Limitations of autore- gressive models and their alternatives. arXiv preprint arXiv:2010.11939, 2020

    2010 arXiv

  22. [30]

    Avr: Active vision-driven robotic precision manipulation with viewpoint and focal length optimization

    Yushan Liu, Shilong Mu, Xintao Chao, Zizhen Li, Yao Mu, Tianxing Chen, Shoujie Li, Chuqiao Lyu, Xiao-ping Zhang, and Wenbo Ding. Avr: Active vision-driven robotic precision manipulation with viewpoint and focal length optimization. arXiv preprint arXiv:2503.01439 , 2025

    2025

  23. [31]

    Manicm: Real-time 3d diffusion policy via consistency model for robotic manipulation

    Guanxing Lu, Zifeng Gao, Tianxing Chen, Wenxun Dai, Ziwei Wang, and Yansong Tang. Manicm: Real-time 3d diffusion policy via consistency model for robotic manipulation. arXiv preprint arXiv:2406.01586 , 2024

    2024 arXiv

  24. [32]

    What matters in learning from offline human demonstrations for robot manipulation

    Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Mart ´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298 , 2021

    2021 arXiv

  25. [33]

    Contact-invariant optimization for hand manipulation

    Igor Mordatch, Zoran Popovi ´c, and Emanuel Todorov. Contact-invariant optimization for hand manipulation. In Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation , pages 137–144, 2012

    2012

  26. [34]

    Robotwin: Dual-arm robot benchmark with generative digital twins (early version)

    Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins (early version). arXiv preprint arXiv:2409.02920, 2024

    2024 arXiv

  27. [35]

    Robotwin: Dual-arm robot benchmark with generative digital twins, 2025

    Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, Lunkai Lin, Zhiqiang Xie, Mingyu Ding, and Ping Luo. Robotwin: Dual-arm robot benchmark with generative digital twins, 2025. URL https://arxiv.org/abs/2504.13059

    2025 arXiv

  28. [36]

    Dgcm-net: dense geometrical correspondence matching network for incremental experience-based robotic grasping

    Timothy Patten, Kiru Park, and Markus Vincze. Dgcm-net: dense geometrical correspondence matching network for incremental experience-based robotic grasping. Frontiers in Robotics and AI , 7:120, 2020

    2020

  29. [37]

    Reconstructing hands in 3D with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024

    2024

  30. [38]

    Imitating human behaviour with diffusion models

    Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hof- mann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 , 2023

    2023 arXiv

  31. [39]

    Dextrous tactile in-hand manipulation using a modular reinforcement learning architecture

    Johannes Pitz, Lennart R ¨ostel, Leon Sievers, and Berthold B¨auml. Dextrous tactile in-hand manipulation using a modular reinforcement learning architecture. In 2023 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 1852–1858. IEEE, 2023

    2023

  32. [40]

    Alvinn: An autonomous land vehicle in a neural network

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988

    1988

  33. [41]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016

    2016 arXiv

  34. [42]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413 , 2017

    2017 arXiv

  35. [43]

    In-hand object rotation via rapid motor adaptation

    Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning , pages 1722–1732. PMLR, 2023

    2023

  36. [44]

    Pointnext: Revisiting pointnet++ with improved training and scaling strategies

    Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems (NeurIPS) , 2022

    2022

  37. [45]

    Dexmv: Imitation learning for dexterous manipulation from human videos, 2021

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos, 2021

    2021

  38. [46]

    Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system

    Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577, 2023

    2023 arXiv

  39. [47]

    Goal-conditioned imitation learning us- ing score-based diffusion policies

    Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning us- ing score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023

    2023 arXiv

  40. [48]

    Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning

    Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning. arXiv preprint arXiv:2309.06440 , 2023

    2023 arXiv

  41. [49]

    12 Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 12 Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning , pages 785–799. PMLR, 2023

    2023

  42. [50]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  43. [51]

    Motion before action: Diffusing object motion as manipulation condition

    Yup Su, Xinyu Zhan, Hongjie Fang, Yong-Lu Li, Cewu Lu, and Lixin Yang. Motion before action: Diffusing object motion as manipulation condition. arXiv preprint arXiv:2411.09658, 2024

    2024 arXiv

  44. [52]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  45. [53]

    Triposr: Fast 3d object reconstruction from a single image

    Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Li, Ding Liang, Chris- tian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151, 2024

    2024 arXiv

  46. [54]

    Mimicplay: Long-horizon imitation learning by watching human play

    Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 , 2023

    2023 arXiv

  47. [55]

    Karen Liu

    Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C. Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788 , 2024

    2024 arXiv

  48. [56]

    Rise: 3d perception makes real-world robot imitation simple and effective

    Chenxi Wang, Hongjie Fang, Hao-Shu Fang, and Cewu Lu. Rise: 3d perception makes real-world robot imitation simple and effective. arXiv preprint arXiv:2404.12281 , 2024

    2024 arXiv

  49. [57]

    Equivariant diffusion policy

    Dian Wang, Stephen Hart, David Surovik, Tarik Keleste- mur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id= wD2kUVLT1g

    2024

  50. [58]

    Dexgrasp- net: A large-scale robotic dexterous grasp dataset for general objects based on simulation

    Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang. Dexgrasp- net: A large-scale robotic dexterous grasp dataset for general objects based on simulation. arXiv preprint arXiv:2210.02697, 2022

    arXiv

  51. [59]

    D(r,o) grasp: A unified representation of robot and object interaction for cross-embodiment dexterous grasping

    Zhenyu Wei, Zhixuan Xu, Jingxiang Guo, Yiwen Hou, Chongkai Gao, Zhehao Cai, Jiayu Luo, and Lin Shao. D(r,o) grasp: A unified representation of robot and object interaction for cross-embodiment dexterous grasping. arXiv preprint arXiv:2410.01702 , 2024

    arXiv

  52. [60]

    FoundationPose: Unified 6d pose estimation and tracking of novel objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024

    2024

  53. [61]

    Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual cor- respondence

    Ruihai Wu, Haoran Lu, Yiyan Wang, Yubo Wang, and Hao Dong. Unigarmentmanip: A unified framework for category-level garment manipulation via dense visual cor- respondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024

    2024

  54. [62]

    Canonical representation and force-based pretraining of 3d tactile for dexterous visuo-tactile policy learning

    Tianhao Wu, Jinzhou Li, Jiyao Zhang, Mingdong Wu, and Hao Dong. Canonical representation and force-based pretraining of 3d tactile for dexterous visuo-tactile policy learning. arXiv preprint arXiv:2409.17549 , 2024

    2024 arXiv

  55. [63]

    Cage: Causal attention enables data-efficient generalizable robotic manipulation

    Shangning Xia, Hongjie Fang, Cewu Lu, and Hao-Shu Fang. Cage: Causal attention enables data-efficient generalizable robotic manipulation. arXiv preprint arXiv:2410.14974, 2024

    2024 arXiv

  56. [64]

    Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation

    Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung-Wei Ke, and Katerina Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023

    2023

  57. [65]

    Rotating without seeing: Towards in-hand dexterity through touch

    Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023

    2023

  58. [66]

    Rotating without seeing: Towards in-hand dexterity through touch

    Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. arXiv preprint arXiv:2303.10880, 2023

    2023 arXiv

  59. [67]

    Kebria, Abbas Khosravi, and Saeid Nahavandi

    Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics , 54(12):7173–7186, 2024. doi: 10.1109/TCYB.2024.3395626

    2024

  60. [68]

    Generalizable humanoid manipulation with improved 3d diffusion policies

    Yanjie Ze, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Jiajun Wu. Generalizable humanoid manipulation with improved 3d diffusion policies. arXiv preprint arXiv:2410.10803, 2024

    2024 arXiv

  61. [69]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation , 2024

    2024

  62. [70]

    Learning fine-grained bimanual manipulation with low-cost hardware

    Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023

    2023 arXiv

  63. [71]

    Demo” refers to the number of demonstrations collected for each task, “Episode Length

    Wenshuai Zhao, Jorge Pe ˜na Queralta, and Tomi Wester- lund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI) , pages 737–744. IEEE, 2020. 13 APPENDIX A. Real-World Task Description P...

    2020

  64. [74]

    The visual results are shown in the figure 10

    to generate digital twins from a single-view image, which enables the creation of high-quality 3D assets. The visual results are shown in the figure 10. 6D Pose Estimation. We utilize FoundationPose [ 60] to perform robust 6D pose estimation for various objects across tasks. F...

  65. [2020]

    PMLR, 2021

    In NeurIPS 2020 Competition and Demonstration Track, pages 233–252. PMLR, 2021

    2020

  66. [2024]

    URL https://arxiv.org/abs/2411.18562

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.