pith. sign in

arxiv: 2606.11767 · v2 · pith:ZOMLERUAnew · submitted 2026-06-10 · 💻 cs.RO · cs.AI

Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning

Pith reviewed 2026-06-27 09:48 UTC · model grok-4.3

classification 💻 cs.RO cs.AI
keywords blind graspingdexterous handtactile sensingReal2Simdiffusion policyreinforcement learningrobotic manipulation
0
0 comments X

The pith

Tactile-only policies learned via Real2Sim calibration enable blind dexterous grasping at 27% real-world success without vision.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors aim to show that a fully tactile policy can be learned for blind grasping with a dexterous hand and transferred to reality. They do this by building a calibrated simulator that matches real tactile contacts, training a geometry-aware encoder on tactile layouts, and distilling multiple object-specific RL policies into one diffusion policy conditioned on tactile input. This produces a system that works on the physical LEAP Hand for both seen and unseen objects. The result matters because it demonstrates tactile-only control is feasible without any real grasping data or cameras. If correct, it opens manipulation in dark or occluded settings.

Core claim

By constructing a contact-calibrated digital-twin simulator, pretraining a layout-aware tactile encoder on sensor geometry, and aggregating trajectories from object-specific reinforcement learning experts into a tactile-conditioned Diffusion Policy, the method achieves a 27% grasp success rate on a physical multi-fingered hand across 20 objects using only tactile observations.

What carries the argument

The three-component Real2Sim2Real framework: contact calibration for simulation fidelity, self-supervised layout-aware tactile encoding, and diffusion policy aggregation from RL experts.

Load-bearing premise

The contact-calibrated digital-twin simulator reproduces real tactile signals with sufficient accuracy for policy transfer.

What would settle it

Direct comparison of tactile signal patterns from the same contact events in the calibrated simulator versus the physical hand showing substantial mismatch would indicate the calibration fails to bridge the gap.

Figures

Figures reproduced from arXiv: 2606.11767 by Chenxi Xiao, Shengcheng Luo, Wanlin Li, Xiyan Huang, Zhe Xu, Ziyuan Jiao.

Figure 1
Figure 1. Figure 1: Zero-shot sim-to-real transfer of a tactile-conditioned control policy for reactive blind grasping. The sequence demonstrates the deployed policy’s ability to perform blind object interac￾tion, including dynamically searching for the object, adjusting grasp contacts, and lifting, all without visual input and using only robot state and sparse tactile feedback. Abstract: Blind grasping with a dexterous hand … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed Real2Sim2Real framework for blind dexterous grasping. We [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Hardware setup, tactile layout, and object sets for blind dexterous grasping. (A) Tactile [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Full-hand tactile LeapHand platform. The platform integrates four curved TwinTac fingertip sensors and twelve distributed FSR patches, forming a 44-channel tactile sensing system. The curved TwinTac module provides high-resolution fingertip contact measurements through an 8- taxel pressure array, while the FSR patches extend tactile coverage to non-fingertip regions including finger links and the palm. The… view at source ↗
Figure 5
Figure 5. Figure 5: Real2Sim tactile alignment system. (A) Paired calibration trajectories are collected in the real setup and replayed in simulation under matched object poses. During real-world collection, the hand interacts with fixed calibration objects while raw tactile readings and hand joint positions (qpos) are recorded; the recorded qpos are then replayed in simulation to obtain temporally aligned real–sim tactile se… view at source ↗
Figure 6
Figure 6. Figure 6: Tactile activation alignment before and after calibration. Binary tactile trigger events are shown for calibration trajectories, with FSR responses on top and fingertip taxel responses on bottom. Each vertical mark denotes a frame where at least one valid tactile channel in that modal￾ity is active. Black indicates real tactile measurements, orange indicates the nominal uncalibrated simulation, and blue in… view at source ↗
read the original abstract

Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27\% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper proposes a Real2Sim2Real framework for tactile-only blind dexterous grasping on a multi-fingered hand. Key components are a contact-calibrated digital-twin simulator via Real2Sim tactile calibration, a layout-aware tactile encoder using sensor-geometry priors and self-supervised pretraining, and aggregation of successful trajectories from object-specific RL experts into a tactile-conditioned Diffusion Policy. On a physical LEAP Hand with distributed tactile sensing, the policy achieves 27% real-world grasp success across 10 seen and 10 unseen objects with no real grasping demonstrations or visual input. Simulation ablations confirm benefits of layout-aware pretraining, and sensing-level checks show improved sim-hardware tactile consistency.

Significance. If the empirical results hold under full experimental reporting, the work demonstrates a concrete, falsifiable path for closing the tactile sim-to-real gap in dexterous manipulation without vision or real demos. The contact-event calibration, geometry-aware representation, and diffusion aggregation together address sparse-signal expressiveness and generalization, offering reusable techniques for tactile robotics that could be extended to other contact-rich tasks.

major comments (1)
  1. [Abstract and Evaluation] Abstract and Evaluation section: the central claim of 27% real-world success across all 20 objects is presented without trial counts per object, explicit success criteria, baseline comparisons on hardware, or statistical tests. These details are load-bearing for assessing whether the result supports the framework's effectiveness.
minor comments (1)
  1. [Abstract] The project page URL is given in abbreviated form; expand to a complete https link for accessibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation of minor revision. We agree that clearer reporting of experimental details is necessary to support the central claims and will revise the abstract and evaluation sections accordingly.

read point-by-point responses
  1. Referee: [Abstract and Evaluation] Abstract and Evaluation section: the central claim of 27% real-world success across all 20 objects is presented without trial counts per object, explicit success criteria, baseline comparisons on hardware, or statistical tests. These details are load-bearing for assessing whether the result supports the framework's effectiveness.

    Authors: We acknowledge that the current presentation lacks sufficient detail on these points. In the revised manuscript we will: (1) report the exact number of trials conducted per object (typically 10 trials each for the 20 objects); (2) provide an explicit definition of success (stable lift of the object for at least 5 seconds without dropping or excessive slip); (3) add hardware baseline comparisons, including a random-action policy and a simple heuristic tactile-threshold policy, evaluated under identical conditions; and (4) include statistical measures such as 95% confidence intervals on the success rates and, where appropriate, paired statistical tests across conditions. These additions will be placed in both the abstract and the main evaluation section. We note that because the method uses no real-world grasping demonstrations, learned baselines from prior work are not directly comparable, but the non-learning baselines will still be reported. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper reports an empirical engineering framework evaluated via physical robot deployment (27% real-world success on 20 objects with no real demos or vision). The three components (Real2Sim calibration, layout-aware encoder, diffusion aggregation) are validated through simulation ablations and hardware sensing consistency checks rather than any derivation chain. No equations, fitted parameters renamed as predictions, self-citations as load-bearing uniqueness theorems, or ansatzes smuggled via prior work appear in the abstract or description. The result is a concrete, externally falsifiable outcome from hardware experiments, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the framework rests on standard assumptions of RL transfer and sim-to-real consistency that are not detailed here.

pith-pipeline@v0.9.1-grok · 5826 in / 1147 out tokens · 24306 ms · 2026-06-27T09:48:44.988627+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 9 canonical work pages

  1. [1]

    Roberto, O

    C. Roberto, O. Andrew, J. Dinesh, L. Justin, Y . Wenzhen, M. Jitendra, E. H. Adelson, and L. Sergey. More than a feeling: Learning to grasp and regrasp using vision and touch.IEEE Robotics & Automation Letters, 3(4):3300–3307, 2018

  2. [2]

    Ward-Cherrier, N

    B. Ward-Cherrier, N. Pestell, L. Cramphorn, B. Winstone, M. E. Giannaccini, J. Rossiter, and N. F. Lepora. The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies.Soft Robotics, pages 216–227, 2018

  3. [3]

    Z.-H. Yin, B. Huang, Y . Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch, 2023. URLhttps://arxiv.org/abs/2303.10880

  4. [4]

    H. Dang, J. Weisz, and P. K. Allen. Blind grasping: Stable robotic grasping using tactile feedback and hand kinematics. InICRA, pages 5917–5922. IEEE, 2011. URLhttp://dblp. uni-trier.de/db/conf/icra/icra2011.html#DangWA11

  5. [5]

    Schnabel, Phys

    Z. Kappassov, J.-A. Corrales, and V . Perdereau. Tactile sensing in dexterous robot hands - review.Robot. Auton. Syst., 74(PA):195–220, Dec. 2015. ISSN 0921-8890. doi:10.1016/j. robot.2015.07.015. URLhttps://doi.org/10.1016/j.robot.2015.07.015

  6. [6]

    Church, J

    A. Church, J. Lloyd, raia hadsell, and N. F. Lepora. Tactile sim-to-real policy transfer via real-to-sim image translation. In5th Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=2NcPgLa7yqD

  7. [7]

    Garcia-Garcia, B

    A. Garcia-Garcia, B. S. Zapata-Impata, S. Orts-Escolano, P. Gil, and J. Garcia-Rodriguez. Tactilegcn: A graph convolutional network for predicting grasp stability with tactile sensors,

  8. [8]

    URLhttps://arxiv.org/abs/1901.06181

  9. [9]

    Y . Li, W. Du, C. Yu, P. Li, Z. Zhao, T. Liu, C. Jiang, Y . Zhu, and S. Huang. Taccel: Scaling up vision-based tactile robotics via high-performance GPU simulation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview. net/forum?id=PtGMadeONU

  10. [10]

    Si and W

    Z. Si and W. Yuan. Taxim: An example-based simulation model for gelsight tactile sensors,

  11. [11]

    URLhttps://arxiv.org/abs/2109.04027

  12. [12]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion, 2024. URLhttps://arxiv.org/ abs/2303.04137

  13. [13]

    X. Hu, A. Venkatesh, Y . Wan, G. Zheng, N. Jawale, N. Kaur, X. Chen, and P. Birkmeyer. Learning to detect slip through tactile estimation of the contact force field and its entropy,

  14. [14]

    URLhttps://arxiv.org/abs/2303.00935

  15. [15]

    Holweg, H

    E. Holweg, H. Hoeve, W. Jongkind, L. Marconi, C. Melchiorri, and C. Bonivento. Slip de- tection by tactile sensors: algorithms and experimental results. InProceedings of IEEE Inter- national Conference on Robotics and Automation, volume 4, pages 3234–3239 vol.4, 1996. doi:10.1109/ROBOT.1996.509205. 9

  16. [16]

    Lin, Z.-H

    T. Lin, Z.-H. Yin, H. Qi, P. Abbeel, and J. Malik. Twisting lids off with two hands. arXiv:2403.02338, 2024

  17. [17]

    Y . Yuan, H. Che, Y . Qin, B. Huang, Z.-H. Yin, K.-W. Lee, Y . Wu, S.-C. Lim, and X. Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing, 2024. URLhttps:// arxiv.org/abs/2312.01853

  18. [18]

    P. Lin, Y . Huang, W. Li, J. Ma, C. Xiao, and Z. Jiao. Pp-tac: Paper picking using tactile feedback in dexterous robotic hands.arXiv preprint arXiv:2504.16649, 2025

  19. [19]

    Huang, M

    H.-J. Huang, M. A. Mirzaee, M. Kaess, and W. Yuan. Gelslam: A real-time, high-fidelity, and robust 3d tactile slam system, 2026. URLhttps://arxiv.org/abs/2508.15990

  20. [20]

    W. Yuan, R. Li, M. A. Srinivasan, and E. H. Adelson. Measurement of shear and slip with a gelsight tactile sensor. In2015 IEEE International Conference on Robotics and Automation (ICRA), pages 304–311, 2015. doi:10.1109/ICRA.2015.7139016

  21. [21]

    Bauza, E

    M. Bauza, E. Valls, B. Lim, T. Sechopoulos, and A. Rodriguez. Tactile object pose estimation from the first touch with geometric contact rendering, 2020. URLhttps://arxiv.org/abs/ 2012.05205

  22. [22]

    Yamaguchi and C

    A. Yamaguchi and C. G. Atkeson. Implementing tactile behaviors using fingervision. In2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), pages 241– 248, 2017. doi:10.1109/HUMANOIDS.2017.8246881

  23. [23]

    Ding, Y .-Y

    Z. Ding, Y .-Y . Tsai, W. W. Lee, and B. Huang. Sim-to-real transfer for robotic manipulation with tactile sensory, 2021. URLhttps://arxiv.org/abs/2103.00410

  24. [24]

    L. Lach, R. Haschke, D. Tateo, J. Peters, H. J. Ritter, J. B. Sol, and C. Torras. To- wards transferring tactile-based continuous force control policies from simulation to robot. ArXiv, abs/2311.07245, 2023. URLhttps://api.semanticscholar.org/CorpusID: 265149679

  25. [25]

    W. Chen, J. Xu, F. Xiang, X. Yuan, H. Su, and R. Chen. General-purpose sim2real protocol for learning contact-rich manipulation with marker-based visuotactile sensors.IEEE Transactions on Robotics, 40:1509–1526, 2024. URLhttps://api.semanticscholar.org/CorpusID: 267005473

  26. [26]

    Narang, B

    Y . Narang, B. Sundaralingam, M. Macklin, A. Mousavian, and D. Fox. Sim-to-real for robotic tactile sensing via physics-based simulation and learned latent projections, 2021. URLhttps: //arxiv.org/abs/2103.16747

  27. [27]

    Church, J

    A. Church, J. Lloyd, R. Hadsell, and N. F. Lepora. Tactile sim-to-real policy transfer via real-to-sim image translation. InConference on Robot Learning, 2021. URLhttps://api. semanticscholar.org/CorpusID:240354208

  28. [28]

    K.-W. Lee, Y . Qin, X. Wang, and S.-C. Lim. Dextouch: Learning to seek and manipulate objects with tactile dexterity.IEEE Robotics and Automation Letters, 9(12):10772–10779, Dec. 2024. ISSN 2377-3774. doi:10.1109/lra.2024.3478571. URLhttp://dx.doi.org/ 10.1109/LRA.2024.3478571

  29. [29]

    Z.-H. Yin, B. Huang, Y . Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch.Robotics: Science and Systems, 2023

  30. [30]

    T. Liu, Z. Liu, Z. Jiao, Y . Zhu, and S.-C. Zhu. Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator.IEEE Robotics and Automation Letters, 7(1):470–477, Jan. 2022. ISSN 2377-3774. doi:10.1109/lra.2021. 3129138. URLhttp://dx.doi.org/10.1109/LRA.2021.3129138. 10

  31. [31]

    J. Chen, Y . Ke, and H. Wang. Bodex: Scalable and efficient robotic dexterous grasp synthesis using bilevel optimization, 2025. URLhttps://arxiv.org/abs/2412.16490

  32. [32]

    J. Chen, Y . Ke, L. Peng, and H. Wang. Dexonomy: Synthesizing all dexterous grasp types in a grasp taxonomy.Robotics: Science and Systems, 2025

  33. [33]

    X. Fei, Z. Xu, H. Fang, T. Zhang, and L. Shao. T(r,o) grasp: Efficient graph diffusion of robot- object spatial transformation for cross-embodiment dexterous grasping, 2025. URLhttps: //arxiv.org/abs/2510.12724

  34. [34]

    M. Yang, C. Lu, A. Church, Y . Lin, C. Ford, H. Li, E. Psomopoulou, D. A. W. Barton, and N. F. Lepora. Anyrotate: Gravity-invariant in-hand object rotation with sim-to-real touch,

  35. [35]

    URLhttps://arxiv.org/abs/2405.07391

  36. [36]

    Lee and N

    J. Lee and N. Fazeli. Vitascope: Visuo-tactile implicit representation for in-hand pose and extrinsic contact estimation, 2025. URLhttps://arxiv.org/abs/2506.12239

  37. [37]

    H. Li, S. Dikhale, S. Iba, and N. Jamali. Vihope: Visuotactile in-hand object 6d pose estimation with shape completion.IEEE Robotics and Automation Letters, 8(11):6963–6970, 2023. doi: 10.1109/LRA.2023.3313941

  38. [38]

    Huang, Y

    J. Huang, Y . Ye, Y . Gong, X. Zhu, Y . Gao, and K. Zhang. Spatially anchored tactile awareness for robust dexterous manipulation, 2026. URLhttps://arxiv.org/abs/2510.14647

  39. [39]

    Huang, Y

    B. Huang, Y . Wang, X. Yang, Y . Luo, and Y . Li. 3d-vitac: Learning fine-grained manipulation with visuo-tactile sensing, 2025. URLhttps://arxiv.org/abs/2410.24091

  40. [40]

    K. Shaw, A. Agarwal, and D. Pathak. Leap hand: Low-cost, efficient, and anthropomorphic hand for robot learning, 2023. URLhttps://arxiv.org/abs/2309.06440

  41. [41]

    Morton and M

    X. Huang, Z. Xu, and C. Xiao. Twintac: A wide-range, highly sensitive tactile sensor with real-to-sim digital twin sensor model. In2025 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 12286–12292, 2025. doi:10.1109/IROS60139.2025. 11247002

  42. [42]

    Fuji Tsang, M

    C. Fuji Tsang, M. Shugrina, J. F. Lafleche, T. Takikawa, J. Wang, and C. e. a. Loop. Kaolin: A pytorch library for accelerating 3d deep learning research.https://github. com/NVIDIAGameWorks/kaolin, 2022

  43. [43]

    O. I. Vinogradova and F. Feuillebois. Interaction of elastic bodies via surface forces: 2. expo- nential decay.Journal of colloid and interface science, 268(2):464–475, 2003

  44. [44]

    J. Xu, S. Kim, T. Chen, A. R. Garcia, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In6th Annual Conference on Robot Learning, 2022. URLhttps://openreview.net/forum?id=6BIffCl6gsM

  45. [45]

    Z. Zhao, W. Li, Y . Li, T. Liu, B. Li, M. Wang, K. Du, H. Liu, Y . Zhu, Q. Wang, K. Althoefer, and S.-C. Zhu. Embedding high-resolution touch across robotic hands enables adaptive human- like grasping.Nature Machine Intelligence, 7(6):889–900, 2025. ISSN 2522-5839. doi:10. 1038/s42256-025-01053-3. URLhttp://dx.doi.org/10.1038/s42256-025-01053-3

  46. [46]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. URLhttps://arxiv.org/abs/2304.13705

  47. [47]

    Zakka, B

    K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y . Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y . Tassa, and P. Abbeel. Mujoco playground, 2025. URLhttps://arxiv.org/abs/2502.08844

  48. [48]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URLhttps: //arxiv.org/abs/1711.05101. 11 Appendix Overview This appendix provides additional details for hardware design, evaluation protocol, failure analysis, ablations, expert data generation, tactile encoder pretraining, and policy deployment. A. Hardware and Tactile Sensing Sys...

  49. [49]

    (A) Paired Real--Sim Calibration Setup TwinTac FSR TwinTac FSR Real

    Learn tactile alignment parameters from paired sequences 00-4ce6-bc21-530309f9327g.png Sphere Cylinder (B) Calibration Fixtures and Objects (C) Real2Sim Tactile Alignment Calibration improves tactile event consistency between simulated and real sensors Real Sim Real Sim Aligned trajectories in real and simulation ensure rich, diverse contact on many tacti...

  50. [50]

    Collect controlled-contact trajectories

  51. [51]

    Replay hand qpos in simulation contact region 3D-printed fixture for repeatable object placement Real Sim Sim Aligned Sim Figure 5:Real2Sim tactile alignment system.(A) Paired calibration trajectories are collected in the real setup and replayed in simulation under matched object poses. During real-world collection, the hand interacts with fixed calibrati...