Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Tactile MNIST: Benchmarking Active Tactile Perception

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces the first benchmark suite built specifically for active tactile perception, with simulated and real touch data.

desk verdict A genuinely useful benchmark package for active tactile perception, held back mainly by an unverified sim-to-real story and a few correctness slips. read the letter →

arxiv 2506.06361 v2 pith:3B4VRGKJ submitted 2025-06-03 cs.RO cs.AIcs.LG

Tim Schneider , Guillaume Duret , Cristiana de Farias , Roberto Calandra , Liming Chen , Jan Peters This is my paper
classification cs.ROcs.AIcs.LG
keywords activeperceptiontactilesensingbenchmarkMNIST3Dvision-basedsensorssim-to-realtransferreinforcementlearningCycleGAN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Active tactile perception—where a robot decides where to touch next and accumulates evidence over many contacts—has lacked standardized evaluation. The authors claim to introduce the first benchmark suite built specifically for this setting: the Tactile MNIST Benchmark Suite (TMBS), built on a new Active Perception Gym (ap_gym), with four touch-only tasks covering digit classification, volume estimation, pose estimation, and object counting. The package includes 13,580 printable 3D digit models and 153,600 real tactile contacts from 600 printed digits, plus three rendering modes for simulated touch. If accepted, the benchmark gives researchers a common protocol for measuring both how quickly an agent learns and how efficiently it explores within an episode, and the paper's baseline results show that current methods leave these tasks far from solved.

What carries the argument

The load-bearing object is the active-perception episode itself, formalized as a partially observable Markov decision process (POMDP) with a special structure: the state splits into a hidden base state and a prediction target, the action splits into a movement command and a prediction, and the reward is a base reward minus a differentiable loss on the prediction. This forces the agent to predict after every contact, so exploration efficiency (how quickly accuracy rises within an episode) and sample efficiency (how many environment steps the learning algorithm needs) become measurable. The framework, ap_gym, packages that protocol into Gymnasium-compatible environments; TMBS adds the tactile layer with three rendering modes—depth, Taxim, and CycleGAN—and randomized object shifts in place of a physics engine.

What would settle it

Deploy a policy trained on TMBS (CycleGAN or Taxim rendering) on a physical robot with a vision-based tactile sensor touching 3D-printed MNIST digits, and compare its classification accuracy to a policy trained on the real tactile dataset. If the simulated-trained policy performs at chance while the real-trained policy succeeds, then the simulator is not faithful enough for the benchmark's transfer goal.

Watch

Extended reading notes

Core claim

The central claim is that active tactile perception lacks fair, reproducible evaluation, and that this paper supplies it. Concretely, the authors build ap_gym, an extensible environment framework in which an agent must predict a target property at every step while choosing where to sense next, and TMBS instantiates this framework with a simulated vision-based tactile sensor and four tasks. To support the simulator and downstream transfer, they generate MNIST 3D, 13,580 meshes of handwritten digits, print 600 of them, and collect 153,600 real tactile contacts with spatial and label annotations. They train a CycleGAN to render realistic tactile images from depth, and they report baseline results for reinforcement-learning active-perception agents, showing that current methods improve over random exploration but leave the tasks far from solved.

Load-bearing premise

The benchmark's usefulness for real robots rests on the assumption that tactile images rendered from depth maps, Taxim, or CycleGAN—with object motion faked by random shifts—accurately enough reflect real contact, an assumption the paper itself flags as untested.

Editorial extensions

If this is right

  • Researchers gain a common set of environments, train/test splits, and metrics so that different active-tactile policies can be compared on equal terms.
  • The reported baselines establish that off-policy reinforcement-learning agents outperform on-policy and random exploration on several tasks, yet none of the environments is close to solved.
  • The 153,600-contact real dataset gives an offline benchmark with realistic sensor noise and a source for pretraining tactile perception models.
  • The CycleGAN rendering mode offers a concrete route toward sim-to-real transfer without requiring paired real and synthetic tactile images.
  • The split into classification, localization, counting, and volume-estimation tasks lets researchers isolate different exploration strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If TMBS is predictive of real touch, the benchmark's two metrics—average and final accuracy—effectively measure an agent's ability to trade off exploration against exploitation, which would make it a useful testbed for POMDP reinforcement learning beyond tactile tasks.
  • The decision to skip a physics engine means contact-rich behaviors such as grasping or in-hand reorientation cannot be evaluated; adding a physics engine would be the natural next step, and the MNIST 3D meshes could then double as a mesh-classification benchmark.
  • A strong prediction implicit in the design is that the CycleGAN rendering mode should improve sim-to-real transfer more than depth or Taxim; that ordering is testable by comparing performance across the three renderers on a real robot.
  • One could measure the sim-to-real gap directly by training a model on simulated tactile images and evaluating on the real tactile contacts, producing a quantitative gap number rather than a qualitative impression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Active Perception Gym (ap_gym), a Gymnasium-compatible framework with eleven lightweight active-perception environments, and the Tactile MNIST Benchmark Suite (TMBS), which adds four simulated active tactile perception tasks—digit classification, volume estimation, tool pose estimation, and star counting—using a simulated GelSight Mini sensor with depth, Taxim, and CycleGAN rendering modes. The authors also release a dataset of 13,580 high-resolution 3D MNIST digit meshes and a real-world dataset of 153,600 tactile contacts collected from 600 3D-printed digits. Baseline experiments compare TAP variants, HAM, and a random agent across classification, localization, and pose-estimation tasks. The paper claims, to the best of the authors' knowledge, the first benchmark suite specifically for active tactile perception.

Significance. If adopted, the benchmark would fill a real gap: most existing tactile datasets are passive or lack standardized evaluation protocols. The public code and data repositories, detailed environment specifications in Appendix F, explicit POMDP formulation in Section 3.1, clear distinction between exploration efficiency and sample efficiency in Section 3.6, and train/test splits are concrete strengths that should make the resource easy to use and compare against. The baseline comparison across many environments is a useful starting point for the community. The main weakness is that the simulated-to-real transfer value is unvalidated: the paper is honest about this in Section 5, but the contributions and the CycleGAN description still imply transferability. As a result, the current significance is primarily as an in-simulation algorithmic benchmark rather than a proven sim-to-real platform.

major comments (2)
  1. [§3.3, §5] Section 3.3 states that "we do not simulate the physical interaction between the sensor and the object, we shift the objects around randomly to simulate unintended object movements," and Section 5 explicitly lists the absence of online, real-world evaluation as a limitation. These admissions leave the central premise of the benchmark's external validity—that progress on TMBS transfers to real active tactile perception—unsupported. The skeptical concern about this point partially lands, but the paper itself acknowledges the limitation; the load-bearing problem is the mismatch between this acknowledgment and the unqualified wording elsewhere, such as the contribution listing a CycleGAN for "realistic tactile simulation rendering." Please either add a minimal transfer experiment (e.g., classifying held-out real tactile images with a model trained on simulated renderings, or deploying a trained exploration policy on a physical robot for a few contacts) or explicitly restrict the claim to an in-simulation algorithmic benchmark and remove the transfer-facilitation language from the abstract and contribution list.
  2. [§C, Fig. 7] The CycleGAN rendering mode is a distinctive contribution and the only rendering mode intended to produce realistic tactile images, yet its evaluation in Section C is purely qualitative: Figure 7 shows example depth-to-RGB cycles, but no quantitative metric is reported—no FID, no held-out real-image classification accuracy, and no sim-to-real policy transfer result. Since the paper describes this mode as producing images that are "visually much more realistic" and "might be beneficial for sim-to-real transfer," the term "realistic" in the contribution list is unsupported. Please add a quantitative proxy, such as the classification accuracy on the real 153,600-contact dataset of a model trained on CycleGAN-rendered images, or revise the wording to "stylistically closer" and mark the mode as an unvalidated transfer aid.
minor comments (5)
  1. [§4, §G] The statement that "on-policy methods generally being more sample-efficient than off-policy methods, as on-policy methods cannot reuse previously collected samples" is the reverse of standard RL terminology: off-policy methods reuse data and are generally the more sample-efficient family. This inverted explanation appears twice and should be corrected.
  2. [Abstract, §3.4, Table 4] The abstract reports 13,500 synthetic 3D MNIST digit models, while Section 3.4 and Table 4 report 13,580; the numbers should be made consistent.
  3. [Table 4] The row labeled "printed_train" appears twice, once with 500 objects and once with 100 objects; the second row should presumably be named "printed_test" or a distinct split name.
  4. [Table 14, Table 16] The captions of Tables 14 and 16 say "in ap_gym," but both tables list TMBS environments (TactileMNIST, Starstruck, Toolbox, TactileMNISTVolume); the captions should refer to TMBS.
  5. [Appendix references] The HEBO reference appears as [65] in the main text and is duplicated as [68] in the appendix reference list; this duplication should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark, datasets, and baselines are constructed independently of the claims they support.

full rationale

The paper's central contributions are a benchmark suite, two released datasets, and baseline evaluations. The benchmark claim rests on the environments' definitions and the comparison table, not on any quantity fitted to a target result. The CycleGAN is fitted to the real tactile dataset, but it is used only as one optional rendering mode inside the simulator; no benchmark score or derived prediction is defined as the CycleGAN's training objective. The only same-author citation is TAP [52], which is used as an evaluation baseline rather than as justification for the benchmark's validity, so it is not load-bearing. The acknowledged absence of contact physics and online real-world evaluation is a limitation on sim-to-real transferability, not a circular derivation. No equation or protocol in the paper reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to establish the benchmark's central claim; HEBO-tuned baseline hyperparameters (Table 3) affect experiments but not benchmark validity. No new physical entities are introduced.

assumptions (4)
  • domain assumption Simulated tactile renderings preserve task-relevant information of real GelSight contacts, so policies improved on TMBS transfer to real active touch.
    Assumed in Sections 3.3 and 5. CycleGAN shown only qualitatively in Fig. 7; no quantitative fidelity or online real-robot validation.
  • domain assumption Active perception can be modeled as a POMDP in which prediction actions do not alter the hidden state.
    Formal assumption in Section 3.1, T(s,(abase,y),s') = T_base(s,abase,s'); valid for benchmark tasks but restrictive.
  • domain assumption Randomly shifting objects in simulation emulates unintended movement caused by physical contact.
    Used in TMBS environments; acknowledged in Section 5 as replacement for physics engine.
  • domain assumption MNIST 3D meshes generated by erosion stacking and marching cubes form valid tactile classification targets.
    Algorithm 1 in Appendix D.1; only 600 of 13,580 meshes were printed and touched, so most meshes are unvalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tactile MNIST: Benchmarking Active Tactile Perception." pith.science (2026). https://pith.science/paper/3B4VRGKJ

@misc{pith2026250606361,
  author       = {Pith},
  title        = {Pith review of: Tactile MNIST: Benchmarking Active Tactile Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3B4VRGKJ}},
  note         = {Machine review of arXiv:2506.06361}
}
read the original abstract

Tactile perception has the potential to significantly enhance dexterous robotic manipulation by providing rich local information that can complement or substitute for other sensory modalities such as vision. However, because tactile sensing is inherently local, it is not well-suited for tasks that require broad spatial awareness or global scene understanding on its own. A human-inspired strategy to address this issue is to consider active perception techniques instead. That is, to actively guide sensors toward regions with more informative or significant features and integrate such information over time in order to understand a scene or complete a task. Both active perception and different methods for tactile sensing have received significant attention recently. Yet, despite advancements, both fields lack standardized benchmarks. To bridge this gap, we introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark specifically designed for active tactile perception tasks, including localization, classification, and volume estimation. Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors. Furthermore, we also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits. Using this dataset, we train a CycleGAN for realistic tactile simulation rendering. By providing standardized protocols and reproducible evaluation frameworks, our benchmark suite facilitates systematic progress in the fields of tactile sensing and active perception.

Figures

Figures reproduced from arXiv: 2506.06361 by the authors.

Figure 1
Figure 1. Overview of the Active Perception Gym (ap_gym), the Tactile MNIST Benchmark Suite (TMBS), and their associated assets. In the center we depict each environment from ap_gym and TMBS, along with the included asset sets (both custom and external). On the left, an agent (e.g., an active perception algorithm) interacts with the environment by receiving observations, rewards, and losses, and returning actions. On the righ… view at source ↗
Figure 2
Figure 2. Illustration of the TactileMNIST classification task. In each episode of the TactileMNIST classification [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The agent receives the sensory information [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Data collection for the Tactile MNIST Real Static dataset. We mounted a GelSight Mini sensor on a Franka Research 3 (a) and collected 153,600 touches across 600 3D￾printed MNIST digits. Examples of individual touches are visible in (b), and a collection of 256 touches …
Figure 5
Figure 5. Figure 5: Average and final prediction accuracies for the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Active perception process in the Toolbox environment. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Qualitative results over four cycles illustrating the translation between depth images and RGB images. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The MNIST 3D dataset comprises 13,580 3D models of handwritten digits. Each model is auto￾generated from a high-resolution MNIST variant [59]. D.2 Starstruck In addition to the MNIST digits, we also introduce a dataset of geometric shapes designed for the Starstruck en…
Figure 9
Figure 9. Figure 9: Sequence of glimpses taken by the agent in the CircleSquare environment. Here we show the variation [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Sequence of glimpses taken by the agent in the MNIST environment on the test set. We show a failed [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Sequence of glimpses taken by the agent in the CIFAR10 environment. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Sequence of glimpses taken by the agent in the TinyImageNet environment. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Agent trajectory in the Room Localization environment. Here we see the 8 LIDAR beams extending [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Agent trajectory in the LIDARLocMazeStatic environment, where walls are narrow but similar. The purple dot is the agent’s predicted pose, and the line representing the path the agent has taken is encoded with a gradient from red to green, where red indicates a high pr…
Figure 15
Figure 15. Figure 15: Agent trajectory in the LightDark environment. The blue dot indicates the agent’s current position, while the light blue dot marks its previous position (i.e., the target for prediction). The purple dot shows the agent’s last prediction, and the green dot is the noisy…
Figure 16
Figure 16. Figure 16: Agent trajectory in CIFAR10Loc. The blue window represents the current glimpse, in transparent purple we have the target, and the opaque purple box shows the prediction. a gradient from red to green, where red indicates a high predicted error and green indicates a low…
Figure 17
Figure 17. Figure 17: Agent trajectory in TinyImageNetLoc. The blue window represents the current glimpse, in transparent purple we have the target, and the opaque purple box shows the prediction. a gradient from red to green, where red indicates a high predicted error and green indicates …
Figure 18
Figure 18. Figure 18: Sequence of tactile glimpses taken by the agent in the [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Sequence of tactile glimpses taken by the agent in the [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: Agent trajectory in the Toolbox environment. The agent collects local tactile readings of the wrench, sequentially integrating observations to regress its (x, y) position and orientation (represented as sine and cosine). The predicted pose is shown in transparent red …
Figure 21
Figure 21. Figure 21: Agent trajectory in the TactileMNISTVolume environment. The sensor explores the digit surface under pose perturbations to regress the object’s normalized volume. We visualize here the predicted volume (transparent red) overlaid with the object mesh throughout the epis…
Figure 22
Figure 22. Figure 22: Average and final prediction accuracies for the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]
Figure 23
Figure 23. Figure 23: Exploration efficiency comparison of the final policies at the end of the training on the Tactile MNIST [PITH_FULL_IMAGE:figures/full_fig_p035_23.png]
Figure 24
Figure 24. Figure 24: In general, the baseline methods seem to struggle more with this set of tasks than with the [PITH_FULL_IMAGE:figures/full_fig_p035_24.png]
Figure 25
Figure 25. Figure 25: Average and final prediction accuracies for the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p036_25.png]
Figure 26
Figure 26. Figure 26: Average and final prediction errors for the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p036_26.png]
Figure 27
Figure 27. Figure 27: Average and final prediction accuracies for the baseline methods [PITH_FULL_IMAGE:figures/full_fig_p037_27.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CONTACT: CONtact-aware TACTile Learning for Robotic Disassembly

    cs.RO 2026-03 conditional novelty 5.0 of 10

    In contact-rich robotic disassembly, compact force-field tactile representations (TacFF) outperform vision-only and high-resolution tactile-image policies, especially in tight-tolerance and deformable tasks; naive fus...

Reference graph

Works this paper leans on

78 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    The mnist database of handwritten digits.http://yann

    Yann LeCun. The mnist database of handwritten digits.http://yann. lecun. com/exdb/mnist/, 1998

    1998

  2. [2]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  3. [3]

    Tiny imagenet visual recognition challenge.CS 231N, 7(7):3, 2015

    Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge.CS 231N, 7(7):3, 2015

    2015

  4. [4]

    Coding and use of tactile signals from the fingertips in object manipulation tasks.Nature Reviews Neuroscience, 10(5):345–359, 2009

    Roland S Johansson and J Randall Flanagan. Coding and use of tactile signals from the fingertips in object manipulation tasks.Nature Reviews Neuroscience, 10(5):345–359, 2009

    work page 2009

  5. [5]

    Hand movements: A window into haptic object recognition.Cognitive psychology, 19(3):342–368, 1987

    Susan J Lederman and Roberta L Klatzky. Hand movements: A window into haptic object recognition.Cognitive psychology, 19(3):342–368, 1987

    work page 1987

  6. [6]

    Prescott, Mathew E

    Tony J. Prescott, Mathew E. Diamond, and Alan M. Wing. Active touch sensing.Phil. Trans. R. Soc. B, 366(1581):2989–2995, Nov 2011

    work page 2011

  7. [7]

    Boehm, T

    A. Boehm, T. Schneider, B. Belousov, A. Kshirsagar, L. Lin, K. Doerschner, K. Drewing, C.A. Rothkopf, and J. Peters. What matters for active texture recognition with vision-based tactile sensors. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  8. [8]

    Learning efficient haptic shape exploration with a rigid tactile sensor array.PloS one, 15(1):e0226880, 2020

    Sascha Fleer, Alexandra Moringen, Roberta L Klatzky, and Helge Ritter. Learning efficient haptic shape exploration with a rigid tactile sensor array.PloS one, 15(1):e0226880, 2020

    work page 2020

Show all 78 references
  1. [9]

    Tandem: Learning joint exploration and decision making with tactile sensors.IEEE Robotics and Automation Letters, 7(4):10391–10398, 2022

    Jingxi Xu, Shuran Song, and Matei Ciocarlie. Tandem: Learning joint exploration and decision making with tactile sensors.IEEE Robotics and Automation Letters, 7(4):10391–10398, 2022

    2022

  2. [10]

    Active tactile object exploration with gaussian processes

    Zhengkun Yi, Roberto Calandra, Filipe Veiga, Herke van Hoof, Tucker Hermans, Yilei Zhang, and Jan Peters. Active tactile object exploration with gaussian processes. In2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4925–4930. IEEE, 2016

    2016

  3. [11]

    Enhancing visual perception of shape through tactile glances

    Marten Björkman, Yasemin Bekiroglu, Virgile Högman, and Danica Kragic. Enhancing visual perception of shape through tactile glances. In2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3180–3186. IEEE, 2013

    2013

  4. [12]

    Evetac: An Event-Based Optical Tactile Sensor for Robotic Manipulation.IEEE Trans

    Niklas Funk, Erik Helmut, Georgia Chalvatzaki, Roberto Calandra, and Jan Peters. Evetac: An Event-Based Optical Tactile Sensor for Robotic Manipulation.IEEE Trans. Rob., 40:3812–3832, July 2024

    2024

  5. [13]

    Self-supervised visuo-tactile pretraining to locate and follow garment features.arXiv preprint arXiv:2209.13042, 2022

    Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features.arXiv preprint arXiv:2209.13042, 2022

    2022 arXiv

  6. [14]

    Sparsh: Self-supervised touch representations for vision-based tactile sensing

    Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, and Mustafa Mukadam. Sparsh: Self-supervised touch representations for vision-based tactile sensing. In 8th A...

    2024

  7. [15]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. InCVPR09, 2009

    2009

  8. [16]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings,...

    2014

  9. [17]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. InConference on Computer Vision and Pattern Recognition (CVPR), 2012

    2012

  10. [18]

    Omni3d: A large benchmark and model for 3d object detection in the wild

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13154–13164, 2023

    2023

  11. [19]

    Neuralfeels with neural fields: Visuotactile perception for in-hand manipulation.Science Robotics, 9(96):eadl0628, 2024

    Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, et al. Neuralfeels with neural fields: Visuotactile perception for in-hand manipulation.Science Robotics, 9(96):eadl0628, 2024

    2024

  12. [20]

    Actexplore: Active tactile exploration on unknown objects

    Amir-Hossein Shahidzadeh, Seong Jong Yoo, Pavan Mantripragada, Chahat Deep Singh, Cor- nelia Fermüller, and Yiannis Aloimonos. Actexplore: Active tactile exploration on unknown objects. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 3411–3418. IEEE, 2024

    2024

  13. [21]

    Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032, 2024

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  14. [22]

    Gelsight: High-resolution robot tactile sensors for estimating geometry and force.Sensors, 17(12):2762, 2017

    Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force.Sensors, 17(12):2762, 2017

    2017

  15. [23]

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2020

    2020

  16. [24]

    Aiding grasp synthesis for novel objects using heuristic-based and data-driven active vision methods.Frontiers in Robotics and AI, 8:696587, 2021

    Sabhari Natarajan, Galen Brown, and Berk Calli. Aiding grasp synthesis for novel objects using heuristic-based and data-driven active vision methods.Frontiers in Robotics and AI, 8:696587, 2021

    2021

  17. [25]

    A real 3d embodied dataset for robotic active visual learning.IEEE Robotics and Automation Letters, 7(3):6646–6652, 2022

    Qianfan Zhao, Lu Zhang, Lingxi Wu, Hong Qiao, and Zhiyong Liu. A real 3d embodied dataset for robotic active visual learning.IEEE Robotics and Automation Letters, 7(3):6646–6652, 2022

    2022

  18. [26]

    The robotic vision scene understanding challenge.arXiv preprint arXiv:2009.05246, 2020

    David Hall, Ben Talbot, Suman Raj Bista, Haoyang Zhang, Rohan Smith, Feras Dayoub, and Niko Sünderhauf. The robotic vision scene understanding challenge.arXiv preprint arXiv:2009.05246, 2020

    2009 arXiv

  19. [27]

    Next best view planning in a single glance: An approach to improve object recognition.SN Computer Science, 4(1):51, 2022

    Pourya Hoseini, Shuvo Kumar Paul, Mircea Nicolescu, and Monica Nicolescu. Next best view planning in a single glance: An approach to improve object recognition.SN Computer Science, 4(1):51, 2022

    2022

  20. [28]

    Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Kosecka, and Alexander C. Berg. A dataset for developing and benchmarking active vision. InIEEE International Conference on Robotics and Automation (ICRA), 2017

    2017

  21. [29]

    Active vision dataset benchmark

    Phil Ammirato, Alexander C Berg, and Jana Kosecka. Active vision dataset benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2046–2049, 2018. 11

    2018

  22. [30]

    Active object search

    Jie Wu, Tianshui Chen, Lishan Huang, Hefeng Wu, Guanbin Li, Ling Tian, and Liang Lin. Active object search. InProceedings of the 28th ACM International Conference on Multimedia, pages 973–981, 2020

    2020

  23. [31]

    Actiview: Evaluating active perception ability for multimodal large language models, 2024

    Ziyue Wang, Chi Chen, Fuwen Luo, Yurui Dong, Yuanchi Zhang, Yuzhuang Xu, Xiaolong Wang, Peng Li, and Yang Liu. Actiview: Evaluating active perception ability for multimodal large language models, 2024

    2024

  24. [32]

    The tip benchmark: A tactile image-based psychophysics-inspired benchmark for artificial tactile sensors

    Tianyi Liu and Benjamin Ward-Cherrier. The tip benchmark: A tactile image-based psychophysics-inspired benchmark for artificial tactile sensors. InInternational Conference on Human Haptic Sensing and Touch Enabled Computer Applications, pages 94–106. Springer, 2024

    2024

  25. [33]

    Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward H. Adelson. Transferable tactile trans- formers for representation learning across diverse sensors and tasks, 2024

    2024

  26. [34]

    Midas- Touch: Monte-Carlo inference over distributions across sliding touch

    Sudharshan Suresh, Zilin Si, Stuart Anderson, Michael Kaess, and Mustafa Mukadam. Midas- Touch: Monte-Carlo inference over distributions across sliding touch. InProc. Conf. on Robot Learning, CoRL, Auckland, NZ, December 2022

    2022

  27. [35]

    Active clothing material perception using tactile sensing and deep learning

    Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H Adelson. Active clothing material perception using tactile sensing and deep learning. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4842–4849. IEEE, 2018

    2018

  28. [36]

    Touch and go: Learning from human-collected vision and touch.arXiv preprint arXiv:2211.12498, 2022

    Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: Learning from human-collected vision and touch.arXiv preprint arXiv:2211.12498, 2022

    2022 arXiv

  29. [37]

    Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition

    Shan Luo, Wenzhen Yuan, Edward Adelson, Anthony G Cohn, and Raul Fuentes. Vitac: Feature sharing between vision and tactile sensing for cloth texture recognition. In2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2722–2727. IEEE, 2018

    2018

  30. [38]

    Connecting look and feel: Associating the visual and tactile properties of physical materials

    Wenzhen Yuan, Shaoxiong Wang, Siyuan Dong, and Edward Adelson. Connecting look and feel: Associating the visual and tactile properties of physical materials. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5580–5588, 2017

    2017

  31. [39]

    Connecting touch and vision via cross-modal prediction

    Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10609–10618, 2019

    2019

  32. [40]

    Octopi: Object property reasoning with large tactile-language models.arXiv preprint arXiv:2405.02794, 2024

    Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile-language models.arXiv preprint arXiv:2405.02794, 2024

    2024 arXiv

  33. [41]

    A touch, vision, and language dataset for multimodal alignment.arXiv preprint arXiv:2402.13232, 2024

    Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg. A touch, vision, and language dataset for multimodal alignment.arXiv preprint arXiv:2402.13232, 2024

    2024 arXiv

  34. [42]

    Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation.arXiv preprint arXiv:2406.03813, 2024

    Ning Cheng, Changhao Guan, Jing Gao, Weihao Wang, You Li, Fandong Meng, Jie Zhou, Bin Fang, Jinan Xu, and Wenjuan Han. Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation.arXiv preprint arXiv:2406.03813, 2024

    2024 arXiv

  35. [43]

    The objectfolder benchmark: Multisensory learning with neural and real objects

    Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jeannette Bohg, Yunzhu Li, Li Fei-Fei, and Jiajun Wu. The objectfolder benchmark: Multisensory learning with neural and real objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1727...

    2023

  36. [44]

    Objectfolder 2.0: A multisensory object dataset for sim2real transfer

    Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages...

    2022

  37. [45]

    Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations.arXiv preprint arXiv:2109.07991, 2021

    Ruohan Gao, Yen-Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations.arXiv preprint arXiv:2109.07991, 2021. 12

    2021 arXiv

  38. [46]

    Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation.IEEE Robotics and Automation Letters, 5(3):3838–3845, 2020

    Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation.IEE...

    2020

  39. [47]

    Rotating without seeing: Towards in-hand dexterity through touch.arXiv preprint arXiv:2303.10880, 2023

    Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch.arXiv preprint arXiv:2303.10880, 2023

    2023 arXiv

  40. [48]

    General in-hand object rotation with vision and touch

    Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. InConference on Robot Learning, pages 2549–2564. PMLR, 2023

    2023

  41. [49]

    Active touch and robot perception.Cognition and Brain Theory, 7(2):199–214, 1984

    Kenneth Y Goldberg and Ruzena Bajcsy. Active touch and robot perception.Cognition and Brain Theory, 7(2):199–214, 1984

    1984

  42. [50]

    Revisiting active perception.Au- tonomous Robots, 42:177–196, 2018

    Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception.Au- tonomous Robots, 42:177–196, 2018

    2018

  43. [51]

    Simultaneous tactile exploration and grasp refinement for unknown objects.IEEE Robotics and Automation Letters, 6(2):3349–3356, 2021

    Cristiana De Farias, Naresh Marturi, Rustam Stolkin, and Yasemin Bekiroglu. Simultaneous tactile exploration and grasp refinement for unknown objects.IEEE Robotics and Automation Letters, 6(2):3349–3356, 2021

    2021

  44. [52]

    Active perception for tactile sensing: A task-agnostic attention-based approach, 2025

    Tim Schneider, Cristiana de Farias, Roberto Calandra, Liming Chen, and Jan Peters. Active perception for tactile sensing: A task-agnostic attention-based approach, 2025

    2025

  45. [53]

    Objectfolder 2.0: A multisensory object dataset for sim2real transfer

    Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In CVPR, 2022

    2022

  46. [54]

    Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors.IEEE Robotics and Automation Letters, 7(2):3930–3937, 2022

    Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors.IEEE Robotics and Automation Letters, 7(2):3930–3937, 2022

    2022

  47. [55]

    Taxim: An example-based simulation model for gelsight tactile sensors.IEEE Robotics and Automation Letters, 7(2):2361–2368, 2022

    Zilin Si and Wenzhen Yuan. Taxim: An example-based simulation model for gelsight tactile sensors.IEEE Robotics and Automation Letters, 7(2):2361–2368, 2022

    2022

  48. [56]

    JAX: composable transformations of Python+NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018

    2018

  49. [57]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  50. [58]

    Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan.IEEE Robotics and Automation Letters, 7(3):6187–6194, 2022

    Weihang Chen, Yuan Xu, Zhenyang Chen, Peiyu Zeng, Renjun Dang, Rui Chen, and Jing Xu. Bidirectional sim-to-real transfer for gelsight tactile sensors with cyclegan.IEEE Robotics and Automation Letters, 7(3):6187–6194, 2022

    2022

  51. [59]

    Rosenthal

    Cédric Beaulac and Jeffrey S. Rosenthal. Introducing a new high-resolution handwritten digits data set with writer characteristics.SN Computer Science, 4(1), November 2022

    2022

  52. [60]

    Homepage, January 2025

    Franka Inc. Homepage, January 2025. [Online; accessed 28. Jan. 2025]

    2025

  53. [61]

    Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications.arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  54. [62]

    Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605, 2019

    Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.arXiv preprint arXiv:1902.05605, 2019. 13

    1902 arXiv

  55. [63]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  56. [64]

    Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In2015 International Conference on Advanced Robotics (ICAR), pages 510–517, 2015

    2015

  57. [66]

    Flax: A neural network library and ecosystem for JAX, 2024

    Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2024

    2024

  58. [67]

    seq" data point contains fine-grained temporal and spatial information about the tactile interaction, useful for modeling contact dynamics or time-series prediction. The

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  59. [68]

    InitializationThe glimpse starts at a uniformly random position within the image

    Negative cross-entropy loss between prediction and target. InitializationThe glimpse starts at a uniformly random position within the image. TerminationThe episode ends with the terminate flag set if the step limit (16) is reached. Notation:K∈Nis the number of classes;G∈Nis th...

  60. [69]

    At each step, the sum of: 1.10 −3⋅∥a t∥(action regularization)

    The negative mean squared error between the agent’s prediction and its true position. At each step, the sum of: 1.10 −3⋅∥a t∥(action regularization)

  61. [70]

    A constant reward of 0.1 to ensure that the reward stays positive and the agent does not learn to terminate the episode on purpose

  62. [71]

    Initialization The agent begins at a uniformly random, valid location within the environment

    The negative mean squared error be- tween the agent’s prediction and its true position. Initialization The agent begins at a uniformly random, valid location within the environment. The agent’s initial position is uniformly ran- domly sampled from the range[−1,1] 2 Termination...

  63. [72]

    InitializationThe agent starts at a uniformly random position within the image

    The negative mean squared error between the agent’s prediction and the true coordinates of the target glimpse. InitializationThe agent starts at a uniformly random position within the image. TerminationThe episode ends if either the step limit is reached. Notation:G∈Nis the gl...

  64. [73]

    InitializationThe tactile sensor starts at a randomly sampled pose in the workspace

    Negative cross-entropy loss between prediction and target. InitializationThe tactile sensor starts at a randomly sampled pose in the workspace. TerminationThe episode ends with the terminate flag set if the step limit is reached. Notation:K∈Nis the number of classes. Implement...

  65. [74]

    At each step, the sum of: 1.10 −3⋅∥a t∥(action regularization)

    he loss of the current prediction of the agent. At each step, the sum of: 1.10 −3⋅∥a t∥(action regularization)

  66. [75]

    Initialization The glimpse starts at a uniformly random position within the workspace

    he loss of the current prediction of the agent. Initialization The glimpse starts at a uniformly random position within the workspace. The glimpse starts at a uniformly random po- sition within the workspace. Termination The episode ends with the terminate flag set if the step...

  67. [76]

    Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation.Journal of Artificial Intelligence Research, 74:1269–1349, 2022

    Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation.Journal of Artificial Intelligence Re...

    2022

  68. [77]

    Alex Church, John Lloyd, Raia Hadsell, and Nathan F. Lepora. Tactile sim-to-real policy transfer via real-to-sim image translation. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors,Proceedings of the 5th Conference on Robot Learning, volume 164 ofProceedings of Mac...

    2022

  69. [78]

    Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch.IEEE Robotics and Automation Letters, 7(4):10754–10761, 2022

    Yijiong Lin, John Lloyd, Alex Church, and Nathan F Lepora. Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch.IEEE Robotics and Automation Letters, 7(4):10754–10761, 2022

    2022

  70. [79]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. InComputer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, 2017. 39

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.