Pith. sign in

REVIEW 2 major objections 8 minor 52 references

ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

T0 review · 2 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single neural implicit field can estimate a grasped object's pose and where it touches the world from vision and touch, trained entirely in simulation.

desk verdict A genuine step beyond NCF/NDCF—joint pose and contact estimation without rigid grasps via neural shear fields—but the contact accuracy numbers are undermined by a self-referential ground truth. read the letter →

arxiv 2506.12239 v1 pith:QLIOGZJE submitted 2025-06-13 cs.RO cs.CV

Jayjun Lee , Nima Fazeli This is my paper · ORCID
classification cs.ROcs.CV
keywords visuo-tactileperceptionneuralimplicitrepresentationsigneddistancefieldextrinsiccontactestimationin-handposesim-to-realtransfertactileshearcontact-richmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that pose and contact can be read out from a single object-centered neural implicit field. The field fuses a partial visual point cloud with high-resolution tactile shear measurements, and it is trained entirely in simulation yet applied to a real robot with no fine-tuning. The result would matter because a manipulation system needs both the in-hand pose of a grasped tool and the location of any contact between that tool and the environment, and prior methods either assumed a rigid, known grasp or did not use high-resolution touch. The paper reports real-world mean translation error of 4.05 mm, rotation error of 0.91 deg, and contact-patch Chamfer distance of 0.092 m², and it argues that tactile shear fields transfer across sensor simulators better than tactile images.

What carries the argument

The machinery is the joint implicit representation built from three neural fields: an object SDF $s=O(q)$, a tactile shear field $[u,v]^T=T(g\mid\xi,\psi)$ defined on each sensor's 2D membrane, and a contact field $c=C(q\oplus z_O, T(g\mid\xi,\psi)\mid\xi,\psi)$, where $g$ is a point on the tactile grid, $\xi$ is a latent code for the in-hand pose, $\psi$ is a latent trial code for the contact interaction, and $z_O$ are intermediate activations of the object module. All three networks are conditioned through hypernetworks, networks that produce the weights of another network, and the shear field is the carrier of extrinsic contact information whose trial code is optimized at inference to explain observed shear. The object SDF is what lets pose estimation become inference-via-optimization: query points from visuo-tactile point clouds are transformed by the candidate pose, and the pose is adjusted until the SDF at those points is as close to zero as possible.

What would settle it

Set up a calibrated contact between a known object at a known pose and a rigid surface, record the real tactile sensor's shear direction field, and compare it with the simulator's prediction for the same configuration; a systematic mismatch in shear direction patterns would break trial-code inference and make the reported contact-patch Chamfer distances unrepresentative of real contact localization.

Watch

Extended reading notes

Core claim

On its own terms, ViTaSCOPE is the proposal that object geometry, in-hand SE(2) pose, and extrinsic contact geometry can be predicted by one implicit function rather than by separate pose trackers and contact detectors. The object module learns a signed distance field in the object's canonical frame, giving access to surface query points and normals; the tactile module learns, for each of two tactile sensors, a shear displacement field over the sensor membrane conditioned on a pose latent $\xi$ and a per-interaction trial code $\psi$; and the contact module reads out contact probability at surface points using object activations, pose, and shear. Nothing in the formulation ties the object to the gripper, so the pose is recovered by gradient descent on the SDF with respect to the world-frame transform of visuo-tactile query points, and the contact configuration is recovered by optimizing $\psi$ against observed shear. The paper's distinctive claim is that this pipeline, trained on simulation data from a penalty-based soft-contact tactile model with unit-normalized shear vectors, transfers zero-shot to a real sensor and yields the pose and contact accuracy quoted above.

Load-bearing premise

The whole contact-localization claim rests on two premises the paper does not directly measure: that unit-normalized shear fields from a penalty-based soft-contact simulator match real tactile sensor shear closely enough to transfer, and that contact patches obtained by SDF penetration thresholding are valid ground truth for training and evaluation; if either premise is wrong, the reported contact accuracy is not supported.

Editorial extensions

If this is right

  • A manipulation system could estimate a grasped tool's pose and its contact with the environment in one perception pipeline, without assuming the tool is rigidly fixed in the gripper.
  • Because the geometry and contact field are continuous functions, contact patches can be queried at any resolution on the object surface, not at the discretization of a tactile image or point cloud.
  • Tactile shear fields, rather than tactile RGB images, become the recommended representation for sim-to-real contact reasoning; swapping shear for RGB lowers average contact-patch accuracy in the paper's ablation.
  • Complementary fusion is concrete: vision alone or touch alone gives worse pose estimates than the combined model, and touch-only pose input degrades contact prediction as much as removing pose information entirely.
  • At roughly 0.3 Hz for the full two-stage inference, the approach is fast enough to be considered for closed-loop manipulation, not only offline perception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's contact metric is computed against labels derived from SDF penetration thresholding, a procedure the authors call less realistic than their predictions; independent contact ground truth, for example from a force-sensitive film, would be the decisive test of whether predicted patches are actually more accurate than the simulated labels.
  • If the simulated-to-real shear transfer holds for the tested tools, the same trial-code mechanism could plausibly be extended to dexterous multi-finger hands by giving each finger its own tactile module and concatenating its trial code, an extension the paper mentions as future work.
  • Because pose estimation and trial-code inference are sequential, pose error propagates into contact prediction; jointly optimizing pose and trial code in a single loop is a natural variant that the paper does not explore.
  • The known-mesh dependency at training time could be relaxed by treating the object module as an auto-decoder over many shapes and querying its latent code for a new object, a route the paper signals but does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper proposes ViTaSCOPE, a neural implicit representation that fuses partial visual point clouds and tactile shear fields to simultaneously estimate the in-hand pose of a grasped tool and register extrinsic contacts onto its 3D geometry. The method represents object geometry as a signed distance field, tactile feedback as a neural shear field, and contact likelihood as a per-surface-point field; all components are trained entirely in simulation with TacSL/Isaac Gym and applied zero-shot to a real Franka robot with GelSlim 4.0 sensors. The authors report mean real-world translation error of 4.05 mm, rotation error of 0.91 deg, and contact-patch Chamfer distance of 0.092 m^2, and they provide ablations on the contribution of tactile shear fields, object pose conditioning, and intermediate object activations.

Significance. If the contact-localization claim were supported by independent evaluation, this would be a valuable contribution: it removes the rigid-grasp assumption that limits prior neural contact-field methods, introduces shear fields as a sim-to-real-robust tactile representation, and demonstrates a plausible zero-shot transfer pipeline. The pose-estimation results are a genuine strength: the real-world errors are consistent with simulation, and the ablations show a clear benefit of combining vision and touch. The paper is also honest about several limitations (lack of normal forces, slippage, scaling to new objects). However, the central claim of accurate extrinsic contact registration is currently supported only by a self-referential evaluation, which is a load-bearing weakness. The contact-patch metrics in Tables I and III therefore do not, in their present form, establish the headline accuracy.

major comments (2)
  1. [IV-C, V-B] The extrinsic-contact evaluation is not independent of the training signal: the ground-truth contact patches used both for training the Contact Module and for evaluating the real-world results are generated by the same soft-contact SDF penetration-thresholding procedure in Isaac Gym, as the authors explicitly state in Section V-B. Because the model is trained to fit these labels, the Chamfer distances in Tables I and III measure agreement with the label-generation rule rather than with physical contact geometry. The authors even note that their predictions are "more realistic than the ground truth soft contact labels" (Section V-B and Fig. 6), which means the GT itself is acknowledged to be an imperfect surrogate. Consequently, the reported CD values (e.g., 0.092 m^2 real-world mean) do not support the claim that ViTaSCOPE accurately registers extrinsic contacts. I request an independent ground-truth check, for example using pressure-sensitive film, controlled indentations with known contact geometry, or a separate calibrated contact sensor, and a re-reporting of Tables I and III against that ground truth.
  2. [IV-C] The sim-to-real transfer of the shear-field representation is asserted but not directly validated. Training uses TacSL's penalty-based tactile model with GelSight R1.5 elastomer parameters, while real experiments use GelSlim 4.0 sensors; the shear vectors are unit-normalized above a threshold and noise is added, but no experiment compares the simulated shear field with the real sensor's shear output for a controlled contact. The strong pose results in Table II do not validate the shear model, since pose estimation uses point clouds, not shear fields. The only downstream evidence is the contact CD, which is compromised by the ground-truth issue above. Please provide a direct validation, such as comparing simulated and real shear fields for a known indentation or sliding motion, or an end-to-end experiment whose success depends on shear fidelity and is checked against an independent contact ground truth.
minor comments (8)
  1. [III-B] Equation (1) defines the unified function f as returning (s, c, phi), but Eq. (4) writes the SDF as s = O(q) with no reference to c or phi; please align the notation so the relationship between the object module and the full model is unambiguous.
  2. [IV-C] The sentence describing contact-label generation says the scene is replicated in Open3D and contacts are thresholded by object-table penetration; the threshold value and the criterion for selecting it should be specified for reproducibility.
  3. [V-A] The ICP baseline description is ambiguous about what inputs the 'Vision' and 'Tactile' columns in Table II represent; please clarify whether ICP has access to ground-truth geometry, and whether the initialization and point-cloud sources match those of ViTaSCOPE.
  4. [V-B] The statement that most rigid-body contacts are point or line while the ground-truth soft-contact labels are larger patches directly contradicts the use of those same labels as ground truth in Tables I and III; this tension should be acknowledged and discussed in the evaluation section.
  5. [IV-B] The initialization of the pose optimization is described both as an SE(2) projection of the end-effector pose and as the end-effector pose itself; please state clearly what the initial guess is and over what SE(3) (or SE(2)) domain Eq. (12) is optimized.
  6. [Fig. 6] The figure caption lists Chamfer distances but the legend for red (predicted) and green (ground truth) patches appears only in the text; please add an explicit in-figure legend and use a colorblind-safe palette.
  7. [IV-B] Table IV reports inference times without specifying the hardware; adding the GPU/CPU model would make the timing results reproducible.
  8. [III-B] There is a typo 'whicn' in the first paragraph of Section III-B, and 'propriceptive' appears in Section II-C; please correct these.

Circularity Check

1 steps flagged · score 6.0 of 10

Contact-patch evaluation is self-referential: the ground-truth patches are generated by the same soft-contact SDF penetration threshold rule used to create training labels.

  1. self definitional [Section IV-C (Data Generation, Simulation) and Section V-B (Extrinsic Contact Estimation), with results in Tables I and III and Fig. 6]
    "As Isaac Gym does not allow the users to access the contact locations directly, we replicate the scene in open3d and estimate the contact location by thresholding on object-table penetration, which effectively generates soft-contact points. ... We note that Isaac Gym does not provide contact locations or wrench and we use mesh and its poses to extract samples of contact points, which uses soft contact thresholding using SDF. Our model learns a better approximation to the contact patch that is closer to realistic contacts than the ground truth soft contact labels."

    The only source of contact-patch labels for training the Contact Module (loss Lcontact, Eq. 11) is the soft-contact SDF penetration thresholding procedure. The same procedure is used to generate the 'ground truth' patches against which the simulated and real-world contact Chamfer distances in Tables I and III (and Fig. 6) are computed. A model trained to reproduce this label rule will therefore score well even if the true rigid-body contact set is different. The authors concede the rule overestimates flat contacts (rectangle, cylinder) and that their predictions are 'more realistic' than the labels, so the metric penalizes physically correct predictions.

full rationale

The derivation chain for in-hand pose estimation is self-contained: the pose is obtained by inference-via-optimization against the pretrained SDF and validated against AprilTag poses, an external reference. Geometry reconstruction CD is also compared against the ground truth meshes. The circularity is confined to the extrinsic contact claim: both training supervision and evaluation ground truth are produced by the same soft-contact SDF penetration thresholding rule (Sections IV-C and V-B). Because the paper itself states that this ground truth is less realistic than the model's predictions, the contact CD cannot establish physical accuracy; it measures agreement with a surrogate label generator. This is a substantive partial circularity in one of the paper's two headline claims, though it is disclosed by the authors rather than hidden. Score 6 reflects that one of the central predictions reduces by construction to its own label-generation rule.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several modeling choices: known meshes for every object, faithful transfer of unit-normalized simulated shear fields to real GelSlim sensors, an SE(2) in-hand pose constraint with no slip, and ground-truth contact labels derived from soft-contact SDF penetration thresholding. The trained weights, latent codes, and loss weights are fitted or hand-chosen quantities; no new physical entities are postulated.

free parameters (5)
  • contact_threshold_epsilon = not reported
    Hand-chosen binary threshold in Eq. 3 that defines the contact patch from the predicted contact field; the paper never gives its value, and it directly controls the size and CD of predicted contact patches.
  • soft_contact_penetration_threshold = not reported
    Threshold used in open3d to generate ground-truth contact labels by detecting object-table penetration; it determines both training labels and real-world evaluation GT, so all reported contact CD numbers depend on it.
  • shear_normalization_threshold = 1e-8
    Vectors below this magnitude are ignored; remaining shear vectors are normalized to unit length before training and inference, discarding force magnitude as a modeling choice for sim-to-real transfer.
  • loss_weights_lambda = lambda_shear=0.1, lambda_emb=0.2, lambda_hyper=25.0, lambda_contact=2.0
    Hand-set weights in Eq. 8 balance shear, embedding, hypernetwork, and contact losses; together with lambda_normal=0.01 they determine the learned field, and no sensitivity analysis is given.
  • latent_code_sizes = pose xi=3, contact psi=12
    Dimensionalities of the per-instance latent codes optimized at inference; chosen by hand to capture SE(2) pose and left+right tactile trial codes, bounding the expressiveness of contact configurations.
assumptions (5)
  • domain assumption Object mesh models are known and provided for every object used in training and evaluation.
    Section IV-C: 'We assume access to the mesh models of these geometries for training purposes.' The method does not handle unseen objects without retraining (Section VI).
  • domain assumption Unit-normalized shear fields from TacSL penalty-based simulation transfer to real GelSlim 4.0 sensors.
    Section IV-C normalizes shear vectors to bridge the sim-to-real gap; no direct fidelity measurement is provided, only downstream contact CDs.
  • domain assumption In-hand object pose during contact is constrained to SE(2) (x, z, theta) with no slip.
    Section III-D parameterizes the pose and Eq. 12 optimizes over SE(2); Section VI lists 'In-Hand Slippage and Deformation' as not addressed. If the object rotates out of plane or slips, pose and contact estimates suffer.
  • domain assumption Ground-truth contact patches can be obtained by thresholding soft penetration between the object SDF and the table SDF.
    Section IV-C and V-B: contact locations are generated by 'thresholding on object-table penetration' in open3d; the authors state this yields less realistic patches than their predictions, undermining its validity as ground truth.
  • standard math Standard neural network training and gradient-based inference converge to useful optima.
    The method relies on Adam optimization for training, pose inference (Eq. 12), and trial code inference (Eq. 13) without convergence guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation." pith.science (2026). https://pith.science/paper/QLIOGZJE

@misc{pith2026250612239,
  author       = {Pith},
  title        = {Pith review of: ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLIOGZJE}},
  note         = {Machine review of arXiv:2506.12239}
}
abstract

Mastering dexterous, contact-rich object manipulation demands precise estimation of both in-hand object poses and external contact locations$\unicode{x2013}$tasks particularly challenging due to partial and noisy observations. We present ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation, an object-centric neural implicit representation that fuses vision and high-resolution tactile feedback. By representing objects as signed distance fields and distributed tactile feedback as neural shear fields, ViTaSCOPE accurately localizes objects and registers extrinsic contacts onto their 3D geometry as contact fields. Our method enables seamless reasoning over complementary visuo-tactile cues by leveraging simulation for scalable training and zero-shot transfers to the real-world by bridging the sim-to-real gap. We evaluate our method through comprehensive simulated and real-world experiments, demonstrating its capabilities in dexterous manipulation scenarios.

Figures

Figures reproduced from arXiv: 2506.12239 by the authors.

Figure 1
Figure 1. ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation. We present an object-centric neural implicit representation that enables simultaneous in-hand object pose and extrinsic contact estimation from vision and high￾resolution tactile sensing. Our method is trained entirely in simulation and zero-shot transferred to the real world. In the example above, the tool grasped by the robot makes extrinsic… view at source ↗
Figure 3
Figure 3. ViTaSCOPE training and inference. ViTaSCOPE is composed of three modules: Object module O, Tactile module T , and Contact module C. We represent the object geometry as a signed distance function (SDF) s using the object module. This representation first enables joint reasoning over visuo-tactile pointcloud observations to perform in-hand pose estimation using inference via optimization. The tactile module represents… view at source ↗
Figure 4
Figure 4. 3D Printed Tool Geometries and SDFs. We 3D print six geometries for evaluation. The tool tip making contact with the environment is facing upwards. The number and complex￾ity of possible extrinsic contact modes (i.e. point, line, patch) increases from left to right. Sample mesh reconstructions of learned SDF are shown in the bottom row. to generate data to train the three modules (O, T , C). Once trained, we generat… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Simulation Environment for Visuo-tactile Data Generation. A visualization of simulated environments in Issac Gym and sampled extrinsic contact interactions of six different tool geometries with TacSL tactile sensor simulation. We evaluate in both simulation (on a held-…
Figure 6
Figure 6. Figure 6: Extrinsic Contact Estimation. Results with varying Chamfer distances and different contact configurations for six tool geometries on the real world test data are shown. The predicted contact patches are represented with the red pointclouds and the ground truth contact …
Figure 7
Figure 7. Figure 7: Sample real-world contact prediction on Stanford bunny. Ablation Real World Contact Patch Chamfer Distance (CD) ↓ [m2 ] mount. rectangle pyramid hex cylinder semisphere mean vitascope 0.0394 0.2216 0.0565 0.0596 0.1716 0.0031 0.0920 wo acts 0.0872 0.3274 0.0992 0.0953 …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 25 canonical work pages

  1. [1]

    Iretiayo Akinola, Jie Xu, Jan Carius, Dieter Fox, and Yashraj S. Narang. TacSL: A Library for Visuotactile Sensor Simulation and Learning.CoRR, abs/2408.06506, 2024

    arXiv 2024

  2. [2]

    Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kam- merer, Dinesh Jayaraman, and Roberto Calandra

    Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian H. Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kam- merer, Dinesh Jayaraman, and Roberto Calandra. DIGIT: A Novel Design for a Low-Cost Compact High- Resolution Tactile Sensor With Application to In-Hand Manipulation.IEEE Robotics Autom. Lett., 5(3):3838– 3845,...

    arXiv 2020

  3. [3]

    Srinivasan, and Ed- ward H

    Rui Li, Robert Platt Jr., Wenzhen Yuan, Andreas ten Pas, Nathan Roscup, Mandayam A. Srinivasan, and Ed- ward H. Adelson. Localization and manipulation of small parts using GelSight tactile sensing. In2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2014, Chicago, IL, USA, Septem- ber 14-18, 2014, pages 3988–3993. IEEE, 2014. ...

    arXiv 2014

  4. [5]

    GelSlim 4.0: Focusing on Touch and Reproducibility

    Andrea Sipos, William van den Bogert, and Nima Fazeli. GelSlim 4.0: Focusing on Touch and Reproducibility. CoRR, abs/2409.19770, 2024. doi: 10.48550/ARXIV . 2409.19770

    arXiv doi:10.48550/arxiv.2409.19770 2024

  5. [6]

    Soft-bubble: A highly compliant dense geometry tactile sensor for robot ma- nipulation

    Alex Alspach, Kunimatsu Hashimoto, Naveen Kup- puswamy, and Russ Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor for robot ma- nipulation. InIEEE International Conference on Soft Robotics, RoboSoft 2019, Seoul, South Korea, April 14- 18, 2019, pages 597–604. IEEE, 2019. doi: 10.1109/ ROBOSOFT.2019.8722713

    work page arXiv 2019

  6. [7]

    Digitizing Touch with an Artificial Multimodal Fingertip.CoRR, abs/2411.02479, 2024

    Mike Lambeta, Tingfan Wu, Ali Sengul, Victoria Rose Most, Nolan Black, Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, Norb Ty- dingco, Gregg Kammerer, Dave Stroud, Jake Khatha, Kurt Jenkins, Kyle Most, Neal Stein, Ricardo Chavira, Thomas Craven-Bartle, Eric Sanchez, Yitian Ding, Jitendra Malik, and Roberto Calandra. Digitizing Touch...

    arXiv doi:10.48550/arxiv.2411.02479 2024

  7. [8]

    TACTO: A Fast, Flexible, and Open- Source Simulator for High-Resolution Vision-Based Tac- tile Sensors.IEEE Robotics Autom

    Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, and Roberto Calandra. TACTO: A Fast, Flexible, and Open- Source Simulator for High-Resolution Vision-Based Tac- tile Sensors.IEEE Robotics Autom. Lett., 7(2):3930– 3937, 2022. doi: 10.1109/LRA.2022.3146945

    arXiv 2022

  8. [9]

    Efficient Tactile Simulation with Differentiability for Robotic Manipulation

    Jie Xu, Sangwoon Kim, Tao Chen, Alberto Rodriguez Garcia, Pulkit Agrawal, Wojciech Matusik, and Shinjiro Sueda. Efficient Tactile Simulation with Differentiability for Robotic Manipulation. InConference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 ofProceedings of Machine Learning Research, pages 1488–1498. PMLR, 2022

    work page 2022

Show all 52 references
  1. [10]

    Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. InProceedings of the Neural I...

    2021

  2. [11]

    Taxim: An Example-Based Simulation Model for GelSight Tactile Sensors.IEEE Robotics Autom

    Zilin Si and Wenzhen Yuan. Taxim: An Example-Based Simulation Model for GelSight Tactile Sensors.IEEE Robotics Autom. Lett., 7(2):2361–2368, 2022. doi: 10. 1109/LRA.2022.3142412

    2022

  3. [12]

    DIFFTACTILE: A Physics- based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

    Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Chao Liu, and Chuang Gan. DIFFTACTILE: A Physics- based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May ...

    2024

  4. [13]

    The Power of the Senses: Gen- eralizable Manipulation from Vision and Touch through Masked Multimodal Learning

    Carmelo Sferrazza, Younggyo Seo, Hao Liu, Youngwoon Lee, and Pieter Abbeel. The Power of the Senses: Gen- eralizable Manipulation from Vision and Touch through Masked Multimodal Learning. InIEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, IROS 2024, Abu D...

    2024

  5. [14]

    Mu- JoCo: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mu- JoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109

    2012

  6. [15]

    Florence, Julian Straub, Richard A

    Jeong Joon Park, Peter R. Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-...

    2019

  7. [16]

    VIRDO: Visio-tactile Implicit Representations of Deformable Objects

    Youngsun Wi, Pete Florence, Andy Zeng, and Nima Fazeli. VIRDO: Visio-tactile Implicit Representations of Deformable Objects. In2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 3583–3590. IEEE,

    2022

  8. [17]

    VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects

    Youngsun Wi, Andy Zeng, Pete Florence, and Nima Fazeli. VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects. InConference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 ofProceedings of Machine Learning Researc...

    2022

  9. [18]

    Van der Merwe, Youngsun Wi, Dmitry Berenson, and Nima Fazeli

    Mark J. Van der Merwe, Youngsun Wi, Dmitry Berenson, and Nima Fazeli. Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback. InRobotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: 10.15607/RSS. 2023.XIX.080

    2023 doi

  10. [19]

    Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing

    Carolina Higuera, Siyuan Dong, Byron Boots, and Mustafa Mukadam. Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing. InIEEE Inter- national Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, pages 12576– 12582. IEEE, 2023. ...

    2023

  11. [20]

    Neural Inverse Source Problem

    Youngsun Wi, Jayjun Lee, Miquel Oller, and Nima Fazeli. Neural Inverse Source Problem. InProceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 4371–

  12. [21]

    AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction

    Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I, volume 13661 ofLecture...

    2022 doi

  13. [22]

    FoundationPose: Unified 6D Pose Estimation and Track- ing of Novel Objects

    Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6D Pose Estimation and Track- ing of Novel Objects. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 17868–17879. IEEE, 2024. doi:...

    2024

  14. [23]

    What’s in your hands? 3D Reconstruction of Generic Objects in Hands

    Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What’s in your hands? 3D Reconstruction of Generic Objects in Hands. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 3885–3895. IEEE, 2022. doi: 10.1109/CV...

    2022

  15. [24]

    Tac2Pose: Tactile object pose estimation from the first touch.Int

    Maria Bauz ´a, Antonia Bronars, and Alberto Rodriguez. Tac2Pose: Tactile object pose estimation from the first touch.Int. J. Robotics Res., 42(13):1185–1209, 2023. doi: 10.1177/02783649231196925

    2023 doi

  16. [25]

    Castro, Calder Phillips-Grafflin, Alex Alspach, and Russ Tedrake

    Naveen Kuppuswamy, Alejandro M. Castro, Calder Phillips-Grafflin, Alex Alspach, and Russ Tedrake. Fast Model-Based Contact Patch and Pose Estimation for Highly Deformable Dense-Geometry Tactile Sensors. IEEE Robotics Autom. Lett., 5(2):1811–1818, 2020. doi: 10.1109/LRA.2019.2961050

    2020

  17. [26]

    Adelson, and Russ Tedrake

    Gregory Izatt, Geronimo Mirano, Edward H. Adelson, and Russ Tedrake. Tracking objects with point clouds from vision and touch. In2017 IEEE International Con- ference on Robotics and Automation, ICRA 2017, Singa- pore, Singapore, May 29 - June 3, 2017, pages 4000–

    2017

  18. [27]

    Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal

    Edward J. Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal. 3D Shape Reconstruction from Vi- sion and Touch. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Sy...

    2020

  19. [28]

    Smith, David Meger, Luis Pineda Roberto Calandra, Jitendra Malik, Adriana Romero-Soriano, and Michal Drozdzal

    Edward J. Smith, David Meger, Luis Pineda Roberto Calandra, Jitendra Malik, Adriana Romero-Soriano, and Michal Drozdzal. Active 3D Shape Reconstruction from Vision and Touch. InAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Proces...

    2021

  20. [29]

    ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation With Shape Completion.IEEE Robotics Autom

    Hongyu Li, Snehal Dikhale, Soshi Iba, and Nawid Ja- mali. ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation With Shape Completion.IEEE Robotics Autom. Lett., 8(11):6963–6970, 2023. doi: 10.1109/LRA. 2023.3313941

    2023

  21. [30]

    VisuoTactile 6D Pose Estimation of an In- Hand Object Using Vision and Tactile Sensor Data

    Snehal Dikhale, Karankumar Patel, Daksh Dhingra, Itoshi Naramura, Akinobu Hayashi, Soshi Iba, and Nawid Jamali. VisuoTactile 6D Pose Estimation of an In- Hand Object Using Vision and Tactile Sensor Data. IEEE Robotics Autom. Lett., 7(2):2148–2155, 2022. doi: 10.1109/LRA.2022.3143289

    2022

  22. [31]

    Simultaneous Contact Location and Object Pose Estimation Using Proprio- ception and Tactile Feedback

    Andrea Sipos and Nima Fazeli. Simultaneous Contact Location and Object Pose Estimation Using Proprio- ception and Tactile Feedback. InIEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan, October 23-27, 2022, pages 3233–3240. IEEE, 202...

    2022

  23. [32]

    MultiSCOPE: Disam- biguating In-Hand Object Poses with Proprioception and Tactile Feedback

    Andrea Sipos and Nima Fazeli. MultiSCOPE: Disam- biguating In-Hand Object Poses with Proprioception and Tactile Feedback. InRobotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. doi: 10.15607/RSS.2023.XIX.078

    2023 doi

  24. [33]

    Localizing external contact using proprioceptive sensors: The Contact Parti- cle Filter

    Lucas Manuelli and Russ Tedrake. Localizing external contact using proprioceptive sensors: The Contact Parti- cle Filter. In2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016, Daejeon, South Korea, October 9-14, 2016, pages 5062–5069. IEEE, 201...

    2016

  25. [34]

    Im2Contact: Vision-Based Contact Local- ization Without Touch or Force Sensing

    Leon Kim, Yunshuang Li, Michael Posa, and Dinesh Jayaraman. Im2Contact: Vision-Based Contact Local- ization Without Touch or Force Sensing. InConference on Robot Learning, CoRL 2023, 6-9 November 2023, At- lanta, GA, USA, volume 229 ofProceedings of Machine Learning Research, ...

    2023

  26. [35]

    Visual-auditory Extrinsic Contact Estimation.CoRR, abs/2409.14608,

    Xili Yi, Jayjun Lee, and Nima Fazeli. Visual-auditory Extrinsic Contact Estimation.CoRR, abs/2409.14608,

  27. [36]

    Perceiving Ex- trinsic Contacts from Touch Improves Learning Insertion Policies.CoRR, abs/2309.16652, 2023

    Carolina Higuera, Joseph Ortiz, Haozhi Qi, Luis Pineda, Byron Boots, and Mustafa Mukadam. Perceiving Ex- trinsic Contacts from Touch Improves Learning Insertion Policies.CoRR, abs/2309.16652, 2023. doi: 10.48550/ ARXIV .2309.16652

    2023 doi arXiv

  28. [37]

    Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control

    Miquel Oller, Mireia Planas, Dmitry Berenson, and Nima Fazeli. Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control. InConference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Mac...

    2022

  29. [38]

    Tactile-Driven Non-Prehensile Object Manipulation via Extrinsic Contact Mode Control

    Miquel Oller, Dmitry Berenson, and Nima Fazeli. Tactile-Driven Non-Prehensile Object Manipulation via Extrinsic Contact Mode Control. InProceedings of Robotics: Science and Systems, Delft, Netherlands, July

  30. [39]

    Extrin- sic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements

    Daolin Ma, Siyuan Dong, and Alberto Rodriguez. Extrin- sic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements. InIEEE International Conference on Robotics and Automation, ICRA 2021, Xi’an, China, May 30 - June 5, 2021, pages 11262– 11268. IEEE,...

    2021

  31. [40]

    Tactile-Based Insertion for Dense Box-Packing

    Siyuan Dong and Alberto Rodriguez. Tactile-Based Insertion for Dense Box-Packing. In2019 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems, IROS 2019, Macau, SAR, China, November 3- 8, 2019, pages 7953–7960. IEEE, 2019. doi: 10.1109/ IROS40897.2019.8968204

    2019

  32. [41]

    Dense Tactile Force Estimation using Gel- Slim and inverse FEM

    Daolin Ma, Elliott Donlon, Siyuan Dong, and Alberto Rodriguez. Dense Tactile Force Estimation using Gel- Slim and inverse FEM. InInternational Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages 5418–5424. IEEE,

    2019

  33. [42]

    Jha, Diego Romeres, Parag Patre, and Alberto Rodriguez

    Sangwoon Kim, Devesh K. Jha, Diego Romeres, Parag Patre, and Alberto Rodriguez. Simultaneous Tactile Estimation and Control of Extrinsic Contact. InIEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023, pages 12563–12569. IEEE, ...

    2023

  34. [43]

    doi: 10.15607/RSS.2024.XX.135

    2024 doi

  35. [44]

    Lorensen and Harvey E

    William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3D surface construction al- gorithm. In Maureen C. Stone, editor,Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1987, Anaheim, Cal- ifornia, USA, Jul...

    1987

  36. [45]

    Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks.CoRR, abs/2401.14159, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks.CoRR,...

    2024 doi arXiv

  37. [46]

    Besl and Neil D

    P.J. Besl and Neil D. McKay. A method for registration of 3-D shapes.IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239–256, 1992. doi: 10. 1109/34.121791. APPENDIXA DIMENSIONS OFOBJECTGEOMETRIES The dimensions of the objects in Fig. 4, measured from the...

    1992

  38. [49]

    TEXterity: Tactile Extrinsic deXter- ity

    Antonia Bronars, Sangwoon Kim, and Parag Patre Al- berto Rodriguez. TEXterity: Tactile Extrinsic deXter- ity. InIEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, pages 7976–7983. IEEE, 2024. doi: 10.1109/ ICRA57147.2024.10610622

    2024

  39. [1987]

    doi: 10.1145/37401.37422

  40. [2019]

    doi: 10.1109/ICRA.2019.8794113

    2019

  41. [2022]

    doi: 10.1109/ICRA46639.2022.9812097

    2022

  42. [2024]

    doi: 10.48550/ARXIV .2409.14608

    doi

  43. [4007]

    doi: 10.1109/ICRA.2017.7989460

    IEEE, 2017. doi: 10.1109/ICRA.2017.7989460

    2017

  44. [4391]

    PMLR, 06–09 Nov 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.