Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Contrastive pretraining against contact-state text and rendered images turns tactile 3D point clouds into an encoder that reads contact states zero-shot and transfers from simulation to real sensors.

desk verdict Solid dataset and pretraining method, but the evaluation stays inside the pretraining ontology, so the 'understanding' claim is narrower than advertised. read the letter →

arxiv 2505.08194 v1 pith:Q4QCS4UV submitted 2025-05-13 cs.RO

Wenxuan Ma , Xiaoge Cao , Yixiang Zhang , Chaofan Zhang , Shaobo Yang , Peng Hao , Bin Fang , Yinghao Cai
show 2 more authors
Shaowei Cui Shuo Wang
This is my paper · ORCID
classification cs.RO
keywords tactile-languagerepresentationpre-trainingtactilesensingmultimodalrobotlearningcontactstateunderstandingcontrastive3Dpointcloudsim-to-realtransferlargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a robot's sense of touch can be taught to understand contact geometry — where an object touches, with what shape, how deeply, and over what area — by aligning 3D tactile point clouds with natural-language descriptions of those contact states. To do this it builds TCL3D, a dataset of 52,425 point-cloud/text/image triples spanning 117 objects in TACTO simulation and on two real visuotactile sensors, and CLTP, a contrastive pretraining method that projects tactile point clouds into a frozen, pre-aligned vision-language (CLIP) space using both templated text and rendered contact images. The reported payoff is a single tactile encoder that classifies contact states without training labels, transfers from simulation to real hardware, and can be attached to a large language model for contact description, tactile question-answering, and closed-loop grasp refinement. If this is right, contact-rich manipulation gains a language-grounded touch representation — the input that tactile-language-action models have been missing.

What carries the argument

The carrying device is the frozen, pre-aligned CLIP embedding space: a tactile 3D point-cloud encoder is trained so its features land where CLIP already puts text and images, with two contrastive losses, the tactile-language loss $\mathcal{L}_{T2L}$ and the tactile-image loss $\mathcal{L}_{T2I}$, minimized together. The language side is a template sentence built from sampling metadata — 'a [Texture] [Shape] object, pressed [Depth] in [Position] with [Area] contact area' — which converts the discrete label ontology into sentences a frozen text encoder can embed. The image side is a rendered contact image produced by meshing the deformed 3D point cloud and rendering it from the sensor's viewpoint, which carries shape and texture detail too fine for the coarse text; the paper credits this branch for the largest gains (zero-shot shape accuracy rises from 52.6% to 70.1% when it is added).

What would settle it

Take contact states that fall outside or between the ontology's bins — pressing depths sampled continuously between 'slight' and 'moderate', shapes such as a cone or torus that are not among the 19 classes, and a third sensor type never seen in pretraining — and run zero-shot classification on them. If the encoder has learned contact geometry, accuracy should degrade gracefully; an accuracy cliff at bin boundaries or on unseen sensors would show the alignment captured the label grid, not the contact states themselves.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that contact states, not just surface texture, can serve as the unit of tactile-language alignment, and that a frozen CLIP space is a sufficient bridge to achieve it. The authors introduce TCL3D, with 52,425 triples of tactile 3D point cloud, rendered contact image, and language description spanning five contact-state dimensions (19 shapes, 5 textures, 4 depths, 9 positions, 5 areas), then train a point-cloud encoder with two contrastive losses — tactile-to-language and tactile-to-image — whose sum it minimizes against frozen CLIP encoders. They report that CLTP beats the point-cloud baselines Point-BERT and Point-MAE, and the tactile-image models UniTouch and TVL, on all five attributes in both synthetic and real data, with contact-shape accuracy on the synthetic set at 84.8% versus 28.7% and 31.6% for the baselines. The same aligned features support zero-shot per-attribute classification and a tactile LLM (Tac3D-LLM) that describes touch in words, answers reasoning questions, and guides a closed-loop strawberry-grasp refinement.

Load-bearing premise

The load-bearing premise is that the hand-made ontology — 19 shapes, 5 textures, 4 depths, 9 positions, 5 areas — together with the template sentences generated from sampling metadata, fully captures the contact states a manipulation system needs; if the categories are too coarse, every downstream result measures the label grid rather than general contact understanding.

Editorial extensions

If this is right

  • Zero-shot classification works per attribute: with no training labels, text prompts alone identify shape, texture, depth, position, and area from a tactile point cloud (Table 1).
  • Sim-to-real transfer holds: an encoder pretrained almost entirely on TACTO simulation reaches 71.2% shape accuracy on real GelStereo and GelSight contacts, far above the point-cloud baselines (Table 2).
  • The aligned space makes tactile-LLM interaction possible: Tac3D-LLM describes contacts in words, answers which object a touch belongs to, and outputs closed-loop grasp actions like 'move down' and 'decrease force' in a strawberry grasping experiment.
  • The rendered-image branch is load-bearing: removing it drops zero-shot shape accuracy from 70.1% to 52.6% and supervised shape accuracy from 84.8% to 61.2%, showing text alone cannot carry contact geometry.
  • A language-grounded contact-state encoder is positioned as the missing tactile input for vision-language-action models in contact-rich manipulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the rendered contact image is generated deterministically from the same 3D mesh as the point cloud, the image branch's large gains may partly come from a second, stronger encoding of identical geometry rather than genuinely new information; ablating the image branch against a larger point-cloud backbone would separate the two.
  • The 4 depth bins, 5 area bins, and 19 shape classes make the representation categorical by construction, and the authors' own limitation note concedes the model 'lacks accurate modeling of numbers'; a natural next test is whether CLTP captures continuous force or position gradations at all.
  • The single template sentence means the text encoder has only ever seen one syntactic pattern, so zero-shot prompts outside it (such as 'am I squeezing too hard?') may not align; extending the template family is a cheap, testable scaling axis.
  • If the sensor-agnostic claim holds, the same pretrained encoder should attach to a third sensor type — for example a resistive sensor — with no more than scale normalization, an experiment the paper's two-sensor evaluation does not yet perform.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CLTP, a contrastive pretraining framework that aligns tactile 3D point clouds with language and rendered contact images in a frozen CLIP feature space, and introduces TCL3D, a dataset of 52,425 tactile-language pairs covering 117 objects across simulated and real sensors. Contact states are described along five hand-defined dimensions (19 shapes, 5 textures, 4 depths, 9 positions, 5 areas), with text generated from sampling metadata and rendered images derived from the same contact point clouds. Downstream evaluations cover zero-shot per-attribute classification, supervised multi-task classification with frozen features, and a tactile LLM interaction module (Tac3D-LLM). The paper reports consistent gains over Point-BERT and Point-MAE on synthetic and real-world data, and substantially better performance when the image alignment loss is included.

Significance. If the claims hold, the paper makes a useful contribution: it is among the first to move tactile-language alignment from material/texture descriptors to contact-state descriptions, it provides an open dataset large enough for contrastive pretraining, and it demonstrates a plausible path toward tactile-LLM reasoning for manipulation. The architectural choice of anchoring tactile embeddings to a frozen CLIP space is sound and reproducible in principle. The dataset and code are open-sourced, which strengthens the paper's potential impact. However, the significance is tempered by the fact that the evaluation measures recognition within the same fixed taxonomy used to generate the pretraining text; the claimed 'contact-state-aware understanding' is therefore only demonstrated for the predefined ontology, not for open-ended contact understanding. The real-world evidence is thin, and the zero-shot evaluation lacks external baselines.

major comments (4)
  1. [Section 3.2, Section 5.1, Section 5.2, Section 7] The evaluation reuses the pretraining label ontology. The downstream labels, the zero-shot prompts (e.g., 'This is a [Shape]', 'Contact at [Position]'), and the pretraining templates in Section 3.2 all draw from the same 19/5/4/9/5 taxonomy of shapes, textures, depths, positions, and areas. Since every category word and its combinations were seen during contrastive pretraining, the reported accuracies demonstrate that the tactile encoder can discriminate the fixed ontology from 3D geometry, but they do not demonstrate open-ended contact-state understanding of the kind implied by the abstract and by the Tac3D-LLM examples. This concern is reinforced by the paper's own Section 7, which concedes that the model lacks accurate quantitative modeling and works only at a qualitative level. The authors should either narrow the claims to ontology recognition or add evaluations on contact states that fall outside the fixed taxonomy, such as unseen objects, novel attribute combinations, or continuous regression of position/depth/area.
  2. [Section 5.2, Table 2] The real-world evaluation rests on very small test sets (1,450 GelStereo and 115 GelSight samples), yet Table 2 reports single accuracy numbers without error bars, confidence intervals, or significance tests. The sim-to-real claim is load-bearing for the paper, and the reported margins (e.g., 71.2% vs. 23.9% for real-world shape classification) are plausible but could be affected by random train/test splits given the small sample sizes. The authors should report mean and standard deviation across at least three random seeds/splits, and ideally per-class breakdowns, for the real-world results.
  3. [Section 5.1, Table 1] The zero-shot classification experiments compare only CLTP against its own variant without the image loss; there are no external baselines such as ULIP-2, PointCLIP, TVL-Link, UniTouch, or AnyTouch. Since the paper claims superiority and 'emergent alignment', the zero-shot results need to be contextualized against at least one existing point-cloud-language or tactile-language baseline. Without such a comparison, the zero-shot numbers in Table 1 cannot be interpreted as evidence that the proposed pretraining is better than existing alignment methods.
  4. [Appendix C, Table 4] The Tac3D-LLM quantitative evaluation contains a naming inconsistency: the caption says the test cases are from 'TVL3D dataset', but the paper's dataset is TCL3D. If the evaluation was in fact performed on TVL (the prior tactile-visual-language dataset), this needs to be stated precisely; if it was performed on TCL3D, the text should be corrected. In addition, the GPT-4 rating is computed against reference captions that are generated from the same metadata templates as the pretraining text, so high scores may reflect template matching rather than semantic breadth. The authors should clarify the dataset and ideally report human agreement or win-rate against a template-following baseline.
minor comments (5)
  1. [Section 4] There are typos in the method section: 'emebddings' and 'lanuage' should be corrected, and the equations (1)-(3) could use a short paragraph explaining the role of the learnable temperature parameter tau.
  2. [Section 3.2, Figure 2] The caption of Figure 2 lists 'Contact Force (4)' while the text in Section 3.2 says '4 depth categories (e.g., slight, moderate, deep)'. The terminology 'force' vs 'depth' should be made consistent throughout the paper, since Table 1 and Table 2 use 'Depth' as the column name.
  3. [Section 5.1] The sentence 'Our zero-shot method shows comparable performance' is vague; the authors should state to what it is comparable and what the takeaway is for the reader.
  4. [Appendix A, Table 3] The names 'Unitouch' and 'TVL' are inconsistent with the capitalized forms 'UniTouch' and 'TVL' used in the main text; please unify the spelling.
  5. [Section 5.3, Figure 7] In Figure 7, the input-reference pairs appear to include text outputs with typos such as 'midium contact area'; if this is model output, it should be noted, and if it is a typo in the figure, it should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

Zero-shot and classification evaluations reuse the exact pretraining label ontology, so the claimed contact-state-aware language understanding is only tested within the model's own closed-set supervision.

  1. fitted input called prediction [Section 3.2 (Natural Language Description Generation) and Section 5.1 (Zero-shot 3D Touch Classification)]
    "We leverage the metadata acquired from contact sampling to generate text descriptions. The metadata includes a series of words, each identifying a contact state of the contact (e.g., shape, position, force). We adopt a simple prompt to construct meaningful sentences that are used during pretraining, such as: ’a [Texture] [Shape] object, pressed [Depth] in [Position] with [Area] contact area.’ ..."

    The pretraining text supervision is generated by filling the very same category vocabulary that defines the downstream evaluation: 19 shapes, 5 textures, 4 depths, 9 positions, and 5 areas. The zero-shot prompts such as “This is a [Shape]” and “Contact at [Position]” are not open-vocabulary queries; they are paraphrases of the template sentences used in the contrastive loss of Eq. (1). The “prediction” therefore measures whether the tactile encoder recognizes the fixed taxonomy it was trained to associate with 3D geometry, not whether it has learned contact-state understanding beyond that closed set.

full rationale

The pretraining objective itself is a standard contrastive loss (Eqs. 1–3) and no learned parameter is fitted to a held-out subset and then renamed a prediction. However, the headline evidence for contact-state understanding is circular in an evaluation sense: both the pretraining text descriptions and the downstream zero-shot prompts are constructed from the same hand-defined 19/5/4/9/5 ontology described in Section 3.2, so zero-shot classification and the linear-probe evaluation reproduce the exact label space used as supervision. The sim-to-real results on GelStereo and GelSight provide some independent signal, but they still use the same predefined categories, so they do not test generalization outside the training taxonomy. The Tac3D-LLM evaluation in Appendix C is rated by GPT-4 against a reference and contains a TVL3D/TCL3D naming inconsistency, so it does not independently break the closed-set loop. Score 6 reflects partial circularity: mapping 3D geometry to the fixed categories is nontrivial and cross-sensor transfer is real, but the central claim of open-ended contact-state-aware language understanding is only demonstrated within the model’s own label ontology.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are postulated; TCL3D and Tac3D-LLM are datasets and models, not invented entities. The central claim rests on the hand-defined label ontology, the frozen CLIP anchor space, the rendered image bridge, and the sim2real comparability assumption.

free parameters (3)
  • Learnable contrastive temperature tau = learned, not reported
    Appears in both contrastive losses (Eqs. 1 and 2); standard CLIP-style hyperparameter, but its final value is not reported, which affects reproducibility.
  • Point sampling count N_p = 1024
    Uniformly sampled points per tactile point cloud for training and testing (Section 5, Implementations); chosen by hand and may affect performance.
  • Contact-state label ontology sizes = 19 shapes / 5 textures / 4 depths / 9 positions / 5 areas
    Hand-defined category counts in Section 3.2 determine the language supervision and all evaluation metrics; no justification is given that these bins are optimal or sufficient.
assumptions (4)
  • domain assumption Frozen CLIP image and text feature spaces are semantically aligned with tactile contact descriptions.
    The entire alignment strategy assumes CLIP language embeddings adequately represent phrases such as 'pressed moderately at the bottom left'; no independent verification is provided (Section 4).
  • domain assumption The rendered 2D image generated from the 3D tactile point cloud preserves contact-state semantics needed for alignment.
    Images are synthesized by converting 3D tactile data into a mesh and rendering an RGB observation (Section 3.1); if rendering loses contact geometry or adds sensor-specific artifacts, image alignment can mislead the tactile encoder.
  • ad hoc to paper The hand-defined qualitative contact-state categories fully capture the contact information relevant to manipulation.
    The five dimensions and their discrete bins are chosen by the authors in Section 3.2; the Limitations section admits the model lacks accurate numeric modeling and is only qualitative.
  • domain assumption Simulated TACTO data and real GelStereo and GelSight data share enough distribution after scale normalization.
    The paper normalizes sensor scales to a unified spatial range (Section 5, Implementations), but real data amounts to only 1,450 GelStereo and 115 GelSight samples, making this assumption hard to validate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding." pith.science (2026). https://pith.science/paper/Q4QCS4UV

@misc{pith2026250508194,
  author       = {Pith},
  title        = {Pith review of: CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4QCS4UV}},
  note         = {Machine review of arXiv:2505.08194}
}
read the original abstract

Recent advancements in integrating tactile sensing with vision-language models (VLMs) have demonstrated remarkable potential for robotic multimodal perception. However, existing tactile descriptions remain limited to superficial attributes like texture, neglecting critical contact states essential for robotic manipulation. To bridge this gap, we propose CLTP, an intuitive and effective language tactile pretraining framework that aligns tactile 3D point clouds with natural language in various contact scenarios, thus enabling contact-state-aware tactile language understanding for contact-rich manipulation tasks. We first collect a novel dataset of 50k+ tactile 3D point cloud-language pairs, where descriptions explicitly capture multidimensional contact states (e.g., contact location, shape, and force) from the tactile sensor's perspective. CLTP leverages a pre-aligned and frozen vision-language feature space to bridge holistic textual and tactile modalities. Experiments validate its superiority in three downstream tasks: zero-shot 3D classification, contact state classification, and tactile 3D large language model (LLM) interaction. To the best of our knowledge, this is the first study to align tactile and language representations from the contact state perspective for manipulation tasks, providing great potential for tactile-language-action model learning. Code and datasets are open-sourced at https://sites.google.com/view/cltp/.

Figures

Figures reproduced from arXiv: 2505.08194 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. TCL3D dataset: a multimodal multi-sensor tactile alignment dataset for CLTP training and evaluation. 3.1 Data Sampling In order to obtain tactile point cloud data at the lowest cost, we selected the TACTO tactile simulator[32] with strong real-time performance combined with Pybullet[53] for main body sim￾ulation data acquisition, and used real GelStereo[25] and GelSight Mini[29, 54] sensors to verify real data acqui… view at source ↗
Figure 3
Figure 3. Architecture of CLTP framework. It leverages a frozen pre-aligned vision-language feature space to establish connections between tactile and language modalities. 4 Method We aim to learn a unified tactile representation for contact-rich manipulation tasks that captures critical contact states across touch and other modalities, such as language and vision. First, we conduct language-tactile pretraining, utilizing 3D … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Case Study: CLTP is more sensitive to the shape, pressing depth, and contact position of the contact 3D point cloud than the baseline models. 5 Experiments We conduct a series of experiments to answer the following key research questions: • Compared with the existing p…
Figure 5
Figure 5. Figure 5: A case study of generating text descriptions from realistic tactile point clouds. Shape: Triangular Shape: Peak Contact Position: Center Contact Position: Bottom Right Model Prediction Ground Truth Case 0 Reference Input Case 1 Reference Input [PITH_FULL_IMAGE:figures…
Figure 6
Figure 6. Figure 6: Analysis of failed cases of CLTP text description generation for tactile point clouds. Case 1: shape description ambiguity; Case 2: large plane contact leads to description noise. Methods TCL3D Dataset Real World Data Shape Texture Depth Position Area Shape Texture Dep…
Figure 7
Figure 7. Figure 7: Tac3D-LLM. Our Tac3D-LLM is capable of performing a range of tactile question￾answering tasks, such as contact description and multi-modal reasoning. We compare our model with two baselines with different tactile encoders: PointMAE-LLM and PointBERT-LLM. We find that t…
Figure 8
Figure 8. Figure 8: Tac3D-LLM Application. Firstly, the robotic arm predicts and executes a grasp. Tac3D￾LLM then processes 3D tactile data from sensors, along with task instructions and goals, to assess the contact state and determine appropriate actions. For instance, at T=1, the sensor…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tactile Modality Fusion for Vision-Language-Action Models

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A FiLM-based tactile fusion method that conditions VLA visual features on frozen pretrained touch embeddings improves real-robot insertion success, speed, and force control relative to vision-only and concatenation baselines.

Reference graph

Works this paper leans on

60 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    R. S. Dahiya, G. Metta, M. Valle, and G. Sandini. Tactile sensing—from humans to humanoids. IEEE transactions on robotics, 26(1):1–20, 2009

    work page 2009

  2. [2]

    Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, and H. J. Ritter. A review of tactile informa- tion: Perception and action through touch. IEEE Transactions on Robotics, 36(6):1619–1634, 2020

    work page 2020

  3. [3]

    Billard and D

    A. Billard and D. Kragic. Trends and challenges in robot manipulation. Science, 364(6446): eaat8414, 2019

    2019

  4. [4]

    J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta, and K. J. Kuchenbecker. Human-inspired robotic grasp control with tactile sensing. IEEE Transactions on Robotics , 27(6):1067–1079, 2011

    work page 2011

  5. [5]

    K. Dai, X. Wang, A. M. Rojas, E. Harber, Y . Tian, N. Paiva, J. Gnehm, E. Schindewolf, H. Choset, V . A. Webster-Wood, et al. Design of a biomimetic tactile sensor for material classification. In 2022 International Conference on Robotics and Automation (ICRA) , pages 10774–10780. IEEE, 2022

    work page 2022

  6. [6]

    S. Cui, R. Wang, J. Hu, J. Wei, S. Wang, and Z. Lou. In-hand object localization using a novel high-resolution visuotactile sensor. IEEE Transactions on Industrial Electronics, 69(6): 6015–6025, 2021

    work page 2021

  7. [7]

    H. Yin, A. Varava, and D. Kragic. Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics, 6(54):eabd8803, 2021

    work page 2021

  8. [8]

    J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, et al. Challenges and outlook in robotic manipulation of deformable objects. IEEE Robotics & Automation Magazine , 29(3):67–77, 2022

    work page 2022

Show all 60 references
  1. [9]

    Sunil, S

    N. Sunil, S. Wang, Y . She, E. Adelson, and A. R. Garcia. Visuotactile affordances for cloth manipulation with local control. In Conference on Robot Learning, pages 1596–1606. PMLR, 2023

    2023

  2. [10]

    B. Fang, Z. Xia, F. Sun, Y . Yang, H. Liu, and C. Fang. Soft magnetic fingertip with particle- jamming structure for tactile perception and grasping. IEEE Transactions on Industrial Elec- tronics, 70(6):6027–6035, 2022

    2022

  3. [11]

    T. Li, Y . Yan, C. Yu, J. An, Y . Wang, and G. Chen. A comprehensive review of robot intelligent grasping based on tactile perception. Robotics and Computer-Integrated Manufacturing , 90: 102792, 2024. 9

    2024

  4. [12]

    Cui and J

    J. Cui and J. Trinkle. Toward next-generation learned robot manipulation. Science robotics, 6 (54):eabd9461, 2021

    2021

  5. [13]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning , pages 8748–8763. PmLR, 2021

    2021

  6. [14]

    J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  7. [15]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  8. [16]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  9. [17]

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan. 3d-vla: A 3d vision- language-action generative world model. arXiv preprint arXiv:2403.09631, 2024

    2024 arXiv

  10. [18]

    C. Li, J. Wen, Y . Peng, Y . Peng, F. Feng, and Y . Zhu. Pointvla: Injecting the 3d world into vision-language-action models. arXiv preprint arXiv:2503.07511, 2025

    2025 arXiv

  11. [19]

    Y . Hong, Z. Zheng, P. Chen, Y . Wang, J. Li, and C. Gan. Multiply: A multisensory object- centric embodied large language model in 3d world. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 26406–26416, 2024

    2024

  12. [20]

    F. Yang, C. Ma, J. Zhang, J. Zhu, W. Yuan, and A. Owens. Touch and go: Learning from human-collected vision and touch. In 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Neural information processing systems foundation, 2022

    2022

  13. [21]

    Jones, O

    J. Jones, O. Mees, C. Sferrazza, K. Stachowicz, P. Abbeel, and S. Levine. Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding.arXiv preprint arXiv:2501.04693, 2025

    2025 arXiv

  14. [22]

    Cheng, C

    N. Cheng, C. Guan, J. Gao, W. Wang, Y . Li, F. Meng, J. Zhou, B. Fang, J. Xu, and W. Han. Touch100k: A large-scale touch-language-vision dataset for touch-centric multimodal repre- sentation. arXiv preprint arXiv:2406.03813, 2024

    2024 arXiv

  15. [23]

    L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg. A touch, vision, and language dataset for multimodal alignment. In International Conference on Machine Learning , pages 14080–14101. PMLR, 2024

    2024

  16. [24]

    Suomalainen, Y

    M. Suomalainen, Y . Karayiannidis, and V . Kyrki. A survey of robot manipulation in contact. Robotics and Autonomous Systems , 156:104224, 2022

    2022

  17. [25]

    Zhang, S

    C. Zhang, S. Cui, S. Wang, J. Hu, Y . Cai, R. Wang, and Y . Wang. Gelstereo 2.0: An im- proved gelstereo sensor with multimedium refractive stereo calibration. IEEE Transactions on Industrial Electronics, 71(7):7452–7462, 2023

    2023

  18. [26]

    Doshi, O

    N. Doshi, O. Taylor, and A. Rodriguez. Manipulation of unknown objects via contact con- figuration regulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2693–2699. IEEE, 2022

    2022

  19. [27]

    X. Liu, P. Huang, and Z. Liu. A novel contact state estimation method for robot manipulation skill learning via environment dynamics and constraints modeling. IEEE Transactions on Automation Science and Engineering , 19(4):3903–3913, 2022. 10

    2022

  20. [28]

    Zhang, Z

    S. Zhang, Z. Chen, Y . Gao, W. Wan, J. Shan, H. Xue, F. Sun, Y . Yang, and B. Fang. Hardware technology of vision-based tactile sensor: A review. IEEE Sensors Journal , 22(22):21410– 21427, 2022

    2022

  21. [29]

    W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017

    2017

  22. [30]

    Stassi, V

    S. Stassi, V . Cauda, G. Canavese, and C. F. Pirri. Flexible tactile sensing based on piezoresistive composites: A review. Sensors, 14(3):5296–5332, 2014

    2014

  23. [31]

    Y . Du, G. Zhang, and M. Y . Wang. 3d contact point cloud reconstruction from vision-based tactile flow. IEEE Robotics and Automation Letters , 7(4):12177–12184, 2022

    2022

  24. [32]

    S. Wang, M. Lambeta, P.-W. Chou, and R. Calandra. Tacto: A fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930–3937, 2022

    2022

  25. [33]

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  26. [34]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  27. [35]

    Hegde, J

    D. Hegde, J. M. J. Valanarasu, and V . Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2028–2038, 2023

    2023

  28. [36]

    L. Xue, M. Gao, C. Xing, R. Mart ´ın-Mart´ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern re...

    2023

  29. [37]

    Jiang, A

    Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2(3):6, 2022

    2022 arXiv

  30. [38]

    Y . Liu, K. Zhang, Y . Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y . Huang, H. Sun, J. Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024

    2024 arXiv

  31. [39]

    Li and H

    S. Li and H. Tang. Multimodal alignment and fusion: A survey. arXiv preprint arXiv:2411.17040, 2024

    2024

  32. [40]

    Z. Li, X. Wu, H. Du, H. Nghiem, and G. Shi. Benchmark evaluations, applications, and challenges of large vision language models: A survey.arXiv preprint arXiv:2501.02189, 2025

    2025 arXiv

  33. [41]

    Zhang, J

    J. Zhang, J. Huang, S. Jin, and S. Lu. Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  34. [42]

    Panagopoulou, L

    A. Panagopoulou, L. Xue, N. Yu, J. Li, D. Li, S. Joty, R. Xu, S. Savarese, C. Xiong, and J. C. Niebles. X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning. In European Conference on Computer Vision , pages 177–

  35. [43]

    Y . Zhao, L. Zhao, X. Zhou, J. Wu, C.-T. Chu, H. Miao, F. Schroff, H. Adam, T. Liu, B. Gong, et al. Distilling vision-language models on millions of videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13106–13116, 2024. 11

    2024

  36. [44]

    L. Xue, N. Yu, S. Zhang, A. Panagopoulou, J. Li, R. Mart ´ın-Mart´ın, J. Wu, C. Xiong, R. Xu, J. C. Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2024

  37. [45]

    X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2639–2650, 2023

    2023

  38. [46]

    Zhang, Z

    R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y . Qiao, P. Gao, and H. Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8552–8562, 2022

    2022

  39. [47]

    S. Yu, K. Lin, A. Xiao, J. Duan, and H. Soh. Octopi: Object property reasoning with large tactile-language models. arXiv preprint arXiv:2405.02794, 2024

    2024 arXiv

  40. [48]

    P. Hao, C. Zhang, D. Li, X. Cao, X. Hao, S. Cui, and S. Wang. Tla: Tactile-language-action model for contact-rich manipulation. arXiv preprint arXiv:2503.08548, 2025

    2025 arXiv

  41. [49]

    Z. Xu, K. Wu, J. Wen, J. Li, N. Liu, Z. Che, and J. Tang. A survey on robotics with foundation models: toward embodied ai. arXiv preprint arXiv:2402.02385, 2024

    2024 arXiv

  42. [50]

    F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y . Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens, et al. Binding touch to everything: Learning unified multimodal tactile repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  43. [51]

    R. Feng, J. Hu, W. Xia, T. Gao, A. Shen, Y . Sun, B. Fang, and D. Hu. Anytouch: Learn- ing unified static-dynamic representation across multiple visuo-tactile sensors. arXiv preprint arXiv:2502.12191, 2025

    2025 arXiv

  44. [52]

    Calli, A

    B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510–517. IEEE, 2015

    2015

  45. [53]

    Coumans and Y

    E. Coumans and Y . Bai. Pybullet quickstart guide.ed: PyBullet Quickstart Guide. https://docs. google. com/document/u/1/d, 2021

    2021

  46. [54]

    Lambeta, P.-W

    M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V . R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation.IEEE Robotics and Automation Letters, 5(...

    2020

  47. [55]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  48. [56]

    X. Yu, L. Tang, Y . Rao, T. Huang, J. Zhou, and J. Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19313–19322, 2022

    2022

  49. [57]

    Y . Pang, W. Wang, F. E. Tay, W. Liu, Y . Tian, and L. Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision , pages 604–621. Springer, 2022

    2022

  50. [58]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  51. [59]

    Si and W

    Z. Si and W. Yuan. Taxim: An example-based simulation model for gelsight tactile sensors. IEEE Robotics and Automation Letters , 7(2):2361–2368, 2022. 12

    2022

  52. [60]

    Z. Chen, S. Zhang, S. Luo, F. Sun, and B. Fang. Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors. IEEE Robotics and Automa- tion Letters, 8(3):1239–1246, 2023. 13 Appendix This supplementary material provides additional...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.