Pith. sign in

REVIEW 3 major objections 5 minor 93 references

RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RA-Touch improves tactile description by retrieving tactile recaptions of ordinary images, without new touch data.

desk verdict Useful new resource and a coherent retrieval-augmented recipe for tactile description, but the headline gains lack direct statistical support and one mechanism control is missing. read the letter →

arxiv 2505.14270 v1 pith:SPWGPWG6 submitted 2025-05-20 cs.CV

Yoorhim Cho , Hongyeob Kim , Semin Kim , Youjia Zhang , Yunseok Choi , Sungeun Hong This is my paper · ORCID
classification cs.CV
keywords visuo-tactileperceptionretrieval-augmentedlearningtactiledescriptiongenerationimagerecaptioningvision-languagemodelstexture-awareretrievalTVLbenchmarkNet-T
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RA-Touch asks whether a model can learn to describe how objects feel without collecting new touch data, using only images and language. The paper's answer is yes: it recaptions a large visual corpus with tactile-focused adjectives, then retrieves visually different but tactilely similar examples when a touch input arrives. Retrieval and integration are both guided by the tactile signal, so the model attends to texture and material rather than object identity. On the TVL benchmark this raises the ViT-Base model's score from 5.03 to 5.36 and outperforms vision-language and touch-supervised baselines. The significance is that a cheap, abundant resource—ordinary images—can serve as tactile supervision.

What carries the argument

The central object is ImageNet-T, a recaptioned version of a standard image corpus in which each image carries five tactile adjectives (texture, flexibility, density, material feel) generated from the class name and a visual caption rather than from physical contact. The Tactile-Guided Retriever forms a query by letting the tactile embedding attend to the visual embedding: self-attention refines each modality, cross-attention uses the tactile feature as query over visual keys and values, and a linear projection aligns the result with the text embedding space. The Texture-Aware Integrator then runs two cross-attentions with the tactile embedding as query over the retrieved image features and retrieved text features, sums the outputs, projects them, and adds the result to the visual prompt embedding fed into the frozen language decoder. This combination is what lets the model keep object-centric noise out and texture-relevant cues in.

What would settle it

Take a subset of objects that appear in both the recaptioned corpus and the touch benchmark, compare the recaptioned adjectives against sensor-based or human-touch labels, and check whether they match better than chance; if they do not, or if substituting randomly permuted adjectives leaves the TVL score unchanged, the retrieval mechanism is not carrying the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that tactile understanding can be improved without additional tactile data by retrieving, from a recaptioned visual corpus, examples that feel similar to the queried object even when they look different. It constructs ImageNet-T by asking a large captioning model to write exactly five tactile adjectives for each image from its class name and visual description, then uses a Tactile-Guided Retriever to build a query from both touch and vision and pull top-K matching image-text pairs. A Texture-Aware Integrator re-weights those pairs by tactile relevance and adds them to the visual prompt of the base visuo-tactile model. On the TVL benchmark the ViT-Base variant improves from 5.03 to 5.36, with gains on both SSVTP and HCT subsets, and the authors conclude that retrieval over tactile-recaptioned visual data is a viable substitute for additional tactile supervision.

Load-bearing premise

The captioning model can infer accurate tactile adjectives for ordinary images from the class name and visual caption alone, and those inferred adjectives transfer to the objects in the touch benchmark—yet no physical touch check was performed on the recaptioned corpus.

Editorial extensions

If this is right

  • If the claim holds, tactile data collection—physical contact with specialized sensors—ceases to be the bottleneck for open-vocabulary tactile description; existing image corpora can be repurposed.
  • The reported gains grow as the recaptioned corpus grows from 10k to 150k samples, implying that more visual data, not more touch data, continues to help.
  • Tactile-focused captions outperform class names and ordinary visual captions, implying that the content of the recaptioned description carries much of the gain.
  • The fused visuo-tactile query outperforms image-only or touch-only retrieval, implying that both modalities are needed to find tactilely similar but visually distinct examples.
  • Performance peaks at K=7 retrieved samples and degrades beyond it, implying that retrieval context helps only up to a focused set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same recipe could be applied to other sparse sensory modalities, such as sound or proprioception, by recaptioning a large corpus and retrieving with a query built from the scarce modality.
  • A testable extension is to replace the proprietary captioning model with a smaller open model or a rule-based material lexicon; sustained gains would show the method depends on the recaptioning strategy rather than on one specific model.
  • The supplement's failure cases, in which small or background-dominated objects trigger texture-irrelevant retrievals, point to a natural follow-up: conditioning visual feature extraction on the tactile contact location.
  • If the recaptioned corpus were validated against physical tactile measurements, it could itself become a reusable benchmark for tactile-language alignment, separate from the TVL test set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RA-Touch, a retrieval-augmented framework for visuo-tactile description generation. To avoid collecting additional tactile data, the authors construct ImageNet-T, a version of ImageNet recaptioned with tactile-focused descriptions generated by GPT-4o mini from class names and BLIP-2 captions. RA-Touch augments TVL-LLaMA with two modules: a Tactile-Guided Retriever, which fuses visual and tactile features into a query to retrieve top-K image-text pairs from ImageNet-T, and a Texture-Aware Integrator, which combines the retrieved visual and textual features with the visual prompt before decoding with frozen LLaMA-2. On the TVL benchmark, the method reports consistent score improvements over the TVL-LLaMA baseline across ViT backbones, dataset subset sizes, and ablations, and the paper includes analyses of caption types, retrieval query designs, loss components, top-K, and dataset scale.

Significance. If the reported effects are real, the paper makes a valuable practical and conceptual contribution: it suggests that recaptioned visual corpora can stand in for additional tactile supervision, an appealing direction given the cost of tactile data collection. The construction of ImageNet-T is a reusable resource, and the ablation studies in Tables 3–6 and Figures 3–7 are unusually thorough for the area. The manuscript also demonstrates reproducible engineering (code release, frozen encoders, parameter-efficient tuning) and a clear experimental protocol. However, the empirical evidence for the central mechanistic claim is currently incomplete: the improvement over the immediate baseline is not tested for significance, and no control rules out the possibility that the gains come from a generic tactile-vocabulary prior rather than from semantically targeted retrieval.

major comments (3)
  1. [Table 2 and §4.3] The p-values reported in Table 2 are two-sided paired t-tests comparing each method to GPT-4V, not comparisons between RA-Touch and its direct TVL-LLaMA baseline. The central quantitative claim, e.g., 'the ViT-Base variant showing improvements of 0.33 on TVL,' is therefore not backed by any significance test. The authors should report paired t-tests (or equivalent) between RA-Touch and TVL-LLaMA for the same backbone, and ideally repeated-seed standard deviations, since the raw score differences are small (0.28–0.33 on TVL).
  2. [§4.3, Table 4; no random-retrieval control] The paper does not include a random-retrieval or semantically unrelated retrieval control, so the observed gains cannot be attributed uniquely to the tactile relevance of the retrieved ImageNet-T samples. Table 4 shows that even class-name-only and visual-description-only retrievals, which lack explicit tactile semantics, improve over the no-retrieval baseline by +0.12 and +0.20 on TVL, respectively, which suggests a generic context or distributional prior effect. A control that randomizes the retrieved pairs while keeping the retriever and integrator architecture unchanged would directly separate the effect of semantic retrieval from the effect of injecting ImageNet-T-like captions into the LLaMA prompt.
  3. [§4.1, Evaluation metric] The primary evaluation is a text-only GPT-4 judge rating semantic similarity to ground-truth labels, and the ground-truth labels themselves are a mixture of human and GPT-4V annotations, while ImageNet-T captions are generated by GPT-4o mini. This creates a same-family evaluator bias: the judge may systematically favor outputs that resemble GPT-generated text distributions rather than genuinely touch-grounded semantics. A small human evaluation or a non-GPT automatic judge on a subset of the TVL test set would help confirm that the reported gains are not an artifact of the evaluation protocol.
minor comments (5)
  1. [§3.3, Eq. (6)] Typo: 'cuas' should be 'cues' in the sentence before Eq. (4). Also clarify the dimensionality in Eq. (6): FFN operates after a linear layer, but it is not explicitly stated whether the FFN is applied in the D'=4096 space or the D=768 space; the residual p is in D'.
  2. [§3.4, Eq. (10)] The temperature τ in the InfoNCE loss is not defined; please state its value or how it is chosen.
  3. [§4.3, Table 2 caption] The text says 'the ViT-Base variant showing improvements of 0.33 on TVL,' but Table 2 reports +0.29 for the 10k subset and +0.33 for the 150k subset; clarify which setting is being summarized.
  4. [§5.5, Figure 7] The x-axis in Figure 7 is not linear (1, 3, 5, 7, 9, 16, 32); consider using a log scale or annotating the non-linear spacing to avoid misleading visual interpolation.
  5. [§5.4, Figure 6] The PCA visualization is qualitative; a quantitative measure of retrieval precision or query-text alignment (e.g., recall@k) would concretize the claim that the stability loss improves query quality.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the retrieval pipeline is supervised on TVL training text embeddings, and ImageNet-T is recaptioned from ImageNet independently of TVL test labels.

full rationale

The claimed derivation chain is that retrieving tactile-recaptioned ImageNet samples improves tactile description generation on the TVL benchmark. The only TVL supervision used is the standard training signal: Eq. (7) aligns the retriever query Q with the ground-truth text embedding L and the tactile embedding T on the training split, and the generator is trained with cross-entropy on TVL captions. At inference, ImageNet-T features are precomputed with the same frozen TVL/OpenCLIP encoders and retrieved by cosine similarity, so no TVL test label enters retrieval, integration, or generation. ImageNet-T captions are produced by GPT-4o mini from ImageNet class names and BLIP-2 captions (Sec. 3.1, Table 1), with no tactile sensor data and no TVL test annotations. The improvement is therefore not equivalent to the inputs by construction. The absence of a random-retrieval control and the fact that the evaluator is GPT-4 while captions are GPT-generated are methodological and attribution concerns, not circular derivation. The paper builds on the external TVL-LLaMA baseline rather than on a self-citation chain, and no load-bearing argument reduces to the authors' own prior claims.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method's gains rest on two unpaid assumptions: that model-generated tactile captions are accurate enough to transfer, and that the TVL/CLIP feature space supports tactile-conditioned retrieval. The loss hyperparameters and default K are hand-set, and no repeated-seed sensitivity analysis is reported.

free parameters (4)
  • lambda_align = 0.2
    Weight for the tactile-query alignment term in Equation 7, chosen by hand.
  • lambda_mse = 10
    Weight for the MSE stability loss in Equation 8, chosen by hand.
  • lambda_div_nce = 0.1
    Weight for the diversity and InfoNCE losses in Equation 8, chosen by hand.
  • retrieval_K = 5
    Default number of retrieved samples in all main experiments, although Figure 7 shows peak performance at K=7.
assumptions (3)
  • domain assumption GPT-4o mini can generate accurate tactile descriptors for ImageNet images from class names and BLIP-2 captions, and these descriptors transfer to TVL objects.
    Section 3.1 and Table D define the recaptioning procedure, but ImageNet-T captions are never verified against physical touch measurements.
  • domain assumption TVL encoders embed vision, language, and touch in a shared space where cosine-similarity retrieval is semantically meaningful for tactile properties.
    Section 3.2 inherits the TVL-LLaMA alignment; the paper provides no independent check that tactile query distances correspond to tactile similarity.
  • domain assumption GPT-4 text-only scoring of semantic similarity is a valid proxy for tactile description quality.
    Section 4.1 defines the evaluation protocol; the judge is from the same model family that generated the ImageNet-T captions, a risk the paper does not discuss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data." pith.science (2026). https://pith.science/paper/SPWGPWG6

@misc{pith2026250514270,
  author       = {Pith},
  title        = {Pith review of: RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPWGPWG6}},
  note         = {Machine review of arXiv:2505.14270}
}
read the original abstract

Visuo-tactile perception aims to understand an object's tactile properties, such as texture, softness, and rigidity. However, the field remains underexplored because collecting tactile data is costly and labor-intensive. We observe that visually distinct objects can exhibit similar surface textures or material properties. For example, a leather sofa and a leather jacket have different appearances but share similar tactile properties. This implies that tactile understanding can be guided by material cues in visual data, even without direct tactile supervision. In this paper, we introduce RA-Touch, a retrieval-augmented framework that improves visuo-tactile perception by leveraging visual data enriched with tactile semantics. We carefully recaption a large-scale visual dataset with tactile-focused descriptions, enabling the model to access tactile semantics typically absent from conventional visual datasets. A key challenge remains in effectively utilizing these tactile-aware external descriptions. RA-Touch addresses this by retrieving visual-textual representations aligned with tactile inputs and integrating them to focus on relevant textural and material properties. By outperforming prior methods on the TVL benchmark, our method demonstrates the potential of retrieval-based visual reuse for tactile understanding. Code is available at https://aim-skku.github.io/RA-Touch

Figures

Figures reproduced from arXiv: 2505.14270 by the authors.

Figure 1
Figure 1. RA-Touch motivation. Objects with different appear [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of RA-Touch. We first construct ImageNet-T, a vision-language dataset recaptioned with tactile-focused [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance comparisons across different subset [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Retrieval results with visual or tactile features. (a) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 7
Figure 7. Figure 7: Effect of the numbers of retrieved samples ( [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 6
Figure 6. Figure 6: Feature visualization of query embeddings. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

93 extracted references · 70 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023). 2, 5, 6

    arXiv 2023

  2. [2]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. InCOLM. 2

    2024

  3. [3]

    Andreas Blattmann and Jonas Müller Björn Ommer Robin Rombach, Kaan Oktay

  4. [4]

    Bresciani, Franziska Dammeier, and M

    J. Bresciani, Franziska Dammeier, and M. Ernst. 2006. Vision and touch are automatically integrated for the perception of sequences of events. InJournal of Vision. 1, 2

    2006

  5. [5]

    Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. 2024. Making Large Multimodal Models Under- stand Arbitrary Visual Prompts. InCVPR. 6

    2024

  6. [6]

    Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, E

    R. Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, E. Adelson, and S. Levine. 2018. More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch.IEEE Robotics and Automation Letters3, 4 (2018), 3300–3307. 1

    2018

  7. [7]

    Calandra, Andrew Owens, M

    R. Calandra, Andrew Owens, M. Upadhyaya, Wenzhen Yuan, Justin Lin, E. Adel- son, and S. Levine. 2017. The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?. InCoRL. 1

    2017

  8. [8]

    Camponogara and R

    I. Camponogara and R. Volcic. 2020. Integration of haptics and vision in human multisensory grasping. InCortex. 1, 2

    2020

Show all 93 references
  1. [9]

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. 2022. MuRAG: Multimodal Retrieval-Augmented Generator for Open Question An- swering over Images and Text. InEMNLP. 2

    2022

  2. [10]

    Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W. Cohen. 2023. Re- Imagen: Retrieval-Augmented Text-to-Image Generator. InICLR. 2

    2023

  3. [11]

    Yizhou Chen, Andrea Sipos, Mark Van der Merwe, and Nima Fazeli. 2022. Visuo- tactile transformers for manipulation. InCoRL. 1

    2022

  4. [12]

    Ning Cheng, Changhao Guan, Jing Gao, Weihao Wang, You Li, Fandong Meng, Jie Zhou, Bin Fang, Jinan Xu, and Wenjuan Han. 2024. Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation. arXiv preprint arXiv:2406.03813(2024). 1, 3

    2024 arXiv

  5. [13]

    Ning Cheng, You Li, Jing Gao, Bin Fang, Jinan Xu, and Wenjuan Han. 2024. To- wards Comprehensive Multimodal Perception: Introducing the Touch-Language- Vision Dataset, In ICIC.arXiv preprint arXiv:2403.09813. 1

    2024 arXiv

  6. [14]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 5

    2023

  7. [15]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In NeurIPS. 2, 6, 11

    2023

  8. [16]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

  9. [17]

    Xiang Fang, Wanlong Fang, Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Renfu Li, Zichuan Xu, Lixing Chen, Panpan Zheng, et al. 2024. Not all inputs are valid: Towards open-set video moment retrieval using language. InACM MM. 28–37. 3

    2024

  10. [18]

    Ruoxuan Feng, Di Hu, Wenke Ma, and Xuelong Li. 2024. Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation. InCoRL. 2

    2024

  11. [19]

    Letian Fu, Gaurav Datta, Huang Huang, Will Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg

  12. [20]

    Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. 2022. ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer. InCVPR. 2

    2022

  13. [21]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang

  14. [22]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024). 2, 3, 6, 11

    2024 arXiv

  15. [23]

    2021.OpenCLIP

    Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021.OpenCLIP. 3, 6

    2021

  16. [24]

    Ittyerah and L

    M. Ittyerah and L. Marks. 2007. Memory for curvature of objects: haptic touch vs. vision. InBritish Journal of Psychology. 2

    2007

  17. [25]

    M. G. Jones, Alexandra Bokinsky, T. Tretter, and Atsuko Negishi. 2005. A com- parison of learning with haptic and visual modalities. 2

    2005

  18. [26]

    Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. 2022. Self-supervised visuo-tactile pre- training to locate and follow garment features.arXiv preprint arXiv:2209.13042 (2022). 5, 6

    2022 arXiv

  19. [27]

    Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victo- ria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. 2020. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipula...

    2020

  20. [28]

    Visual Layer. 2024. imagenet-1k-vl-enriched. https://huggingface.co/datasets/ visual-layer/imagenet-1k-vl-enriched. 5

    2024

  21. [29]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. InICLR. 2

    2025

  22. [30]

    Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and K. Sohn. 2019. Context-Aware Emotion Recognition Networks. InICCV. 3

    2019

  23. [31]

    Yebin Lee, Imseong Park, and Myungjoo Kang. 2024. FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multi- modal Model. InACL. 11

    2024

  24. [32]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InNeurIPS. 2

    2020

  25. [33]

    Guohao Li, Xin Wang, and Wenwu Zhu. 2020. Boosting Visual Question Answer- ing with Context-aware Knowledge Aggregation.ACM MM(2020), 1227–1235. 3

    2020

  26. [34]

    Hongyu Li, Snehal Dikhale, Soshi Iba, and Nawid Jamali. 2023. ViHOPE: Vi- suotactile In-Hand Object 6D Pose Estimation With Shape Completion. InIEEE Robotics and Automation Letters. 1

    2023

  27. [35]

    Lee, Huazhe Xu, E

    Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A. Lee, Huazhe Xu, E. Adelson, Li Fei-Fei, Ruohan Gao, and Jiajun Wu. 2022. See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation. InCoRL. 1, 2

    2022

  28. [36]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InICML. 2, 5, 6, 11

    2023

  29. [37]

    Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. 2023. IntentQA: Context- aware Video Intent Reasoning. InICCV. 3

    2023

  30. [38]

    Platt, Wenzhen Yuan, A

    Rui Li, Robert W. Platt, Wenzhen Yuan, A. T. Pas, Nathan Roscup, M. Srinivasan, and E. Adelson. 2014. Localization and manipulation of small parts using GelSight tactile sensing. InIROS. 2

    2014

  31. [39]

    Shengdong Li, Chen Gong, Yuqing Zhu, Chuanwen Luo, Yi Hong, and Xue- qiang Lv. 2024. Context-aware multi-level question embedding fusion for visual question answering.Information Fusion102 (2024), 102000. 3

    2024

  32. [40]

    Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, and Antonio Torralba. 2019. Connecting Touch and Vision via Cross-Modal Prediction. InCVPR. 1

    2019

  33. [41]

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. 2023. Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering. InNeurIPS. 2

    2023

  34. [42]

    Zudi Lin, Erhan Bas, Kunwar Yashraj Singh, Gurumurthy Swaminathan, and Rahul Bhotika. 2023. Relaxing contrastiveness in multimodal representation learning. InW ACV. 2227–2236. 5

    2023

  35. [43]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InCVPR. 26296–26306. 6, 11

    2024

  36. [44]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. InNeurIPS, Vol. 36. 34892–34916. 2, 5, 6

    2023

  37. [45]

    Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. 2023. Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval. InICLR. 2

    2023

  38. [46]

    Fotios Lygerakis, Vedant Dave, and Elmar Rueckert. 2024. M2CURL: Sample- Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation. InInternational Conference on Ubiquitous Robots. 490–497. 1

    2024

  39. [47]

    Qian Mao, Zijian Liao, Jinfeng Yuan, and Rong Zhu. 2024. Multimodal tactile sensing fused with vision for dexterous robotic housekeeping.Nature Communi- cations15, 1 (2024), 6871. 1

    2024

  40. [48]

    Ronanki, and Kyu J Han

    Do June Min, Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi, S. Ronanki, and Kyu J Han. 2025. Speech Retrieval-Augmented Generation without Automatic Speech Recognition. InICASSP. 1, 2

    2025

  41. [49]

    Ojala, M

    T. Ojala, M. Pietikäinen, and Topi Mäenpää. 2002. Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns. InIEEE TPAMI. 2

    2002

  42. [50]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748(2018). 7

    2018 arXiv

  43. [51]

    Leszek Pecyna, Siyuan Dong, and Shan Luo. 2022. Visual-tactile multimodality for following deformable linear objects using reinforcement learning. InIROS. IEEE, 3987–3994. 1

    2022

  44. [52]

    9 Cho et al

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 9 Cho et al

  45. [53]

    Rita Ramos, Desmond Elliott, and Bruno Martins. 2023. Retrieval-augmented Image Captioning. InACL. 2

    2023

  46. [54]

    Jiahua Rao, Zifei Shan, Longpo Liu, Yao Zhou, and Yuedong Yang. 2023. Retrieval- based knowledge augmented vision language pre-training. InACM MM. 5399–

    2023

  47. [55]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Visi...

    2015

  48. [56]

    D’Andrea

    Carmelo Sferrazza and R. D’Andrea. 2019. Design, Motivation and Evaluation of a Full-Resolution Optical Tactile Sensor. InSensors. 2

    2019

  49. [57]

    Shimonomura

    K. Shimonomura. 2019. Tactile Image Sensors Employing Camera: A Review. In Sensors. 2

    2019

  50. [58]

    Edward Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal. 2020. 3D Shape Reconstruction from Vision and Touch.NeurIPS33 (2020), 14193–14206. 1

    2020

  51. [59]

    Stone and Claudia L

    K. Stone and Claudia L. R. Gonzalez. 2015. The contributions of vision and haptics to reaching and grasping. InFrontiers in Psychology. 1, 2

    2015

  52. [60]

    Sudharshan Suresh, Zilin Si, Joshua G Mangelson, Wenzhen Yuan, and Michael Kaess. 2022. ShapeMap 3-D: Efficient shape mapping through dense touch and vision. InICRA. IEEE, 7073–7080. 1

    2022

  53. [61]

    Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Felsberg, Mubarak Shah, and Fahad Shahbaz Khan. 2024. Composed Video Retrieval via Enriched Context and Discriminative Embeddings. InCVPR. 2

    2024

  54. [62]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023). 3, 5, 6, 11

    2023 arXiv

  55. [63]

    Xue Wang, Zhanshan Li, Yongping Huang, and Yingying Jiao. 2022. Multi- modal medical image segmentation using multi-scale context-aware network. Neurocomputing486 (2022), 135–146. 3

    2022

  56. [64]

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. InECCV. Springer, 387–404. 2

    2024

  57. [65]

    Guanfeng Wu, Abbas Haider, Ivor Spence, and Hui Wang. 2024. Multi Modal Fusion for Video Retrieval based on CLIP Guide Feature Alignment. InMVRMLM ’24: Proceedings of 2024 ACM ICMR Workshop on Multimodal Video Retrieval. 2

    2024

  58. [66]

    Chen-Wei Xie, Siyang Sun, Xiong Xiong, Yun Zheng, Deli Zhao, and Jingren Zhou

  59. [67]

    Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang, Cael Fitch, Philip Glen Cran- dall, Wan Shou, Dongyi Wang, and Yu She. 2024. UniT: Unified Tactile Repre- sentation for Robot Learning. InarXiv.org. 2

    2024

  60. [68]

    Akihiko Yamaguchi and C. Atkeson. 2016. Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables. InIEEE- RAS International Conference on Humanoid Robots. 2

    2016

  61. [69]

    Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, and Alex Wong. 2024. Binding Touch to Everything: Learning Unified Multimodal Tactile Representations. InCVPR. 1, 2, 3

    2024

  62. [70]

    Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. 2022. Touch and Go: Learning from Human-Collected Vision and Touch. InNeurIPS. 2, 3

    2022

  63. [71]

    Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. 2024. Octopi: Object property reasoning with large tactile-language models.arXiv preprint arXiv:2405.02794(2024). 1

    2024 arXiv

  64. [72]

    RA-CLIP: Retrieval Augmented Contrastive Language-Image Pre-Training. InCVPR. 1, 2

  65. [73]

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow twins: Self-supervised learning via redundancy reduction. InICML. PMLR, 12310– 12320. 5

    2021

  66. [74]

    Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, and Feng Wu

  67. [75]

    Fan Zhang and Yiannis Demiris. 2023. Visual-tactile learning of garment unfold- ing for robot-assisted dressing.IEEE Robotics and Automation Letters8, 9 (2023), 5512–5519. 1

    2023

  68. [76]

    Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023. ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. InICCV. 2

    2023

  69. [77]

    Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and S. Li. 2020. Context-Aware Attention Network for Image-Text Retrieval. InCVPR. 3

    2020

  70. [78]

    Wenzhen Yuan, Siyuan Dong, and E. Adelson. 2017. GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force. InSensors. 1, 2

    2017

  71. [85]

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. 2024. LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention. InICLR. 2, 6 10 Supplementary for RA-Touch: Retrieval-Augmented Touch Understandi...

    2024

  72. [86]

    Provide exactly 5 adjectives that refer solely to how the object feels to the touch–focusing on texture, flexibility, density, and material properties

  73. [87]

    Try to include more varied and nuanced tactile descriptors

  74. [88]

    Do not include adjectives related to visual appearance, shape, color, temperature, sound, weight, or any non-tactile properties or any non-tactile properties

  75. [89]

    adj1, adj2, adj3, adj4, adj5

    Respond using the exact format: "adj1, adj2, adj3, adj4, adj5". Remember: Your ENTIRE response must be ONLY 5 adjectives separated by commas. Table D: Overview of prompt used for recaptioning. D.2 Distribution of Vocabulary Words In open-vocabulary tactile perception tasks, ha...

  76. [90]

    For a smooth and cold surface, the description might be 㑃slippery, icy, hard, unyielding, glassy.㑃 3

    For a rough and hot surface, the description might be 㑃scratchy, searing, scorching, jagged, scalding.㑃 2. For a smooth and cold surface, the description might be 㑃slippery, icy, hard, unyielding, glassy.㑃 3. For a rough and hot surface, the description might be 㑃scratchy, sco...

  77. [91]

    LLaVA 1.5 13B 5.5Slick, gritty, textured, sharp, roughViP㎿LLaVA 7B 1Slick, gritty, textured, sharp, roughViP㎿LLaVA 13B

    Slick, hard, unyielding, cold, glossy.LLaMA㎿Adapter 1There is no tactile surface on this image.BLIP㎿2 OPT㎿6.7 2the close up picture of a blue denim jacket with button on the right sideInstructBLIP 7B 1light blue jean jacket nothing 3InstructBLIP 13B Score ㏙1㎿10㏚DescriptionMode...

  78. [92]

    Slick 2. Cold 3. Hard 4. Unyielding 5. Glossy.LLaMA㎿Adapter 2 The first word that came to my mind was 㑃sharp㑃 because I can see the pointy edge. I also think it㑄s very comfortable because it㑄s BLIP㎿2 OPT㎿6.7 2some rocks are being exposed to some shining lightInstructBLIP 7B 1a...

  79. [93]

    Slick, hard, unyielding, cold, glossy.LLaMA㎿Adapter 4I㑄m not exactly sure, but it looks like the patch has markings on it, so I guess it makes the sensor feel rough and hardBLIP㎿2 OPT㎿6.7 2there is a white and black striped objectInstructBLIP 7B 1a close up of an old football ...

  80. [2019]

    IEEE TPAMI44, 2 (2019), 710–722

    Context-aware visual policy network for fine-grained image captioning. IEEE TPAMI44, 2 (2019), 710–722. 3

    2019

  81. [2020]

    REALM: Retrieval-Augmented Language Model Pre-Training. InICML. 2

  82. [2021]

    Learning transferable visual models from natural language supervision. In ICML. 2, 7

  83. [2022]

    InNeurIPS

    Semi-Parametric Neural Image Synthesis. InNeurIPS. 2

  84. [2023]

    In ICASSP

    CLAP Learning Audio Concepts from Natural Language Supervision. In ICASSP. 1–5. 1, 2

  85. [2024]

    A Touch, Vision, and Language Dataset for Multimodal Alignment. In ICML. 1, 2, 3, 4, 5, 6, 7, 11, 12, 14

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.