Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Universal Visuo-Tactile Video Understanding for Embodied Interaction

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VTV-LLM, a multimodal language model trained on 150,000 visuo-tactile video frames, answers natural-language questions about hardness, protrusion, elasticity, and friction, outperforming GPT-4o on the VTV150K benchmark (60.4% vs 28.0%…

desk verdict A genuinely novel tactile-video QA dataset with a three-stage LLM pipeline, but the headline accuracy gap over zero-shot baselines is likely inflated by in-distribution template matching. read the letter →

arxiv 2505.22566 v1 pith:C7ZPQRVM submitted 2025-05-28 cs.CV cs.AI

Yifan Xie , Mingyang Li , Shoujie Li , Xingting Li , Guangyu Chen , Fei Ma , Fei Richard Yu , Wenbo Ding This is my paper
classification cs.CVcs.AI
keywords visuo-tactilevideounderstandingtactileperceptionmultimodallargelanguagemodelreasoningVTV150Kembodiedinteractioncross-modalalignmentsensors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give embodied agents a sense of touch in language: given a video from a visuo-tactile sensor, VTV-LLM should answer questions about physical attributes that vision alone cannot reliably reveal. To that end the authors collected VTV150K, 150,000 frames of 100 everyday objects recorded with GelSight Mini, DIGIT, and Tac3D, annotated for hardness, protrusion, elasticity, and friction, plus 10,000 template-generated question-answer pairs. On a held-out 600-pair test with novel objects, the 7B model reaches 60.4% average accuracy across five tactile reasoning tasks, compared with 28.0% for GPT-4o and 29.5% for Gemini-2.5-Pro evaluated zero-shot. If the result holds, robots and assistive systems could describe what they feel and justify material choices in plain language.

What carries the argument

The load-bearing object is the VTV encoder: a ViT-base initialized from VideoMAE and retrained in Stage 1 to reconstruct masked visuo-tactile frames while classifying the four tactile attributes. The masking is the novel part. Rather than random tubes, the middle frame (maximal contact area) is segmented with a mixture of Gaussian kernels, and RAFT computes bidirectional optical flow so the mask can be warped backward across the video; masked regions therefore align with actual deformation trajectories. Because reconstruction and attribute classification are optimized jointly, the encoder must keep the information that matters for hardness, protrusion, elasticity, and friction. Stages 2 and 3 then lock this encoder, train V- and T-projectors to map video and text into Qwen 2.5's embedding space, and finetune the LLM on 10,000 template-generated QA pairs.

What would settle it

Fine-tune one open-source baseline (e.g., Qwen2.5-VL-7B) on the VTV150K training QA pairs with the same compute budget and compare on the held-out 600-pair test set; if it approaches or exceeds 60.4%, the claimed advantage is mostly training on the task rather than a new tactile capability. Additionally, a 300-question test written by humans with no template phrasing would test whether the model generalizes beyond generated language.

Watch

Extended reading notes

Core claim

The central claim is that tactile-video understanding is a solvable cross-modal language problem once the video encoder is trained on the right inductive bias. The paper argues that generic video-language models fail because tube-style masking assumes small motion, while tactile videos are dominated by deformation, and because generalist encoders have never seen sensor contact patterns. VTV-LLM therefore uses an optical-flow-guided masked autoencoder to force the representation to track deformation, aligns the resulting embeddings to a frozen Qwen 2.5 backbone, and finetunes on structured QA pairs. The reported demonstration is consistent superiority on every task in Table 1, with the largest margins on combined attribute assessment (35.6% vs 2.1% for the best generalist baseline).

Load-bearing premise

The results depend on the comparison design: every baseline is evaluated zero-shot on template-generated questions, while VTV-LLM is trained on the same template distribution; if that evaluation asymmetry were removed, the reported margin could shrink.

Editorial extensions

If this is right

  • A single model can watch tactile sensor videos and answer in natural language about four physical attributes, at 60.4% average accuracy on the VTV150K test set.
  • Tactile understanding transfers across three sensor types with different resolutions, so applications are not locked to one sensor's visual style.
  • The same trained model handles higher-level reasoning tasks: comparing objects, matching sensations to object identity, and choosing a surface for a practical scenario, including a scenario task held out from training.
  • Larger language backbones improve performance monotonically from 3B to 14B, so the approach should continue to benefit from stronger base models.
  • Every training stage contributes: ablations show removing the text-alignment stage drops average accuracy from 59.0% to 50.8%, and removing prompt finetuning drops it to 43.5%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural check the paper does not run: fine-tune a generalist baseline on the same VTV150K training QA pairs. If the reported 32-point gap over GPT-4o shrinks to a few points, the advantage is largely task familiarity rather than tactile perception per se.
  • The template-generated questions share phrasing with the training data, so the model may be learning a controlled language of attributes; a human-written, naturally phrased test set would show whether the model understands touch or parses templates.
  • The optical-flow-guided masking idea should transfer to any deformable-contact video source, such as pressure arrays or soft robotic skins, not just elastomer-based tactile cameras.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces VTV-LLM, a multimodal large language model for visuo-tactile video understanding, and the VTV150K dataset. The dataset comprises 150,000 video frames from 100 objects captured with three tactile sensors (GelSight Mini, DIGIT, Tac3D), annotated with four attribute labels and augmented with 10,000 template-generated question-answer pairs for training and 600 for testing. The model uses a three-stage training paradigm: optical-flow-guided masked autoencoding for representation learning, projector-based cross-modal alignment, and supervised finetuning of the LLM. Experiments compare VTV-LLM-7B against seven zero-shot baselines, reporting 60.4% average accuracy versus 28.0% for GPT-4o. The paper also includes ablations on the LLM backbone, the VTV encoder, and the training stages.

Significance. The VTV150K dataset is a potentially valuable resource for the tactile-language community, and the three-stage training pipeline is clearly motivated by the differences between tactile video and natural video. The reported performance gains, however, are not convincing because the evaluation is confounded: the model is trained on template-generated question-answer pairs from the same pipeline that produces the test set, while all baselines are evaluated zero-shot. If the evaluation is corrected with fine-tuned baselines and a non-template benchmark, the method may still be competitive, but the current manuscript does not establish the claimed universal or superior visuo-tactile understanding. The ablations do provide some internal support for the design choices, which is a strength.

major comments (5)
  1. [Section 4.2, Table 1] The headline comparison is not a controlled experiment. VTV-LLM is trained on VTV150K, including the template-generated question-answer pairs used in Stage 3, while all seven baselines are evaluated zero-shot with no exposure to the dataset or its answer format. The reported gap (60.4% vs 28.0% for GPT-4o) therefore conflates task-specific training with visuo-tactile understanding. To support the claim of consistent superiority, the authors should either fine-tune all baselines on the same training split under comparable protocols, or report a zero-shot VTV-LLM and compare it against zero-shot baselines; an external non-template benchmark would additionally guard against template overfitting.
  2. [Sections 3.1 and 4.1] The test set of 600 question-answer pairs is generated by the same hierarchical template pipeline used to produce the 20,000 training pairs, and the ground-truth labels are deterministic functions of the same four attribute annotations. Consequently, the test distribution is essentially identical to the training distribution, and high accuracy may reflect learning the template grammar and the annotation-to-text mapping rather than grounding in the visual-tactile signal. The use of novel objects reduces object-level leakage but does not change the template distribution. I request an evaluation on naturally phrased questions (e.g., human-written prompts or an existing tactile-language benchmark) and an analysis of accuracy as a function of template novelty.
  3. [Section 4.2, Results paragraph] The paper states that 'the TSA task is not included in the training set,' but Section 3.1 lists 'texture optimal selection' as one of the template types, which appears to be semantically equivalent to tactile scenario analysis. This is a direct internal contradiction. If TSA-style templates are in the training data, the claim that TSA tests zero-shot generalization is false; if they are not, the paper must explain how the test set can contain a task type that was never trained, and how the baselines were prompted for it. The authors should clarify the exact relationship and remove the contradiction.
  4. [Section 4.2, Tables 1-3 and Figure 5] The text reports 'the average results of the triplicate test with random seeds,' but no standard deviations, confidence intervals, or per-run results are given anywhere in the paper, including the ablation tables and Figure 5. Without a measure of variance, the differences between the top baselines (e.g., 28.0% vs 29.5% for Gemini) and the ablation improvements (e.g., 51.5% to 59.0% in Table 2) cannot be assessed for statistical significance. The authors should report means with standard deviations or the results of each of the three runs.
  5. [Section 4.2] The accuracy metric is not defined. The model produces free-form answers such as 'a) It presents an extremely hard...', so it is unclear whether a response is scored by exact string match, attribute-value extraction, or an LLM-based judge. This is essential for interpreting all quantitative results and for reproducibility. Please specify the scoring procedure, including how partial credit is handled for multi-attribute answers.
minor comments (6)
  1. [Table 1] The model names 'LLaV A-OneVision-7B' and 'LLaV A-Video-Qwen2-7B' should be written as 'LLaVA-OneVision-7B' and 'LLaVA-Video-Qwen2-7B' for correctness.
  2. [Section 3.2] The sentence 'At the core of VTV-LLM lies a (Qwen 2.5 [4, 5])...' is a fragment; it should read 'a Qwen 2.5 LLM [4, 5]'.
  3. [Equation (1)] The bracket notation in Equation (1) is unclear; consider rewriting the sequence of patch embeddings and temporal embeddings with standard mathematical notation.
  4. [Figure 3] The caption does not define the variables R0...RT shown as reconstructed video frames; please add a sentence explaining the notation.
  5. [Section 4.2] The phrase 'triplicate test with random seeds' should specify the number of seeds and whether the same test set was used across runs; additionally, report the seed values in the supplementary material.
  6. [Overall] The paper does not include a data or code availability statement; given that the dataset is a central contribution, the authors should state how VTV150K and the implementation will be released.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline performance gap is partly an artifact of same-template train/test: VTV-LLM is trained on VTV150K template-generated QA and evaluated on a test set created from the same template-generation approach, while baselines are not fine-tuned; novel objects and a held-out TSA task provide only partial independence.

  1. fitted input called prediction [Sec. 3.2 'Text Prompt Finetuning'; Sec. 4.1 'Setup'; Table 1]
    "These pairs are created using the same template generation approach as our VTV150K dataset... To evaluate model performance, we create a separate test set comprising 600 question-answer pairs for novel objects not present in the training data... VTV-LLM-7B (Ours) ... 60.4."

    The model is trained on instruction-following QA generated by the VTV150K template hierarchy, and the test set used for the headline comparison is drawn from the same VTV150K/template-generation pipeline. The evaluation therefore measures in-distribution template mastery rather than an externally grounded universal capability: the trained model has seen the exact question formats, answer vocabulary, and label-generation conventions, while the seven baselines are evaluated without fine-tuning on this distribution. The reported 60.4% versus 28.0% gap partly encodes this train/test distribution match, so the 'superior performance' claim is not an independent derivation of general visuo-tactile understanding.

full rationale

The paper's derivation chain is largely empirical rather than formal, and no equation in the methods reduces to its own input by construction. The self-citations present (e.g., the authors' own sensor review and a prior generative-model paper) are not load-bearing for the central claims. The main circularity-adjacent issue is evaluation design: the VTV-LLM is trained on 10,000 template-generated question-answer pairs created with the same template approach used to construct VTV150K, and the 600-question test set is also generated within the same VTV150K protocol. Because the baselines are not fine-tuned on this template distribution, the headline performance gap is partly a measure of in-distribution template familiarity rather than proof of universal visuo-tactile understanding. However, the held-out novel objects and the explicit exclusion of the TSA task from training provide some independent signal, so the central claim is not completely forced. Therefore a moderate score of 4 reflects partial circularity in the evaluation while acknowledging the remaining independent content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest primarily on the VTV150K dataset, its human annotations, and the template-based evaluation. The free parameters are mask-generation hyperparameters whose values are not disclosed. The domain assumptions about middle-frame keyframes, optical-flow tracking, and annotation accuracy are plausible but unverified. The template generation is an ad hoc-to-paper choice that shapes both training and evaluation. No invented physical entities appear.

free parameters (5)
  • alpha (mask sampling density)
    Controls the number of Gaussian sampling points N = ceil(alpha * H * W / beta^2) for the keyframe mask in Section 3.2; no value is reported.
  • beta (sampling grid size)
    Grid size for placing Gaussian sampling points in the keyframe mask; no value is reported in Section 3.2.
  • lambda (Gaussian kernel scale)
    Scale parameter of each localized Gaussian kernel in the keyframe mask; no value is reported in Section 3.2.
  • keyframe index (middle frame) = T/2
    The middle frame is chosen as keyframe based on an empirical claim that it shows maximum contact area; this choice is a modeling assumption, not a measured optimum across objects.
  • train and test QA pair counts = 10,000 / 10,000 / 600
    Numbers of QA pairs for stage 2, stage 3, and testing are chosen by the authors; the test set size is small and may limit statistical power.
assumptions (5)
  • domain assumption Human annotations of hardness, protrusion, elasticity, and friction on a three-level scale are accurate ground truth for every object and sensor.
    Section 3.1 Attribute Annotation uses these labels for training and evaluation; no inter-annotator agreement or physical measurement is reported.
  • domain assumption The middle frame of each tactile video exhibits the maximum contact surface area.
    Section 3.2 selects the middle frame as keyframe, motivated by empirical observations; if this fails for sliding or transient contacts, the generated masks are not temporally consistent.
  • domain assumption RAFT optical flow on tactile gel images tracks physical deformation, so backward warping of the keyframe mask yields meaningful temporal masks.
    Section 3.2 uses RAFT to compute bidirectional flow and warp masks; this assumes pixel-level motion corresponds to tactile surface deformation.
  • ad hoc to paper Template-generated QA pairs are a valid proxy for natural-language embodied interaction queries.
    Section 3.1 Template Generation creates all training and test questions from structured templates; the paper gives no evidence that this distribution matches real user or robot queries.
  • domain assumption Attribute labels are sensor-invariant: the same physical object has the same labels across GelSight Mini, DIGIT, and Tac3D.
    Section 3.1 collects videos of the same objects across three sensors and uses shared annotations; cross-sensor label consistency is asserted, not measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Universal Visuo-Tactile Video Understanding for Embodied Interaction." pith.science (2026). https://pith.science/paper/C7ZPQRVM

@misc{pith2026250522566,
  author       = {Pith},
  title        = {Pith review of: Universal Visuo-Tactile Video Understanding for Embodied Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7ZPQRVM}},
  note         = {Machine review of arXiv:2505.22566}
}
read the original abstract

Tactile perception is essential for embodied agents to understand physical attributes of objects that cannot be determined through visual inspection alone. While existing approaches have made progress in visual and language modalities for physical understanding, they fail to effectively incorporate tactile information that provides crucial haptic feedback for real-world interaction. In this paper, we present VTV-LLM, the first multi-modal large language model for universal Visuo-Tactile Video (VTV) understanding that bridges the gap between tactile perception and natural language. To address the challenges of cross-sensor and cross-modal integration, we contribute VTV150K, a comprehensive dataset comprising 150,000 video frames from 100 diverse objects captured across three different tactile sensors (GelSight Mini, DIGIT, and Tac3D), annotated with four fundamental tactile attributes (hardness, protrusion, elasticity, and friction). We develop a novel three-stage training paradigm that includes VTV enhancement for robust visuo-tactile representation, VTV-text alignment for cross-modal correspondence, and text prompt finetuning for natural language generation. Our framework enables sophisticated tactile reasoning capabilities including feature assessment, comparative analysis, scenario-based decision making and so on. Experimental evaluations demonstrate that VTV-LLM achieves superior performance in tactile video understanding tasks, establishing a foundation for more intuitive human-machine interaction in tactile domains.

Figures

Figures reproduced from arXiv: 2505.22566 by the authors.

Figure 1
Figure 1. The workflow consists of four key components: (a) Data Collection, which includes [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) VTV-LLM framework: A multi-modal system integrating visual-tactile video data [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Training pipeline of VTV enhancement. Given these aligned representations, the large language model fLLM performs reasoning to generate a response A describing tactile attributes: A = fLLM(EV , ET ) = Qwen(Concat([EV ; ET ])). (3) Given the complexity of integrating visuo-tactile information with language representations, we implement a staged training approach to develop our framework. As shown in [PITH_FULL_IMAGE… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Several task examples from the proposed VTV150K along with predictions from VTV [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of VTV-LLM on the different parameters. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TacReasoner: A Dynamic Tactile-Language Framework for Interactive Reasoning in Real-World Scenarios

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A dynamic-aware tactile encoder plus TouchCoT-10k chain-of-thought data lets a 7B model outperform larger tactile-language baselines on physical-property and real-world reasoning tasks.

Reference graph

Works this paper leans on

60 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    A review of tactile information: Perception and action through touch

    Qiang Li, Oliver Kroemer, Zhe Su, Filipe Fernandes Veiga, Mohsen Kaboli, and Helge Joachim Ritter. A review of tactile information: Perception and action through touch. IEEE Transactions on Robotics, 36(6):1619–1634, 2020

    work page 2020

  2. [2]

    Task and material properties interac- tively affect softness explorations along different dimensions

    Müge Cavdan, Katja Doerschner, and Knut Drewing. Task and material properties interac- tively affect softness explorations along different dimensions. IEEE Transactions on Haptics, 14(3):603–614, 2021

    work page 2021

  3. [3]

    Predicting perceptual haptic attributes of textured surface from tactile data based on deep cnn-lstm network

    Mudassir Ibrahim Awan, Waseem Hassan, and Seokhee Jeon. Predicting perceptual haptic attributes of textured surface from tactile data based on deep cnn-lstm network. In Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology, pages 1–9, 2023

    work page 2023

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  6. [6]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  7. [7]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024

    2024

  8. [8]

    CCIS-Diff: A Generative Model with Stable Diffusion Prior for Controlled Colonoscopy Image Synthesis

    Yifan Xie, Jingge Wang, Tao Feng, Fei Ma, and Yang Li. Ccis-diff: A generative model with sta- ble diffusion prior for controlled colonoscopy image synthesis.arXiv preprint arXiv:2411.12198, 2024

    work page Pith review arXiv 2024

Show all 60 references
  1. [9]

    When vision meets touch: A contemporary review for visuotactile sensors from the signal processing perspective

    Shoujie Li, Zihan Wang, Changsheng Wu, Xiang Li, Shan Luo, Bin Fang, Fuchun Sun, Xiao- Ping Zhang, and Wenbo Ding. When vision meets touch: A contemporary review for visuotactile sensors from the signal processing perspective. IEEE Journal of Selected Topics in Signal Processing, 2024

    2024

  2. [10]

    Gelsight: High-resolution robot tactile sensors for estimating geometry and force

    Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):2762, 2017

    2017

  3. [11]

    Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation

    Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IE...

    2020

  4. [12]

    Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution

    Lunwei Zhang, Yue Wang, and Yao Jiang. Tac3d: A novel vision-based tactile sensor for measuring forces distribution and estimating friction coefficient distribution. arXiv preprint arXiv:2202.06211, 2022

    2022 arXiv

  5. [13]

    Anytouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors

    Ruoxuan Feng, Jiangyu Hu, Wenke Xia, Ao Shen, Yuhao Sun, Bin Fang, Di Hu, et al. Anytouch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  6. [14]

    Transferable tactile transformers for representation learning across diverse sensors and tasks

    Jialiang Zhao, Yuxiang Ma, Lirui Wang, and Edward Adelson. Transferable tactile transformers for representation learning across diverse sensors and tasks. In 8th Annual Conference on Robot Learning, 2024

    2024

  7. [15]

    Octopi: Object property reasoning with large tactile-language models

    Samson Yu, Kelvin Lin, Anxing Xiao, Jiafei Duan, and Harold Soh. Octopi: Object property reasoning with large tactile-language models. arXiv preprint arXiv:2405.02794, 2024. 10

    2024 arXiv

  8. [16]

    A touch, vision, and language dataset for multimodal alignment

    Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg. A touch, vision, and language dataset for multimodal alignment. In International Conference on Machine Learning, pages...

    2024

  9. [17]

    Binding touch to everything: Learn- ing unified multimodal tactile representations

    Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, et al. Binding touch to everything: Learn- ing unified multimodal tactile representations. In Proceedings of the IEEE/CVF Conference on Compute...

    2024

  10. [18]

    Sparsh: Self-supervised touch representations for vision-based tactile sensing

    Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, et al. Sparsh: Self-supervised touch representations for vision-based tactile sensing. In 8th Annual Conferen...

    2024

  11. [19]

    Visuo- tactile affordances for cloth manipulation with local control

    Neha Sunil, Shaoxiong Wang, Yu She, Edward Adelson, and Alberto Rodriguez Garcia. Visuo- tactile affordances for cloth manipulation with local control. In Conference on Robot Learning, pages 1596–1606. PMLR, 2023

    2023

  12. [20]

    A survey of embodied learning for object-centric robotic manipulation

    Ying Zheng, Lei Yao, Yuejiao Su, Yi Zhang, Yi Wang, Sicheng Zhao, Yiyi Zhang, and Lap-Pui Chau. A survey of embodied learning for object-centric robotic manipulation. arXiv preprint arXiv:2408.11537, 2024

    2024 arXiv

  13. [21]

    Touch and go: learning from human-collected vision and touch

    Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, and Andrew Owens. Touch and go: learning from human-collected vision and touch. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 8081–8103, 2022

    2022

  14. [22]

    Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations

    Ruohan Gao, Yen-Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In Conference on Robot Learning, 2021

    2021

  15. [23]

    Objectfolder 2.0: A multisensory object dataset for sim2real transfer

    Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10598–...

    2022

  16. [24]

    See, hear, and feel: Smart sensory fusion for robotic manipulation

    Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao, and Jiajun Wu. See, hear, and feel: Smart sensory fusion for robotic manipulation. In Conference on Robot Learning, pages 1368–1378. PMLR, 2023

    2023

  17. [25]

    Active clothing material perception using tactile sensing and deep learning

    Wenzhen Yuan, Yuchen Mo, Shaoxiong Wang, and Edward H Adelson. Active clothing material perception using tactile sensing and deep learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4842–4849. IEEE, 2018

    2018

  18. [26]

    Self-supervised visuo-tactile pretraining to locate and follow garment features

    Justin Kerr, Huang Huang, Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. arXiv preprint arXiv:2209.13042, 2022

    2022 arXiv

  19. [27]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022

    2022

  20. [28]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14549–14560, 2023

    2023

  21. [29]

    Sigma: Sinkhorn-guided masked video modeling

    Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhorn-guided masked video modeling. In European Conference on Computer Vision, pages 293–312. Springer, 2024

    2024

  22. [30]

    Mgmae: Motion guided masking for video masked autoencoding

    Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13493–13504, 2023. 11

    2023

  23. [31]

    Videomap: Toward scalable mamba-based video autoregressive pretraining

    Yunze Liu, Peiran Wu, Cheng Liang, Junxiao Shen, Limin Wang, and Li Yi. Videomap: Toward scalable mamba-based video autoregressive pretraining. arXiv preprint arXiv:2503.12332, 2025

    2025 arXiv

  24. [32]

    Videomac: Video masked autoencoders meet convnets

    Gensheng Pei, Tao Chen, Xiruo Jiang, Huafeng Liu, Zeren Sun, and Yazhou Yao. Videomac: Video masked autoencoders meet convnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22733–22743, 2024

    2024

  25. [33]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154–38180, 2023

    2023

  26. [34]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023

    2023

  27. [35]

    Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36:71995–72007, 2023

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36:71995–72007, 2023

    2023

  28. [36]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  29. [37]

    Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. In ICLR, 2024

    2024

  30. [38]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  32. [40]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 125...

    2024

  33. [41]

    Videollama 3: Frontier multimodal foundation models for image and video understanding

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025

    2025 arXiv

  34. [42]

    Llama-omni: Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  35. [43]

    Rdt-1b: a diffusion foundation model for bimanual manipulation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.07864, 2024

    2024 arXiv

  36. [44]

    Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding

    Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, and Sergey Levine. Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding. arXiv preprint arXiv:2501.04693, 2025

    2025 arXiv

  37. [45]

    Touch2touch: Cross-modal tactile generation for object manipulation

    Samanta Rodriguez, Yiming Dou, Miquel Oller, Andrew Owens, and Nima Fazeli. Touch2touch: Cross-modal tactile generation for object manipulation. arXiv preprint arXiv:2409.08269, 2024

    2024 arXiv

  38. [46]

    Cubic spline interpolation

    Sky McKinley and Megan Levine. Cubic spline interpolation. College of the Redwoods , 45(1):1049–1060, 1998

    1998

  39. [47]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2020

  40. [48]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  41. [49]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  42. [50]

    Gaussian mixture models

    Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659- 663):3, 2009

    2009

  43. [51]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020

    2020

  44. [52]

    Forward and backward warping for optical flow-based frame interpolation

    Joi Shimizu, Heming Sun, and Jiro Katto. Forward and backward warping for optical flow-based frame interpolation. In 2022 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 082–086. IEEE, 2022

    2022

  45. [53]

    Extrapolation-based video retargeting with backward warping using an image-to-warping vector generation network

    Sung In Cho and Suk-Ju Kang. Extrapolation-based video retargeting with backward warping using an image-to-warping vector generation network. IEEE Signal Processing Letters, 27:446– 450, 2020

    2020

  46. [54]

    Cross-entropy loss functions: Theoretical analysis and applications

    Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International conference on Machine learning, pages 23803–23828. PMLR, 2023

    2023

  47. [55]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  48. [56]

    gemini-2.5-pro-preview-05-06

    Google DeepMind. gemini-2.5-pro-preview-05-06. https://ai.google.dev/gemini-api/ docs/models#gemini-2.5-pro-preview-05-06 , 2025

    2025

  49. [57]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  50. [58]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  51. [59]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  52. [60]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 13

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.