Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

TacFiLM integrates touch into a vision-language-action model by using pretrained tactile embeddings to modulate intermediate visual features with FiLM layers, and the paper reports consistent gains in insertion success, direct-insertion rat

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:08 UTC pith:XOVJ6V6V

load-bearing objection TacFiLM is a promising lightweight way to condition a VLA on touch, but the missing random-conditioning ablation leaves its central claim about tactile content unproven. the 3 major comments →

arxiv 2603.14604 v2 pith:XOVJ6V6V submitted 2026-03-15 cs.RO cs.CVcs.LG

Tactile Modality Fusion for Vision-Language-Action Models

Charlotte Morissette , Amin Abyaneh , Wei-Di Chang , Anas Houssaini , David Meger , Hsiu-Chin Lin , Jonathan Tremblay , Gregory Dudek This is my paper
classification cs.RO cs.CVcs.LG
keywords Tactile sensingVision-language-action modelsFiLM conditioningModality fusionContact-rich manipulationPeg insertionPretrained tactile representationsLoRA finetuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a pretrained vision-language-action (VLA) policy can be given a working sense of touch without retraining a large encoder or lengthening its input sequence. Instead of appending tactile tokens, TacFiLM pools an embedding from a frozen pretrained tactile encoder and uses it to generate per-channel scale and shift parameters that reweight intermediate visual features, a mechanism called feature-wise linear modulation (FiLM). On over 700 real-robot rollouts of peg and cable insertion, the authors report that this touch-conditioned policy consistently beats both a vision-only VLA and a tactile-concatenation VLA in success rate, direct-insertion rate, lower peak contact force, and faster completion, including on out-of-distribution shapes and under degraded camera conditions. The sympathetic reading of the paper is that feature-level conditioning is a lightweight, effective way to fuse touch into large behaviour models.

Core claim

The central claim is that conditioning a VLA's intermediate visual features on pretrained tactile representations — rather than concatenating tactile tokens into the language-model context — produces a more reliable and contact-sensitive insertion policy. TacFiLM applies FiLM to each ViT block of the visual encoder: a pooled tactile embedding is projected by an MLP into per-channel (gamma, beta) parameters, the visual features are transformed as F ⊙ (1+γ) + β, and the parameters are zero-initialised so the intervention starts as an identity map. The paper reports, for example, 100% success on the 3 mm circle-peg task, a 30-point improvement on USB plugging, a 50-point improvement on out-of-d

What carries the argument

Feature-wise Linear Modulation (FiLM) — a conditioning layer that modifies an intermediate feature map along its channels using learned scale and shift (gamma, beta). In TacFiLM, the tactile observation (two time-separated DIGIT frames, channel-concatenated, background removed, resized to 224×224) is encoded by a frozen pretrained tactile encoder (Sparsh-DINO), pooled to a single embedding, and an MLP projects it to gamma and beta for each ViT block of the fused DINOv2/SigLIP visual backbone. The transformation is applied after normalisation and before multi-head self-attention, with zero initialisation so training begins from the vision-language prior. This carries the argument because it i

Load-bearing premise

The load-bearing premise is that the frozen pretrained tactile encoder (Sparsh-DINO) produces embeddings that actually carry the contact-state information insertion needs, even though it was selected on proxy binary classification tasks rather than by downstream policy success; if those embeddings are uninformative for insertion contact, the FiLM conditioning signal has nothing useful to inject.

What would settle it

Train and deploy TacFiLM exactly as described but replace the tactile embedding with a constant vector, or shuffle tactile embeddings across timesteps so they carry no contact information. If success, force, and completion-time metrics stay at the reported TacFiLM levels, the gains are not caused by tactile content; if they fall toward vision-only levels, the tactile representation is doing the work.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Tactile feedback can be added to a pretrained VLA through post-training LoRA finetuning, keeping the input sequence the same length as vision-only.
  • On the reported insertion benchmarks, the touch-conditioned policy raises success and direct-insertion rates while lowering peak contact forces and completion times compared with vision-only and concatenation fusions.
  • Tactile conditioning appears to help most where vision is least informative: out-of-distribution HDMI plugging rises from near-zero to 66.7% success, and performance holds under dimmed or partially frozen camera input.
  • The fusion-location ablation suggests the method is insensitive to exactly where FiLM is inserted in the ViT stack, making it easy to apply without architecture search.
  • Multiple pretrained tactile encoders (T3 and the Sparsh variants) are compatible with the fusion mechanism, with selection possible via simple proxy classification tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the identity-initialised FiLM mechanism is the source of the gains, the same conditioning pattern could carry other auxiliary signals — proprioception, joint torques, or estimated external forces — into VLA policies without retraining the backbone; this is a natural extension the paper leaves implicit.
  • The encoder choice is made on proxy classification benchmarks (Rotation-High/Rotation-Low/Contact), not on downstream insertion success, so an untested corollary is that insertion performance should track encoder quality on those proxies; that is a testable prediction beyond the paper's reported experiments.
  • The pattern of larger gains on harder, out-of-distribution, and camera-degraded tasks suggests the value of touch grows as visual information degrades; sweeping camera degradation levels quantitatively would test that directly.
  • The paper's own limitation section notes that only the OpenVLA-OFT backbone was used and that a lack of precise visuotactile simulators limited the task set; whether FiLM fusion transfers to other VLA backbones such as flow-based policies or to tasks beyond insertion remains open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TacFiLM, a lightweight modality-fusion method that conditions intermediate visual features of a pretrained vision-language-action model (OpenVLA-OFT, Llama2-7B) on a pooled embedding from a frozen pretrained tactile encoder (Sparsh-DINO) through feature-wise linear modulation (FiLM; Eq. 1). No extra tokens are added to the language backbone; adaptation is by LoRA plus small FiLM MLPs. Real-robot experiments on a Franka Panda with a DIGIT sensor, over 700 rollouts, cover in-distribution circle-peg (2/3 mm clearance) and USB plugging and out-of-distribution square/pentagon peg and HDMI plugging. Against a vision-only baseline and a tactile-concatenation baseline, the paper reports success rate, direct-insertion percentage, average maximum force, and completion time, plus ablations of FiLM placement, camera degradation, and tactile-encoder choice (the last on classification proxies only). The central claim is that TacFiLM yields consistent improvements across all metrics in both in-distribution and out-of-distribution settings.

Significance. If the results hold, the paper makes a useful contribution: a token-efficient, post-training route to adding touch to a 7B-parameter VLA, with practically meaningful reductions in interaction force and better robustness under visual degradation. The strengths are real: a genuine real-robot evaluation with 700+ rollouts, consistent average gains across tasks, OOD generalization checks, camera-degradation robustness, and careful ablations of FiLM placement. I also credit the authors for an honest limitations paragraph (§5.1). Against this, the causal attribution of the gains to tactile content is not yet established, the success-rate differences carry no uncertainty quantification, and the encoder comparison is only on proxy tasks. These are fixable and do not, in my reading, falsify the central claim, but they currently bound its strength.

major comments (3)
  1. [§4.3, Eq. (1)] The ablation suite varies FiLM insertion depth and camera conditions, but never the information content of the conditioning signal. Because the tactile stream is reduced to a single pooled vector z, a control with a constant, random, or non-tactile conditioning input is needed to distinguish gains caused by tactile information from gains caused by the extra FiLM MLP capacity or by the FiLM mechanism itself. The camera-degradation results are suggestive but not decisive for this attribution. Without this control, the abstract's and conclusion's causal phrasing ('integrating tactile signals', 'allows the policy to...') is stronger than the evidence. This is the main gap I see, and it is directly fixable.
  2. [§4.2, Table 1] Headline success rates are point estimates from 15–30 rollouts with no confidence intervals or significance tests. For example, Circle-Peg (2 mm) is 86.67% vs. 73.33% at n=30, and every OOD row has n=15 per method; the 66.67% vs. 6.67% HDMI result, while suggestive, would benefit from an exact binomial interval. Force and time include standard deviations but no group comparisons. Given the paper's central claim of 'consistent improvements', I ask for per-condition binomial CIs or tests (e.g., Fisher's exact) for success and appropriate tests for force/time, or an explicit power analysis.
  3. [§4.3 / Table 3; §3.2] Q1 asks how different pretrained tactile encoders influence downstream policy performance, and §3.2 claims the method is 'agnostic' to the encoder, but the downstream policy experiments use only Sparsh-DINO. All other candidates (T3, Sparsh-MAE, Sparsh-IJEPA) are compared only on the three binary classification proxies of Table 3, and the ranking there is not validated on manipulation success. Since §5 also claims to have 'evaluated multiple pretrained tactile representations', a downstream comparison with at least one additional encoder (or an explicit argument for why the proxy transfer should hold) is needed to support Q1 and the encoder-agnostic statement.
minor comments (4)
  1. [Abstract] The first abstract paragraph mentions evaluation on 'insertion and drawer opening tasks', but no drawer-opening experiment appears in Section 4; the full-text abstract correctly says 'insertion tasks'. Please correct the inconsistency.
  2. [Table 1 (OOD block)] In the OOD block, several rows merge two task names (e.g., 'Circle-Peg (3mm) Square-Peg (3mm)', 'USB-Cable-Plug HDMI-Cable-Plug'). Combined with the stated 225 OOD rollouts at 15 per method, the task-to-row mapping is ambiguous. Please put each task in its own row or otherwise state the per-task rollout counts.
  3. [§4.2, Table 1] Table 1 does not specify whether force and completion time are averaged over all rollouts or only successful ones; Fig. 5 restricts force to 'successfully recovered insertions'. Please state the conditioning explicitly and note the selection bias it introduces.
  4. [§3.1, Eq. (1)] F_n is not defined explicitly; please state that it is the normalized feature tensor of ViT block n and clarify whether (gamma, beta) are shared over spatial positions. Also, 'direct task performance' in the abstract should read 'direct insertion percentage' for consistency.

Circularity Check

0 steps flagged

No significant circularity; TacFiLM's claims rest on real-robot rollouts, not on fitted constants or self-citation chains.

full rationale

The paper's central claim is an empirical comparison: a FiLM-conditioned VLA improves success rate, direct insertion rate, completion time, and force stability relative to vision-only and tactile-concatenation baselines. These results come from over 700 physical rollouts on held-out insertion and OOD tasks, not from any derivation that reduces to its inputs. The tactile encoder is selected on a separate binary classification benchmark (Rotation-High, Rotation-Low, Contact; Table 3) and then used in downstream policy experiments; this is model selection on a proxy task, not a prediction of the same manipulation metrics, so it is not circular. The FiLM scale/shift parameters are learned via LoRA fine-tuning on demonstrations, not fitted to the evaluation numbers. Background self-citations (e.g., visuotactile RL, STS sensor work) are contextual and not load-bearing for the central claim. The lack of a random/constant tactile embedding control is an experimental attribution concern, but it is not a case where the paper defines its result in terms of its input or derives the conclusion from the fit. The paper is self-contained in the relevant sense: its evidence is external rollout data, not a self-referential derivation chain.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entity, conserved quantity, or new force is introduced. The central claim rests on standard supervised learning plus several domain assumptions about transferability of frozen pretrained encoders, stability of LoRA finetuning, and validity of the real-robot insertion benchmark. The tactile encoder choice is effectively a hand-selected design decision based on proxy classification accuracy.

free parameters (3)
  • Tactile encoder model selection = Sparsh-DINO
    Selected via classification accuracy on Rotation-High, Rotation-Low, and Contact proxy tasks (Table 3) rather than by downstream insertion success; a different encoder choice could change results.
  • FiLM insertion depth = All ViT blocks by default
    Conditioning all visual encoder blocks is a hand-set architecture choice; the ablation shows similar performance across early/middle/late blocks, so the default is not uniquely motivated.
  • Training budget = 80k steps
    All deployed methods are trained to 80k steps with no learning curves or convergence analysis, so comparisons are contingent on this budget.
axioms (5)
  • domain assumption Pretrained tactile representations (Sparsh-DINO, T3) retain contact-relevant features for insertion tasks when frozen.
    The method never fine-tunes the tactile encoder; its transferability is assumed from Section 3.2 and probed only by proxy classification tasks in Section 4.3.
  • domain assumption Zero-init FiLM conditioning (gamma=beta=0) starts near identity and preserves pretrained VLA visual-language priors during finetuning.
    Section 3.1 states conditioning starts near identity, but no experiment isolates whether the base VLA priors are actually preserved.
  • domain assumption LoRA finetuning of linear layers is sufficient to adapt the pretrained VLA to tactile-conditioned insertion tasks without harmful forgetting.
    Section 3.3 adopts LoRA as the efficient finetuning recipe; no forgetting or representation-drift analysis is provided.
  • domain assumption Real-robot peg/cable insertion with the object already grasped is a valid proxy for contact-rich manipulation generalization.
    Section 4.1 explicitly excludes grasping and evaluates only insertion; conclusions are scoped to that setting.
  • standard math Autoregressive next-token action decoding, as used by OpenVLA, is a sound action representation for contact-rich tasks.
    The paper relies on the standard OpenVLA action-token pipeline without re-deriving it.

pith-pipeline@v1.3.0-alltime-deepseek · 14728 in / 13709 out tokens · 138929 ms · 2026-08-02T18:08:14.358123+00:00 · methodology

0 comments
read the original abstract

We propose TacFiLM, a lightweight modality-fusion approach that integrates visual-tactile signals into vision-language-action (VLA) models. While advances in VLAs have introduced robot policies that are both generalizable and semantically grounded, these models mainly rely on vision-based perception. Vision alone, however, cannot capture the complex interaction dynamics that occur during contact-rich manipulation, including contact forces, surface friction, compliance, and shear. While recent attempts to integrate tactile signals into VLA models often increase complexity through token concatenation or large-scale pretraining, the heavy computational demands of behaviour models necessitate lightweight fusion strategies. To address these challenges, TacFiLM outlines a post-training finetuning approach that conditions intermediate visual features on pretrained tactile representations using feature-wise linear modulation (FiLM). Experimental results on insertion and drawer opening tasks demonstrate consistent improvements in success rate, direct task performance, completion time, and force stability across both in-distribution and out-of-distribution tasks. Together, these results support our method as an effective approach to integrating tactile signals into VLA models, improving contact-rich manipulation behaviours. Project page: https://charliem7.github.io/projects/TacFilm/

Figures

Figures reproduced from arXiv: 2603.14604 by Amin Abyaneh, Anas Houssaini, Charlotte Morissette, David Meger, Gregory Dudek, Hsiu-Chin Lin, Jonathan Tremblay, Wei-Di Chang.

Figure 1
Figure 1. Figure 1: TacFiLM Overview We present TacFiLM, a lightweight modality-fusion approach for integrating visual-tactile signals into VLA models. The left panel shows the model inputs, including tactile, visual, and language modalities. In grey, baseline approaches: a vision-only VLA and a tactile-concatenation architecture. To the right, we show our proposed TacFiLM-augmented VLA, where FiLM layers condition inter￾medi… view at source ↗
Figure 2
Figure 2. Figure 2: TacFiLM’s modality fusion pipeline. Tactile embeddings are integrated into the vision backbone immediately preceding the multi-head attention layers. The resulting multimodal tokens, combined with language inputs, serve as the basis for action generation within the Llama backbone. approach because it is computationally light, provides an interpretable global tactile bias, eliminates the need to append addi… view at source ↗
Figure 3
Figure 3. Figure 3: Task definitions. Insertion tasks are characterized by different shapes and clearances of the peg or connector. The shared goal across all tasks is characterized by successful insertion of the peg or connector. manipulation capabilities. The benchmark of peg insertions and cable plugging spans multiple object geometries, varying peg and connector shapes, and intro￾duces different levels of difficulty by va… view at source ↗
Figure 4
Figure 4. Figure 4: Experiment setup (left) and sample rollouts (right). [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Force and task completion time analysis. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0

    Future tactile prediction applied to intermediate action-expert features, rather than visual-language or final-action features, improves contact-rich manipulation in SmolVLA and π0.

  2. ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

    cs.RO 2026-07 conditional novelty 5.0

    An action-conditioned visuo-tactile world model generates synthetic camera-plus-touch rollouts that, mixed with real demonstrations, improve downstream contact-rich manipulation policies.

  3. AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning

    cs.RO 2026-06 unverdicted novelty 5.0

    Presents arm-worn AetheRock hardware for multi-modal data collection and ForceVT learning method to improve tactile inference robustness despite sensor variations.

Reference graph

Works this paper leans on

73 extracted references · 34 linked inside Pith · cited by 3 Pith papers

  1. [1]

    arXiv preprint arXiv:2204.01691 (2022)

    Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al.: Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022)

    Pith/arXiv arXiv 2022

  2. [2]

    Advances in neural information processing systems35, 23716– 23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Men- sch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716– 23736 (2022)

    2022

  3. [3]

    arXiv preprint arXiv:2507.17294 (2025)

    Bi, J., Ma, K.Y., Hao, C., Shou, M.Z., Soh, H.: Vla-touch: Enhancing vision-language-action models with dual-level tactile feedback. arXiv preprint arXiv:2507.17294 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    arXiv preprint arXiv:2410.24164 (2024)

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al.:π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 (2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Psychological science15(6), 397–402 (2004)

    Blake, R., Sobel, K.V., James, T.W.: Neural synergy between kinetic vision and touch. Psychological science15(6), 397–402 (2004)

    2004

  6. [6]

    arXiv preprint arXiv:2212.06817 (2022)

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakr- ishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.: Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 (2022)

    Pith/arXiv arXiv 2022

  7. [7]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    1901

  8. [8]

    IEEE Robotics and Automation Letters3(4), 3300–3307 (2018)

    Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E.H., Levine, S.: More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters3(4), 3300–3307 (2018)

    2018

  9. [9]

    arXiv preprint arXiv:2305.18565 (2023)

    Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Ruiz, C.R., Goodman, S., Wang, X., Tay, Y., et al.: Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565 (2023)

    Pith/arXiv arXiv 2023

  10. [10]

    arXiv preprint arXiv:2209.06794 (2022)

    Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A.J., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.: Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 (2022)

    Pith/arXiv arXiv 2022

  11. [11]

    Information Fusion p

    Cheng, N., Xu, J., Guan, C., Gao, J., Wang, W., Li, Y., Meng, F., Zhou, J., Fang, B., Han, W.: Touch100k: A large-scale touch-language-vision dataset for touch- centric multimodal representation. Information Fusion p. 103305 (2025)

    2025

  12. [12]

    arXiv preprint arXiv:2508.08706 (2025)

    Cheng, Z., Zhang, Y., Zhang, W., Li, H., Wang, K., Song, L., Zhang, H.: Omnivtla: Vision-tactile-language-action model with semantic-aligned tactile sensing. arXiv preprint arXiv:2508.08706 (2025)

    arXiv 2025

  13. [13]

    Journal of machine learning research24(240), 1– 113 (2023)

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling lan- guage modeling with pathways. Journal of machine learning research24(240), 1– 113 (2023)

    2023

  14. [14]

    arXiv preprint arXiv:2510.06339 (2025)

    Cui, L., Zhao, Z., Xie, S., Zhang, W., Han, Z., Zhu, Y.: Vi-tacman: Articulated object manipulation via vision and touch. arXiv preprint arXiv:2510.06339 (2025)

    arXiv 2025

  15. [15]

    In: 2021 IEEE International Conference on Robotics and Automation (ICRA)

    Dong, S., Jha, D.K., Romeres, D., Kim, S., Nikovski, D., Rodriguez, A.: Tactile- rl for insertion: Generalization to objects of unknown geometry. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 6437–6443. IEEE (2021)

    2021

  16. [16]

    Morissette et al

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is 16 C. Morissette et al. worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  17. [17]

    arXiv preprint arXiv:2303.03378 (2023)

    Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    arXiv preprint arXiv:2303.07280 (2023)

    Du, Y., Konyushkova, K., Denil, M., Raju, A., Landon, J., Hill, F., De Fre- itas, N., Cabi, S.: Vision-language models as success detectors. arXiv preprint arXiv:2303.07280 (2023)

    Pith/arXiv arXiv 2023

  19. [19]

    arXiv preprint arXiv:2502.12191 (2025)

    Feng, R., Hu, J., Xia, W., Gao, T., Shen, A., Sun, Y., Fang, B., Hu, D.: Any- touch: Learning unified static-dynamic representation across multiple visuo-tactile sensors. arXiv preprint arXiv:2502.12191 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    arXiv preprint arXiv:2602.09617 (2026)

    Feng, R., Zhou, Y., Mei, S., Zhou, D., Wang, P., Cui, S., Fang, B., Yao, G., Hu, D.: Anytouch 2: General optical tactile representation learning for dynamic tactile perception. arXiv preprint arXiv:2602.09617 (2026)

    arXiv 2026

  21. [21]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    George, A., Gano, S., Katragadda, P., Farimani, A.B.: Vital pretraining: Visuo- tactile pretraining for tactile and non-tactile manipulation policies. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 258–264. IEEE (2025)

    2025

  22. [22]

    arXiv preprint arXiv:2502.19638 (2025)

    Gupta, H., Mo, Y., Jin, S., Yuan, W.: Sensor-invariant tactile representation. arXiv preprint arXiv:2502.19638 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    In: 2022 International Conference on Robotics and Automation (ICRA)

    Hansen, J., Hogan, F., Rivkin, D., Meger, D., Jenkin, M., Dudek, G.: Visuotactile- rl: Learning multimodal manipulation policies with deep reinforcement learning. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 8298–

    2022

  24. [24]

    arXiv preprint arXiv:2503.08548 (2025)

    Hao, P., Zhang, C., Li, D., Cao, X., Hao, X., Cui, S., Wang, S.: Tla: Tactile-language-action model for contact-rich manipulation. arXiv preprint arXiv:2503.08548 (2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Heng, L., Geng, H., Zhang, K., Abbeel, P., Malik, J.: Vitacformer: Learning cross-modalrepresentationforvisuo-tactiledexterousmanipulation.arXivpreprint arXiv:2506.15953 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    arXiv preprint arXiv:2410.24090 (2024)

    Higuera, C., Sharma, A., Bodduluri, C.K., Fan, T., Lancaster, P., Kalakrishnan, M.,Kaess,M.,Boots,B.,Lambeta,M.,Wu,T.,etal.:Sparsh:Self-supervisedtouch representations for vision-based tactile sensing. arXiv preprint arXiv:2410.24090 (2024)

    Pith/arXiv arXiv 2024

  27. [27]

    In: 2018 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS)

    Hogan, F.R., Bauza, M., Canal, O., Donlon, E., Rodriguez, A.: Tactile regrasp: Grasp adjustments via simulated tactile transformations. In: 2018 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS). pp. 2963–2970. IEEE (2018)

    2018

  28. [28]

    In: Proceedings of the IEEE/CVF winter conference on applications of computer vision

    Hogan, F.R., Jenkin, M., Rezaei-Shoshtari, S., Girdhar, Y., Meger, D., Dudek, G.: Seeing through your skin: Recognizing objects with a novel visuotactile sensor. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1218–1227 (2021)

    2021

  29. [29]

    ICLR1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)

    2022

  30. [30]

    arXiv preprint arXiv:2507.09160 (2025)

    Huang, J., Wang, S., Lin, F., Hu, Y., Wen, C., Gao, Y.: Tactile-vla: unlocking vision-language-action model’s physical knowledge for tactile generalization. arXiv preprint arXiv:2507.09160 (2025)

    Pith/arXiv arXiv 2025

  31. [31]

    arXiv preprint arXiv:2504.16054 (2025) TacFiLM 17

    Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., et al.:π0.5: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054 (2025) TacFiLM 17

    Pith/arXiv arXiv 2025

  32. [32]

    In: International conference on machine learning

    Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)

    2021

  33. [33]

    Nature Reviews Neuroscience10(5), 345–359 (2009)

    Johansson, R.S., Flanagan, J.R.: Coding and use of tactile signals from the fin- gertips in object manipulation tasks. Nature Reviews Neuroscience10(5), 345–359 (2009)

    2009

  34. [34]

    arXiv preprint arXiv:2501.04693 (2025)

    Jones, J., Mees, O., Sferrazza, C., Stachowicz, K., Abbeel, P., Levine, S.: Beyond sight: Finetuning generalist robot policies with heterogeneous sensors via language grounding. arXiv preprint arXiv:2501.04693 (2025)

    Pith/arXiv arXiv 2025

  35. [35]

    arXiv preprint arXiv:2302.12766 (2023)

    Karamcheti, S., Nair, S., Chen, A.S., Kollar, T., Finn, C., Sadigh, D., Liang, P.: Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12766 (2023)

    Pith/arXiv arXiv 2023

  36. [36]

    arXiv preprint arXiv:2502.19645 (2025)

    Kim, M.J., Finn, C., Liang, P.: Fine-tuning vision-language-action models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645 (2025)

    Pith/arXiv arXiv 2025

  37. [37]

    arXiv preprint arXiv:2406.09246 (2024)

    Kim, M.J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al.: Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    IEEE Robotics and Automation Letters5(3), 3838–3845 (2020)

    Lambeta, M., Chou, P.W., Tian, S., Yang, B., Maloon, B., Most, V.R., Stroud, D., Santos, R., Byagowi, A., Kammerer, G., et al.: Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters5(3), 3838–3845 (2020)

    2020

  39. [39]

    In: 2019 International conference on robotics and automation (ICRA)

    Lee, M.A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., Bohg, J.: Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In: 2019 International conference on robotics and automation (ICRA). pp. 8943–8950. IEEE (2019)

    2019

  40. [40]

    In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Li, J., Wu, T., Zhang, J., Chen, Z., Jin, H., Wu, M., Shen, Y., Yang, Y., Dong, H.: Adaptive visuo-tactile fusion with predictive force attention for dexterous manip- ulation. In: 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3232–3239. IEEE (2025)

    2025

  41. [41]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    2023

  42. [42]

    In: International confer- ence on machine learning

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International confer- ence on machine learning. pp. 12888–12900. PMLR (2022)

    2022

  43. [43]

    arXiv preprint arXiv:2505.08194 (2025)

    Ma, W., Cao, X., Zhang, Y., Zhang, C., Yang, S., Hao, P., Fang, B., Cai, Y., Cui, S., Wang, S.: Cltp: Contrastive language-tactile pre-training for 3d contact geometry understanding. arXiv preprint arXiv:2505.08194 (2025)

    Pith/arXiv arXiv 2025

  44. [44]

    arXiv preprint arXiv:2203.12601 (2022)

    Nair, S., Rajeswaran, A., Kumar, V., Finn, C., Gupta, A.: R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 (2022)

    Pith/arXiv arXiv 2022

  45. [45]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    Pith/arXiv arXiv 2023

  46. [46]

    In: 2024 IEEE International Conference on Robotics and Automation (ICRA)

    O’Neill, A., Rehman, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., et al.: Open x-embodiment: Robotic learn- ing datasets and rt-x models: Open x-embodiment collaboration 0. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). pp. 6892–6903. IEEE (2024) 18 C. Morissette et al

    2024

  47. [47]

    In: Proceedings of the AAAI conference on artificial intelligence

    Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual rea- soning with a general conditioning layer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

    2018

  48. [48]

    In: Conference on Robot Learning

    Qi, H., Yi, B., Suresh, S., Lambeta, M., Ma, Y., Calandra, R., Malik, J.: General in-hand object rotation with vision and touch. In: Conference on Robot Learning. pp. 2549–2564. PMLR (2023)

    2023

  49. [49]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  50. [50]

    In: Conference on robot learning

    Shah, D., Osiński, B., Levine, S., et al.: Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action. In: Conference on robot learning. pp. 492–504. pmlr (2023)

    2023

  51. [51]

    arXiv preprint arXiv:2406.16851 (2024)

    Sharma, A., Saxon, M., Wang, W.Y.: Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts. arXiv preprint arXiv:2406.16851 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    The International Journal of Robotics Research40(12-14), 1385–1401 (2021)

    She, Y., Wang, S., Dong, S., Sunil, N., Rodriguez, A., Adelson, E.: Cable ma- nipulation with a tactile-reactive gripper. The International Journal of Robotics Research40(12-14), 1385–1401 (2021)

    2021

  53. [53]

    In: Conference on robot learning

    Shridhar, M., Manuelli, L., Fox, D.: Cliport: What and where pathways for robotic manipulation. In: Conference on robot learning. pp. 894–906. PMLR (2022)

    2022

  54. [54]

    arXiv preprint arXiv:2303.00905 (2023)

    Stone, A., Xiao, T., Lu, Y., Gopalakrishnan, K., Lee, K.H., Vuong, Q., Wohlhart, P., Kirmani, S., Zitkovich, B., Xia, F., et al.: Open-world object manipulation using pre-trained vision-language models. arXiv preprint arXiv:2303.00905 (2023)

    Pith/arXiv arXiv 2023

  55. [55]

    arXiv preprint arXiv:2405.12213 (2024)

    Team, O.M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al.: Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 (2024)

    Pith/arXiv arXiv 2024

  56. [56]

    arxiv 2023

    Touvron,H.,Lavril,T.,Izacard,G.,Martinet,X.,Lachaux,M.A.,Lacroix,T.,Roz- ière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foun- dation language models. arxiv 2023. arXiv preprint arXiv:2302.1397110(2023)

    Pith/arXiv arXiv 2023

  57. [57]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    Pith/arXiv arXiv 2023

  58. [58]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    2017

  59. [59]

    Wang, Z., Yu, W., Ren, X., Zhang, J., Zhao, Y., Saxena, R., Cheng, L., Wong, G., See, S., Minervini, P., et al.: Mmlongbench: Benchmarking long-context vision-languagemodelseffectivelyandthoroughly.arXivpreprintarXiv:2505.10610 (2025)

    arXiv 2025

  60. [60]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  61. [61]

    arXiv preprint arXiv:2303.11765 (2023)

    Wilson, A., Jiang, H., Lian, W., Yuan, W.: Cable routing and assembly using tactile-driven motion primitives. arXiv preprint arXiv:2303.11765 (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, F., Feng, C., Chen, Z., Park, H., Wang, D., Dou, Y., Zeng, Z., Chen, X., Gangopadhyay, R., Owens, A., et al.: Binding touch to everything: Learning unified multimodal tactile representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26340–26353 (2024) TacFiLM 19

    2024

  63. [63]

    arXiv preprint arXiv:2505.22159 (2025)

    Yu, J., Liu, H., Yu, Q., Ren, J., Hao, C., Ding, H., Huang, G., Huang, G., Song, Y., Cai, P., et al.: Forcevla: Enhancing vla models with a force-aware moe for contact-rich manipulation. arXiv preprint arXiv:2505.22159 (2025)

    arXiv 2025

  64. [64]

    arXiv preprint arXiv:2405.02794 (2024)

    Yu, S., Lin, K., Xiao, A., Duan, J., Soh, H.: Octopi: Object property reasoning with large tactile-language models. arXiv preprint arXiv:2405.02794 (2024)

    Pith/arXiv arXiv 2024

  65. [65]

    Sensors17(12), 2762 (2017)

    Yuan, W., Dong, S., Adelson, E.H.: Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors17(12), 2762 (2017)

    2017

  66. [66]

    In: 2024 IEEE International Conference on Robotics and Automation (ICRA)

    Yuan, Y., Che, H., Qin, Y., Huang, B., Yin, Z.H., Lee, K.W., Wu, Y., Lim, S.C., Wang, X.: Robot synesthesia: In-hand manipulation with visuotactile sensing. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). pp. 6558–6565. IEEE (2024)

    2024

  67. [67]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975–11986 (2023)

    2023

  68. [68]

    arXiv preprint arXiv:2505.09577 (2025)

    Zhang, C., Hao, P., Cao, X., Hao, X., Cui, S., Wang, S.: Vtla: Vision-tactile- language-action model with preference learning for insertion manipulation. arXiv preprint arXiv:2505.09577 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    arXiv preprint arXiv:2304.08587 (2023)

    Zhang, X., Ding, Y., Amiri, S., Yang, H., Kaminski, A., Esselink, C., Zhang, S.: Grounding classical task planners via vision-language models. arXiv preprint arXiv:2304.08587 (2023)

    Pith/arXiv arXiv 2023

  70. [70]

    arXiv preprint arXiv:2601.20239 (2026)

    Zhang, Z., Ma, J., Yang, X., Wen, X., Zhang, Y., Li, B., Qin, Y., Liu, J., Zhao, C., Kang, L., et al.: Touchguide: Inference-time steering of visuomotor policies via touch guidance. arXiv preprint arXiv:2601.20239 (2026)

    Pith/arXiv arXiv 2026

  71. [71]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Zhao,J.,Kuppuswamy,N.,Feng,S.,Burchfiel,B.,Adelson,E.:Polytouch:Arobust multi-modaltactilesensorforcontact-richmanipulationusingtactile-diffusionpoli- cies. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). pp. 104–110. IEEE (2025)

    2025

  72. [72]

    arXiv preprint arXiv:2406.13640 (2024)

    Zhao, J., Ma, Y., Wang, L., Adelson, E.H.: Transferable tactile transform- ers for representation learning across diverse sensors and tasks. arXiv preprint arXiv:2406.13640 (2024)

    Pith/arXiv arXiv 2024

  73. [73]

    In: Conference on Robot Learning

    Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. In: Conference on Robot Learning. pp. 2165–2183. PMLR (2023)

    2023