Pith. sign in

REVIEW 4 major objections 4 minor 12 cited by

Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Finetuning generalist robot policies with touch and audio via language grounding raises task success by over 20 percent.

desk verdict FuSe's action-success results are solid and believable, but the 'zero-shot descriptions' headline capability is never directly evaluated; the paper deserves review, but the claims need fixing. read the letter →

arxiv 2501.04693 v3 pith:WY6ZE7WI submitted 2025-01-08 cs.RO cs.AI

Joshua Jones , Oier Mees , Carmelo Sferrazza , Kyle Stachowicz , Pieter Abbeel , Sergey Levine This is my paper · ORCID
classification cs.ROcs.AI
keywords robotmanipulationmultimodalfinetuninglanguagegroundingtactilesensingaudiogeneralistpoliciesvision-language-actionmodelszero-shotprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a robot policy pre-trained to act from vision and language can be extended to use touch and audio with only a modest amount of new data, if natural language is used as the shared bridge between all sensors. The authors propose FuSe, a finetuning recipe that adds two auxiliary losses to the standard action-matching loss: a contrastive loss that aligns each sensory view with matched language instructions, and a generative loss that predicts the instruction from the sensory observation. On a real robot, FuSe policies outperform vision-only finetuning and from-scratch training by over 20 percent, and they gain zero-shot abilities such as following prompts like "pick the round object that feels squishy" or "press the button that plays piano." The same recipe transfers to a 3-billion-parameter vision-language-action model, not just the transformer-based generalist policy used for the main experiments.

What carries the argument

The central mechanism is the pair of auxiliary losses that ground every new sensor in language. A multimodal contrastive loss takes observation embeddings built from any subset of vision, touch, and audio, and pulls them toward matched language instructions while pushing them away from unmatched ones; a language-generation head then predicts the instruction text from those same embeddings, using one shared transformer with modality tokens to distinguish inputs. Together these losses force the new sensors to attach to meanings the pre-trained policy already understands, which is what allows zero-shot cross-modal prompting without joint multimodal-action training data. A pre-trained tactile encoder and a spectrogram-based audio encoder provide the raw sensor features that feed this grounding process.

What would settle it

Give FuSe the same training data but replace every touch- and audio-related phrase in the instructions with random or visual-only language, then test on prompts such as "pick the object that feels corded"; if the policy still succeeds, the claimed language grounding is not what drives the result. A cleaner check is to record whether test success on the shopping-bag task drops when the tactile images are corrupted, which would show the policy actually depends on the new sensors.

Watch

Extended reading notes

Core claim

The central claim is that natural language is a sufficient common embedding space to connect vision, touch, and audio for action selection, so that paired multimodal-action data is not required to teach a policy to reason across senses. FuSe finetunes a pre-trained generalist policy with a behavior-cloning loss plus a multimodal contrastive loss and a language-generation loss, both computed over every combination of the available modalities. This makes the touch and audio encoders learn to project their signals into the semantic vocabulary the policy already knows from vision and language pre-training. The finetuned policy can then follow multimodal prompts, compose concepts across modalities such as matching a button's sound to an object's color, and describe a grasped object in words, even for objects that were absent from the finetuning data.

Load-bearing premise

The method assumes that language descriptions of what objects feel and sound like carry enough information to teach the policy which sensor signals matter, so that the contrastive and generative losses can substitute for paired cross-modal action data; if the annotations miss the task-relevant sensory features, the new modalities will be ignored.

Editorial extensions

If this is right

  • Adding touch and audio to a vision-only generalist policy raises average task success by over 20 percent, with the largest gain in the occluded shopping-bag scenario where vision is least reliable.
  • FuSe policies can follow zero-shot multimodal prompts that disambiguate objects by touch or sound when a single modality is ambiguous.
  • The same policy can perform compositional cross-modal reasoning, such as pressing the training button that plays the same sound as a novel button, and can describe a grasped object in words without explicit training for that description task.
  • Both auxiliary losses are necessary; ablations show that omitting either the contrastive loss or the generative loss degrades performance, particularly on unseen test objects.
  • The recipe transfers across architectures, working for both a transformer-based generalist policy and a 3-billion-parameter vision-language-action model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence is that the same language-grounding recipe should extend to other rare modalities, such as proprioceptive force cues or thermal sensing, whenever those signals can be described in words; the paper only demonstrates touch and audio.
  • If the auxiliary losses are the active ingredient, then FuSe should also improve when the language annotations are generated automatically from sensor readings, which would reduce the human annotation burden in a way the paper does not evaluate.
  • The gap on unseen objects suggests FuSe learns property-level semantics rather than memorizing training scenes; a stress test that scales the number of objects and distractors would show whether the 20 percent advantage persists as scene complexity grows.
  • The generative head could be reused as an automatic annotator: a FuSe policy that can describe what it grasps could label new multimodal demonstrations, bootstrapping larger training sets for future multimodal policies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes FuSe, a fine-tuning recipe that extends pre-trained generalist visuomotor policies (Octo and a 3B PaliGemma-based VLA) to heterogeneous sensor modalities, specifically touch and audio, by using natural language as a common cross-modal grounding. The training objective combines the original action loss with a multimodal CLIP-style contrastive loss and a language generation loss. The authors collect and release a 26,866-trajectory real-world dataset with vision, touch, audio, proprioception, action, and language annotations across tabletop grasping, shopping-bag grasping, and button-pressing tasks. Experiments report that FuSe outperforms vision-only fine-tuning, from-scratch training, and a ResNet baseline, especially in the partially observable shopping-bag task, and that it enables multimodal prompting, compositional cross-modal prompting, and zero-shot descriptions of grasped objects.

Significance. If the claims hold, FuSe is a practical and timely recipe for adapting large generalist robot policies to non-visual sensors without large paired multimodal datasets. The paper's strengths include a large open-sourced real-world dataset, evaluation on unseen test objects, two architecturally different backbone policies, and an ablation showing that both auxiliary losses contribute to action success. The work is potentially useful for the broader robot-learning community, but the current version overclaims one of its three headline capabilities and lacks uncertainty quantification on the main quantitative results.

major comments (4)
  1. [Abstract, Fig. 1, Section IV] The headline capability 'zero-shot descriptions of grasped objects' is never evaluated anywhere in the experiments. Section III introduces the generative loss L_gen, but Section IV reports no metric for generated descriptions (e.g., object retrieval accuracy, semantic similarity, human evaluation), no held-out-object evaluation, and not even qualitative example outputs from the deployed policy. Figure 7 only measures grasp success, and Figure 6 uses the generative head only as an intermediate step in an action-conditioning pipeline. Since all L_gen training annotations are templated and ChatGPT-rephrased, the model could be retrieving memorized template associations; evidence that generated descriptions generalize to unseen objects is required before claiming 'zero-shot descriptions'. Either add a direct evaluation of the generated text or remove/qualify this capability claim.
  2. [Section IV-A, Figures 5-8] The main quantitative results are presented as bar-chart point estimates without error bars, confidence intervals, or significance tests. The text states that each scenario is run for 5 rollouts, but the plotted aggregated success rates do not show variance. This makes the central 'over 20% improvement' claim statistically unquantifiable as stated, and some comparisons appear close (for example, in Table I the touch-ambiguous grasp success is 0.35 in both the tactile-only and vision-plus-tactile conditions). Please report per-condition trial counts and standard errors or confidence intervals, and perform significance tests where appropriate, or explicitly present the results as preliminary.
  3. [Abstract] The phrase 'increase success rates by over 20% compared to all considered baselines' is undefined. It does not specify whether this is an absolute percentage-point increase, a relative improvement, an average over tasks, or the minimum improvement across baselines. The per-task figures show different gaps, so this headline number must be accompanied by the exact aggregation rule and the underlying per-task numbers.
  4. [Section IV-B, IV-C] The use of 'zero-shot' needs a precise definition. The evaluations include unseen test objects, but the language prompts appear to be drawn from the same templated-and-rephrased instruction distribution used during training, so calling the prompting 'zero-shot' is ambiguous: the policy may not be zero-shot with respect to the instruction modality combination. Please define which aspects are unseen at training time (objects, prompts, modality combinations, or task compositions) and state explicitly what was excluded from the training data for each reported capability.
minor comments (4)
  1. [Sections I and III] There are typos in the prose: 'hetereogeneous' in Section I and 'heteregoneous' in Section III should be corrected.
  2. [Section IV-G] The phrase 'the pre-training phrase' should read 'the pre-training phase'; the observation that OXE pretraining appears to harm language understanding is stated but not quantified, so consider adding a brief supporting result or softening the claim.
  3. [Table I] The column layout of Table I is difficult to parse because it combines condition labels with success metrics across two task rows; consider using separate, clearly labeled columns for each condition and metric, and include the number of trials per condition.
  4. [Section IV-F, Figure 7] The ablation in Figure 7 appears to report a single point estimate per condition; please state whether these results are averaged over multiple seeds or rollouts and report the associated variance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed gains are empirical action-success improvements on held-out objects, and the auxiliary losses are ordinary training objectives rather than fitted predictions.

full rationale

FuSe's central claims are empirical: finetuning pretrained generalist policies with a CLIP-style contrastive loss and a language-generation loss improves real-robot task success. These results are quantified on held-out test objects and unseen distractors (Section IV-A, Figures 5-8), so the reported improvements are not forced by the loss definitions. The losses themselves are standard training objectives (L = LBC + βLgen + λLcontrast, Section III), and the evaluation measures robot actions rather than the loss values. The use of pretrained Octo and TVL encoders is independent transfer, not a self-citation chain. The only self-citation of note is the PaliVLA reference [53] for OXE pretraining of the VLA baseline, but that checkpoint choice is not a load-bearing premise and does not define the evaluation outcome. One capability — 'zero-shot descriptions of grasped objects' — is asserted in the abstract and Figure 1 and attributed to Lgen, but Section IV reports no quantitative or qualitative evaluation of generated descriptions; this is a missing-evidence/correctness concern, not circularity, because the claim does not reduce to a fitted input or to a self-cited result. Overall, no derivation step reduces by construction to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. The load-bearing assumptions are that language can bridge heterogeneous modalities and that the chosen encoders and loss weights are effective without extensive tuning.

free parameters (3)
  • language generation loss weight beta = 1
    Chosen by hand and fixed for all experiments; no sensitivity analysis is reported, yet it controls the balance between action prediction and language grounding.
  • contrastive loss weight lambda = 1
    Chosen by hand and fixed for all experiments; the ablation study varies the presence of the loss but not its weight, so the optimal balance is unknown.
  • PaliGemma OXE pretraining steps = 50000
    Selected manually to avoid overtraining on OXE, which the paper notes harms language understanding; the choice is not derived from data.
assumptions (4)
  • domain assumption Natural language can serve as a common cross-modal grounding that connects vision, touch, and audio for robot action selection.
    The whole FuSe recipe relies on this assumption, introduced in Section III, to justify the two auxiliary losses.
  • standard math A CLIP-style contrastive loss maximizes mutual information between observations and language instructions in a way that transfers to unseen compositions.
    The contrastive loss is used as a standard tool, following [49], without further justification that it generalizes to novel modality combinations.
  • domain assumption The pre-trained TVL tactile encoder and the ResNet26 audio spectrogram encoder provide sufficient features for policy fine-tuning on a small dataset.
    The paper uses these encoders expecting that they capture task-relevant touch and audio properties, but their transferability to this domain is not independently validated.
  • domain assumption A dataset of 26,866 trajectories from a single robot setup is sufficient to fine-tune large policies without catastrophic forgetting of pre-trained knowledge.
    The experimental design assumes this scale and diversity are adequate to retain OXE generalization while adding new modalities, which is a core premise of the finetuning recipe.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding." pith.science (2026). https://pith.science/paper/WY6ZE7WI

@misc{pith2026250104693,
  author       = {Pith},
  title        = {Pith review of: Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WY6ZE7WI}},
  note         = {Machine review of arXiv:2501.04693}
}
read the original abstract

Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.

Figures

Figures reproduced from arXiv: 2501.04693 by the authors.

Figure 1
Figure 1. We introduce FuSe, an approach that enables finetuning large image-based pre-trained generalist policies, including vision-language-action [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture: We finetune pre-trained generalist robot policies by tokenizing all heteregoneous sensing modalities and passing them though a pre-trained transformer backbone. Crucially, we apply two auxiliary losses that help connect the semantic knowledge of pre-trained generalist policies with new heterogeneous modalities, such as touch and audio. Concretely, we apply both a contrastive loss that aims to maximize … view at source ↗
Figure 4
Figure 4. Visualization of objects for real-world experiments, including objects [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: FuSe performance on evaluation tasks compared against baselines. Our approach outperforms baselines trained from scratch or finetuned with vision [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Results on the compositional tasks devised in the button pressing [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: We study the effect of the proposed losses in an ablation experiment [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tactile Modality Fusion for Vision-Language-Action Models

    cs.RO 2026-03 conditional novelty 6.0 of 10

    A FiLM-based tactile fusion method that conditions VLA visual features on frozen pretrained touch embeddings improves real-robot insertion success, speed, and force control relative to vision-only and concatenation baselines.

  2. Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

    cs.RO 2025-12 unverdicted novelty 6.0 of 10

    DreamTacVLA grounds VLA models in contact physics by aligning multi-scale vision-tactile inputs and predicting future tactile states, reaching up to 95% success on contact-rich tasks.

  3. Universal Visuo-Tactile Video Understanding for Embodied Interaction

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VTV-LLM is a tactile-video large language model, trained on a new VTV150K dataset, that reasons about hardness, protrusion, elasticity, and friction in natural language.

  4. Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Planners

    cs.RO 2025-05 conditional novelty 6.0 of 10

    RLVR fine-tuning teaches small LLMs to reason about reachability and collisions, letting them beat far larger ungrounded LLMs on multi-robot box-moving tasks.

  5. CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A contrastive pre-training framework and dataset that map tactile 3D point clouds to contact-state language descriptions, enabling zero-shot and LLM-based tactile understanding.

  6. ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy

    cs.RO 2025-02 conditional novelty 6.0 of 10

    ConRFT fine-tunes VLA policies with an offline behavior-cloning plus Q-learning stage and an online consistency-policy RL stage with human corrections, reporting 96.3% average success on eight real-world tasks.

  7. VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedback

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Tactile feedback, provided both as language descriptions for planning and as force signals for action refinement, improves vision-language-action robot policies on contact-rich manipulation tasks.

  8. Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Tactile-VLA fuses tactile sensing into a vision-language-action model so force-related instructions and corrective reasoning transfer to new contact-rich tasks with few demonstrations.

  9. VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

    cs.RO 2025-05 conditional novelty 5.0 of 10

    VTLA, a vision-tactile-language-action model trained purely in simulation, achieves over 90% success on peg insertion in simulation and 95% in the real world.

  10. Modality Selection and Skill Segmentation via Cross-Modality Attention

    cs.RO 2025-04 reject novelty 4.0 of 10

    Attention weights in a cross-modal transformer trained on furniture assembly cluster into distinct patterns for different manipulation primitives, suggesting the possibility of unsupervised skill segmentation, but the...

  11. Foundation Model Driven Robotics: A Comprehensive Review

    cs.RO 2025-07 conditional novelty 2.0 of 10

    A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.

  12. Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience

    q-bio.NC 2025-05 conditional novelty 2.0 of 10

    A survey and taxonomy that organizes AI agentic reasoning into four neuroscience-inspired categories without introducing new empirical results.

Reference graph

Works this paper leans on

53 extracted references · 19 canonical work pages · cited by 12 Pith papers

  1. [1]

    Coding and use of tactile signals from the fingertips in object manipulation tasks,

    R. S. Johansson and J. R. Flanagan, “Coding and use of tactile signals from the fingertips in object manipulation tasks,” Nature Reviews Neuroscience, vol. 10, no. 5, pp. 345–359, 2009

    2009

  2. [2]

    More than a feeling: Learning to grasp and regrasp using vision and touch,

    R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine, “More than a feeling: Learning to grasp and regrasp using vision and touch,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3300–3307, 2018

    2018

  3. [3]

    Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning,

    M. Du, O. Y . Lee, S. Nair, and C. Finn, “Play it by ear: Learning skills amidst occlusion through audio-visual imitation learning,” arXiv preprint arXiv:2205.14850, 2022

    arXiv 2022

  4. [4]

    Octo: An open-source generalist robot policy,

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine, “Octo: An open-source generalist robot policy,” in Proceedings of Robotics: Science and Systems , Delft, Netherlands, 2024

    work page 2024

  5. [5]

    Open- vla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi et al. , “Open- vla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

  6. [6]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

  7. [7]

    Rt-2: Vision- language-action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choro- manski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “Rt-2: Vision- language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023

    arXiv 2023

  8. [8]

    Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation,

    R. Doshi, H. Walke, O. Mees, S. Dasari, and S. Levine, “Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation,” in Conference on Robot Learning , 2024

    work page 2024

Show all 53 references
  1. [9]

    Open X-Embodiment: Robotic learning datasets and RT-X models,

    O. X.-E. Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, B. Kim, B. Sch ¨olkopf, B. Ichter, C. Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. ...

    2024

  2. [10]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, L. Y . Chen, K. Ellis et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024

    2024 arXiv

  3. [11]

    Bridgedata v2: A dataset for robot learning at scale,

    H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, A. He, V . Myers, K. Fang, C. Finn, and S. Levine, “Bridgedata v2: A dataset for robot learning at scale,” in Conference on Robot Learning (CoRL) , 2023

    2023

  4. [12]

    Latent plans for task agnostic offline reinforcement learning,

    E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard, “Latent plans for task agnostic offline reinforcement learning,” in Pro- ceedings of the 6th Conference on Robot Learning (CoRL) , Auckland, New Zealand, 2022

    2022

  5. [13]

    Grounding language with visual affordances over unstructured data,

    O. Mees, J. Borja-Diaz, and W. Burgard, “Grounding language with visual affordances over unstructured data,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , London, UK, 2023

    2023

  6. [14]

    A touch, vision, and language dataset for multimodal alignment,

    L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg, “A touch, vision, and language dataset for multimodal alignment,” in Forty-first International Conference on Machine Learning

  7. [15]

    Maniwav: Learning robot manipulation from in-the-wild audio-visual data,

    Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, B. Burchfiel, and S. Song, “Maniwav: Learning robot manipulation from in-the-wild audio-visual data,” in 8th Annual Conference on Robot Learning

  8. [16]

    Towards com- prehensive multimodal perception: Introducing the touch-language- vision dataset,

    N. Cheng, Y . Li, J. Gao, B. Fang, J. Xu, and W. Han, “Towards com- prehensive multimodal perception: Introducing the touch-language- vision dataset,” arXiv preprint arXiv:2403.09813 , 2024

    2024 arXiv

  9. [17]

    Socratic models: Composing zero-shot multimodal reasoning with language,

    A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani et al. , “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598 , 2022

    2022 arXiv

  10. [18]

    Code as policies: Language model programs for em- bodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for em- bodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500

    2023

  11. [19]

    Do as i can, not as i say: Grounding language in robotic affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman et al. , “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022

    2022 arXiv

  12. [20]

    Visual language maps for robot navigation,

    C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , London, UK, 2023

    2023

  13. [21]

    Audio visual language maps for robot navigation,

    ——, “Audio visual language maps for robot navigation,” in Proceed- ings of the International Symposium on Experimental Robotics (ISER), Chiang Mai, Thailand, 2023

    2023

  14. [22]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  15. [23]

    Bridging language and action: A survey of language-based robot manipulation,

    H. Zhou, X. Yao, O. Mees, Y . Meng, T. Xiao, Y . Bisk, J. Oh, E. Johns, M. Shridhar, D. Shah, J. Thomason, K. Huang, J. Chai, Z. Bing, and A. Knoll, “Bridging language and action: A survey of language-based robot manipulation,” arXiv preprint arXiv:2312.10807 , 2024

    2024 arXiv

  16. [24]

    Binding touch to everything: Learning unified multimodal tactile representations,

    F. Yang, C. Feng, Z. Chen, H. Park, D. Wang, Y . Dou, Z. Zeng, X. Chen, R. Gangopadhyay, A. Owens et al. , “Binding touch to everything: Learning unified multimodal tactile representations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2024

  17. [25]

    The feeling of success: Does touch sensing help predict grasp outcomes?

    R. Calandra, A. Owens, M. Upadhyaya, W. Yuan, J. Lin, E. H. Adelson, and S. Levine, “The feeling of success: Does touch sensing help predict grasp outcomes?” in Conference on Robot Learning . PMLR, 2017, pp. 314–323

    2017

  18. [26]

    Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation,

    T. Bi, C. Sferrazza, and R. D’Andrea, “Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation,” IEEE Robotics and Automation Letters , vol. 6, no. 3, pp. 5761–5768, 2021

    2021

  19. [27]

    Swing- bot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation,

    C. Wang, S. Wang, B. Romero, F. Veiga, and E. Adelson, “Swing- bot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 5633–5640

    2020

  20. [28]

    Paligemma: A versatile 3b vlm for transfer,

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello et al. , “Paligemma: A versatile 3b vlm for transfer,” arXiv preprint arXiv:2407.07726, 2024

    2024 arXiv

  21. [29]

    Robot utility models: General policies for zero-shot deployment in new environments,

    H. Etukuru, N. Naka, Z. Hu, S. Lee, J. Mehu, A. Edsinger, C. Paxton, S. Chintala, L. Pinto, and N. M. M. Shafiullah, “Robot utility models: General policies for zero-shot deployment in new environments,”arXiv preprint arXiv:2409.05865, 2024

    2024 arXiv

  22. [30]

    Robonet: Large-scale multi-robot learning,

    S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, “Robonet: Large-scale multi-robot learning,” arXiv preprint arXiv:1910.11215 , 2019

    1910 arXiv

  23. [31]

    Robotic control via embodied chain-of-thought reasoning,

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” in Con- ference on Robot Learning , 2024

    2024

  24. [32]

    General in-hand object rotation with vision and touch,

    H. Qi, B. Yi, S. Suresh, M. Lambeta, Y . Ma, R. Calandra, and J. Malik, “General in-hand object rotation with vision and touch,” in Conference on Robot Learning . PMLR, 2023, pp. 2549–2564

    2023

  25. [33]

    The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning,

    C. Sferrazza, Y . Seo, H. Liu, Y . Lee, and P. Abbeel, “The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning,” arXiv preprint arXiv:2311.00924, 2023

    2023 arXiv

  26. [34]

    Choosing smartly: Adaptive multimodal fusion for object detection in changing environments,

    O. Mees, A. Eitel, and W. Burgard, “Choosing smartly: Adaptive multimodal fusion for object detection in changing environments,” in Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, 2016

    2016

  27. [35]

    Hearing touch: Audio-visual pretraining for contact-rich manipulation,

    J. Mejia, V . Dean, T. Hellebrekers, and A. Gupta, “Hearing touch: Audio-visual pretraining for contact-rich manipulation,” arXiv preprint arXiv:2405.08576, 2024

    2024 arXiv

  28. [36]

    See, hear, and feel: Smart sensory fusion for robotic manipulation,

    H. Li, Y . Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu, “See, hear, and feel: Smart sensory fusion for robotic manipulation,” arXiv preprint arXiv:2212.03858 , 2022

    2022 arXiv

  29. [37]

    Robot synesthesia: In-hand manipulation with visuotactile sensing,

    Y . Yuan, H. Che, Y . Qin, B. Huang, Z.-H. Yin, K.-W. Lee, Y . Wu, S.-C. Lim, and X. Wang, “Robot synesthesia: In-hand manipulation with visuotactile sensing,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 6558–6565

    2024

  30. [38]

    Connecting touch and vision via cross-modal prediction,

    Y . Li, J.-Y . Zhu, R. Tedrake, and A. Torralba, “Connecting touch and vision via cross-modal prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 10 609–10 618

    2019

  31. [39]

    Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,

    M. A. Lee, Y . Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,” IEEE Transactions on Robotics , vol. 36, no. 3, pp. 582–596, 2020

    2020

  32. [40]

    See to touch: Learning tactile dexterity through visual incentives,

    I. Guzey, Y . Dai, B. Evans, S. Chintala, and L. Pinto, “See to touch: Learning tactile dexterity through visual incentives,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 13 825–13 832

    2024

  33. [41]

    Contrastive multiview coding,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16 . Springer, 2020, pp. 776–794

    2020

  34. [42]

    Learning visuotactile skills with two multifingered hands,

    T. Lin, Y . Zhang, Q. Li, H. Qi, B. Yi, S. Levine, and J. Malik, “Learning visuotactile skills with two multifingered hands,” arXiv preprint arXiv:2404.16823, 2024

    2024 arXiv

  35. [43]

    Integration of vision, force and tactile sensing for grasping,

    P. Miller and P. Leibowitz, “Integration of vision, force and tactile sensing for grasping,” Int. J. Intell. Mach , vol. 4, pp. 129–149, 1999

    1999

  36. [44]

    Octopi: Object property reasoning with large tactile-language models,

    S. Yu, K. Lin, A. Xiao, J. Duan, and H. Soh, “Octopi: Object property reasoning with large tactile-language models,” arXiv preprint arXiv:2405.02794, 2024

    2024 arXiv

  37. [45]

    Esresne (x) t-fbsp: Learning robust time-frequency transformation of audio,

    A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresne (x) t-fbsp: Learning robust time-frequency transformation of audio,” in 2021 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2021, pp. 1–8

    2021

  38. [46]

    Audioclip: Extending clip to image, text and audio,

    ——, “Audioclip: Extending clip to image, text and audio,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 976–980

    2022

  39. [47]

    Ast: Audio spectrogram trans- former,

    Y . Gong, Y .-A. Chung, and J. Glass, “Ast: Audio spectrogram trans- former,” arXiv preprint arXiv:2104.01778 , 2021

    2021 arXiv

  40. [48]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  41. [49]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  42. [50]

    Chatgpt: Optimizing language models for dialogue,

    J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny et al. , “Chatgpt: Optimizing language models for dialogue,” OpenAI blog, vol. 2, no. 4, 2022

    2022

  43. [51]

    Film: Visual reasoning with a general conditioning layer,

    E. Perez, F. Strub, H. De Vries, V . Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in Proceedings of the AAAI conference on artificial intelligence , vol. 32, no. 1, 2018

    2018

  44. [52]

    Bc-z: Zero-shot task generalization with robotic imitation learning,

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning. PMLR, 2022, pp. 991–1002

    2022

  45. [53]

    PaliVLA,

    K. Stachowicz, “PaliVLA,” 2024, GitHub repository. [Online]. Available: https://github.com/kylestach/bigvision-palivla

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.