REVIEW 8 cited by
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.
Forward citations
Cited by 8 Pith papers
-
Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots
A self-supervised multimodal encoder trained with vision, proprioception, and force yields a vision-only latent that recovers end-effector state and force above vision baselines on RH20T, with modest absolute force accuracy.
-
OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies
A policy-agnostic two-stage real-world RL method learns tactile residual corrections on frozen visual policies, lifting contact-rich task success from 5–40% to 85–100% in under 80 minutes.
-
Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation
DreamTacVLA grounds VLA models in contact physics by aligning multi-scale vision-tactile inputs and predicting future tactile states, reaching up to 95% success on contact-rich tasks.
-
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
MSDP pre-trains a transformer encoder with masked multisensory autoencoding, then uses an asymmetric actor-critic bridge (cross-attention for critic, pooling for actor) to accelerate and robustify contact-rich RL acro...
-
Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding
FuSe fine-tunes generalist robot policies on touch and audio data using language as a bridge, enabling multimodal prompts and compositional cross-modal tasks.
-
SonicBoom: Contact Localization Using Array of Microphones
A six-microphone array on a robot arm, combined with a learned audio and motion model, localizes contact points on the arm to within 0.4 to 2.2 cm, including on novel objects and human strikes.
-
GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion
A vision-led cross-attention fusion of camera and GelSight touch images improves imitation-learning success rates on three contact-rich manipulation tasks.
-
Modality Selection and Skill Segmentation via Cross-Modality Attention
Attention weights in a cross-modal transformer trained on furniture assembly cluster into distinct patterns for different manipulation primitives, suggesting the possibility of unsupervised skill segmentation, but the...
Discussion (0). Continue with ORCID to comment.