REVIEW 4 cited by
Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Although pre-training on a large amount of data is beneficial for robot learning, current paradigms only perform large-scale pretraining for visual representations, whereas representations for other modalities are trained from scratch. In contrast to the abundance of visual data, it is unclear what relevant internet-scale data may be used for pretraining other modalities such as tactile sensing. Such pretraining becomes increasingly crucial in the low-data regimes common in robotics applications. In this paper, we address this gap by using contact microphones as an alternative tactile sensor. Our key insight is that contact microphones capture inherently audio-based information, allowing us to leverage large-scale audio-visual pretraining to obtain representations that boost the performance of robotic manipulation. To the best of our knowledge, our method is the first approach leveraging large-scale multisensory pre-training for robotic manipulation. For supplementary information including videos of real robot experiments, please see https://sites.google.com/view/hearing-touch.
Forward citations
Cited by 4 Pith papers
-
S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information
S2A2 adds microphone-array spatial audio and spectrograms to imitation-learning policies, substantially improving success on manipulation tasks where vision alone cannot identify the target or destination.
-
Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion
LAG-Fusion composes asynchronous diffusion policies by rebasing delayed guidance into the current action frame and fusing with latency-aware weights, improving contact-rich manipulation performance.
-
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
MSDP pretrains a transformer encoder via masked multisensory reconstruction and feeds the embeddings into an asymmetric actor-critic RL setup, yielding faster learning and high real-robot success rates with only 6,000...
-
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
MSDP pre-trains a transformer encoder with masked multisensory autoencoding, then uses an asymmetric actor-critic bridge (cross-attention for critic, pooling for actor) to accelerate and robustify contact-rich RL acro...
Discussion (0). Sign in to comment.