REVIEW 9 cited by
Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation
read the original abstract
We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions collected with the Digit 360 sensor, Sparsh-X captures complementary touch signals at diverse temporal and spatial scales. By leveraging self-supervised learning, Sparsh-X fuses these modalities into a unified representation that captures physical properties useful for robot manipulation tasks. We study how to effectively integrate real-world touch representations for both imitation learning and tactile adaptation of sim-trained policies, showing that Sparsh-X boosts policy success rates by 63% over an end-to-end model using tactile images and improves robustness by 90% in recovering object states from touch. Finally, we benchmark Sparsh-X ability to make inferences about physical properties, such as object-action identification, material-quantity estimation, and force estimation. Sparsh-X improves accuracy in characterizing physical properties by 48% compared to end-to-end approaches, demonstrating the advantages of multisensory pretraining for capturing features essential for dexterous manipulation.
Forward citations
Cited by 9 Pith papers
-
Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks
Tactile Genesis provides a scalable multi-type tactile simulator and ablation results showing whole-hand coverage with per-taxel force/torque sensing outperforms fingertip-only or other modalities across three dextero...
-
Tactile Genesis: Exploring Tactile Sensors at Scale for Learning Dexterous Tasks
Whole-hand tactile coverage and per-taxel force/torque dominate sensor type and resolution for learning three dexterous tasks in a new high-throughput tactile simulator.
-
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Future tactile prediction applied to intermediate action-expert features, rather than visual-language or final-action features, improves contact-rich manipulation in SmolVLA and π0.
-
TactX: Learning Shared Tactile Representations Across Diverse Sensors
TactX learns a shared latent representation across three tactile sensor modalities via joint training on paired contacts, enabling zero-shot policy transfer and higher success on pick-and-place, insertion, wiping, and...
-
Heterogeneous Tactile Transformer
HTT learns shared representations across heterogeneous tactile sensors using a new paired dataset and pretraining objectives, enabling transfer to unseen sensors and tasks.
-
Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation
MiTaS fuses multi-resolution tactile data from GelSight and Evetac sensors with vision using modality-specific stems and transformer fusion to condition flow-matching policies, reporting 80% average success on five co...
-
TacO: Benchmarking Tactile Sensors for Object Manipulation
The paper provides a task-driven benchmark comparing visual, acoustic, magnetic, and resistive tactile sensors on three manipulation tasks and concludes that sensor utility depends on modality, material friction, and ...
-
Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation
MPAIL2 demonstrates real-world manipulation learning from observation alone, without rewards or action labels, plus positive online transfer.
-
DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
A decoupled multimodal diffusion transformer with a LoRA tactile adapter improves real-world bimanual manipulation success by 21 percentage points over a diffusion-policy baseline; a new 50-hour tactile bimanual datas...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.