Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Masked multisensory pretraining yields a frozen representation that, bridged asymmetrically, accelerates contact-rich robot reinforcement learning and stays reliable under sensor noise and dynamics changes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:35 UTC pith:A3HJGGMV

load-bearing objection Useful empirical recipe for multisensory pretraining in contact-rich RL, but the paper overclaims robustness on the basis of its own failure cases. the 4 major comments →

arxiv 2511.14427 v4 pith:A3HJGGMV submitted 2025-11-18 cs.RO cs.LG

Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning

Rickmer Krohn , Vignesh Prasad , Gabriele Tiboni , Georgia Chalvatzaki This is my paper
classification cs.RO cs.LG
keywords multisensory pretrainingmasked autoencodingcontact-rich manipulationreinforcement learningsensor fusionforce-torque sensingreal robot learninglatent bridging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MSDP is a self-supervised pretraining framework in which a transformer encoder is trained to reconstruct multisensory observations—vision, force-torque, and proprioception—from a random subset of sensor embeddings. The paper argues that this masked-autoencoding objective yields a fused, informative representation that is more robust to sensor noise and changes in object dynamics than concatenation, product-of-experts, or transformer-bottleneck alternatives. For downstream reinforcement learning, the encoder is frozen and bridged asymmetrically: a cross-attention layer supplies the critic with task-specific features, while mean pooling supplies the actor with a stable input. In real-world peg insertion and cube pushing, MSDP reaches high success rates with about 6,000 online interactions and no sim-to-real transfer; force-torque readings account for a 14% improvement in success.

Core claim

MSDP's central claim is that a single frozen multisensory encoder—trained offline on random interaction data by reconstructing the current or next observation from a masked subset of sensor tokens—provides a representation that makes downstream actor-critic RL sample-efficient and robust to perturbations. The key design insight is the asymmetric latent bridge: a learnable cross-attention query gives the critic dynamic, task-specific features, while parameter-free mean pooling gives the actor a stable representation. With this setup, MSDP outperforms concatenation and product-of-experts baselines across four simulated contact-rich tasks and enables direct real-world training without sim-to-re

What carries the argument

The central object is the MSDP transformer encoder trained by multisensory masked autoencoding. A CNN stem embeds visual patches; linear projections embed force-torque and proprioception; learnable positional/modality encodings are added. During pretraining, a random subset of sensor tokens is masked and the encoder must reconstruct the current observation (MSDP-R) or the action-conditioned next observation (MSDP-P) from the remaining tokens. This forces cross-modal prediction and fuses the modalities. For downstream RL, the frozen embeddings are bridged asymmetrically: a single cross-attention layer with a learnable query provides the critic with dynamic task-specific features, while mean p

Load-bearing premise

The method rests on the assumption that a single pre-learned sensor encoding stays useful for the final task across different visual complexity and dynamics; the paper's own real-world cube-pushing result, where the next-observation pretraining version fails, shows this assumption is not always safe.

What would settle it

A decisive test is to run the real-world cube-pushing task with the next-observation pretraining variant (which failed in the paper) but add a short observation history, e.g., the last two frames; if first success still does not appear within about 6,000 online interactions while the reconstruction variant succeeds, the claim that the pre-trained representation alone transfers to vision-demanding tasks is falsified. Alternatively, compare the method against a simple concatenation baseline on a new contact-rich task with the same data budget; a tie would refute the claimed acceleration.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Real-robot contact-rich policies (peg insertion, cube pushing) can be trained directly on multisensory observations in under an hour, with no sim-to-real transfer, using about 6,000 online interactions.
  • The frozen representation is robust to sensor noise and to changes in object mass, friction, and center of mass, where concatenation and product-of-experts baselines degrade.
  • Force-torque feedback is not a marginal extra: removing it lowers real-world success by 14% and leads to high-force, unsafe exploratory contacts.
  • Pretraining with additional sensors enriches the visual representation so that a vision-only policy can later benefit from force and proprioception information encoded implicitly in the vision tokens.
  • The 'latent bridging' design (asymmetric cross-attention for the critic, pooling for the actor) is a distinct and important choice; simpler bridges like CLS or pure pooling are markedly worse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to use the same frozen encoder with different sensor subsets at deployment time, since the architecture supports variable-length inputs and masking; the paper does not run this experiment, but it follows directly from the design.
  • The failure of the prediction objective on real-world Push Cube suggests that forward-dynamics reconstruction may be too hard for vision-dominated tasks without observation history; adding a short history or a latent target could be a straightforward fix.
  • If the 30,000-sample offline pretraining works this well, scaling masked multisensory autoencoding to larger, multi-task offline datasets could produce a reusable manipulation representation, a direction the authors mention as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MultiSensory Dynamic Pretraining (MSDP), a self-supervised framework for contact-rich robot RL. MSDP pretrains a transformer-based multisensory encoder with masked autoencoding, reconstructing either the current observation (MSDP-R) or the next observation conditioned on action (MSDP-P). The encoder is frozen during downstream RL, and the paper introduces an asymmetric latent-bridging design: a cross-attention layer for the critic and a mean-pooled representation for the actor. The method is evaluated in four simulated panda-gym/PyBullet tasks and two real-world Franka tasks, with comparisons to Concat, PoE, and VTT baselines. The claims include accelerated RL, robustness to sensor noise and dynamics changes, and high real-world success rates with 6,000 online interactions.

Significance. If the claims hold, MSDP would be a useful contribution to multisensory representation learning for RL: it combines masked autoencoding with a task-oriented latent-bridging strategy and demonstrates real-world training without sim-to-real transfer. The paper's strengths are the 12-run confidence intervals in simulation, several ablations (sensor sets, latent bridging, objective choice), and real-robot validation. However, the central claim that a single frozen representation transfers robustly across tasks is contradicted by the paper's own results: MSDP-P fails on real-world Push Cube while MSDP-R succeeds, and the Close Drawer Gently simulation task is described as inconclusive. These issues prevent the broad claims from being accepted as stated, though the underlying approach appears promising.

major comments (4)
  1. [Sec. V-D / Fig. 10b] The central claim that MSDP provides a single robust representation is contradicted by the paper's own results: on real-world Push Cube, MSDP-P (action-conditioned next-observation prediction) fails to extract a usable representation while MSDP-R succeeds. The choice between the two objectives is not principled; on the same real task it flips success/failure. Since the abstract and contributions attribute superiority to 'MSDP' without qualification, the reader cannot tell which objective to use a priori. Either scope the claim to the successful variant (and analyze when each objective transfers), or provide a principled criterion for selecting P vs R. The attribution to 'lack of observation histories' in Sec. V-D also reveals a limitation that should be stated as such.
  2. [Sec. V-C / Fig. 4] The text states Close Drawer Gently 'offers no clear results, mainly due to simulation bottlenecks,' but the Figure 4 caption asserts 'Our method significantly accelerates RL training and achieves the highest final success rate across all tasks.' These statements are incompatible. If the task is inconclusive, it cannot support the 'across all tasks' claim; if it does, the caption should be backed by reported quantitative results and error bars. Please resolve this contradiction and avoid over-generalizing from inconclusive data.
  3. [Sec. V-D / Fig. 10b] In real-world Push Cube, only MSDP-R and VTT attain high final success; MSDP-P fails. The text reports that 'MSDP-R and VTT are able to obtain a high final success rate' but does not report the quantitative difference or confidence intervals. This leaves open the possibility that MSDP-R does not outperform the VTT baseline on this task, which further weakens the general 'outperforming baselines' claim. Please provide final success rates (mean and CI) for all real-world methods and discuss cases where MSDP does not beat the baseline.
  4. [Sec. V / Abstract] The claim of 'robust performance under diverse perturbations, including sensor noise, and changes in object dynamics' is not supported by any systematic perturbation study. The simulation environments randomize noise/dynamics during training, but there is no comparison of MSDP against baselines under test-time perturbations, nor an ablation measuring sensitivity to noise levels or dynamics changes. The sensor ablations (Fig. 5, 6) remove sensors but do not evaluate robustness to perturbations. Either add such experiments or soften the claim to 'performs well in environments with sensor noise and varying dynamics.'
minor comments (6)
  1. [Sec. IV] Implementation details are insufficient for reproducibility: architecture dimensions, number of transformer layers, embedding sizes, masking ratio, learning rates, and optimization hyperparameters are not reported. Please provide these or a supplementary appendix.
  2. [Sec. IV-C] There are several typos: 'assymmetric' should be 'asymmetric,' 'similiar' should be 'similar.' The author affiliation also contains 'Univsersity.'
  3. [Sec. V-D] Real-world results are based on only 5 runs (Peg Insertion) and 3 runs (Push Cube), with no confidence intervals or statistical tests. This should be acknowledged and, if possible, the number of runs increased or at least the variance reported.
  4. [Sec. V-D] The claim that FT-sensor usage improves performance by 14% needs a precise definition: which curves are compared, at what training step, and with what variance? As written, it is not clear how the 14% is computed, and the same percentage for both tasks seems coincidental.
  5. [Sec. V-D / Fig. 10] The statement 'first successful episodes after only 2,000 / 1,000 online interactions' should specify which task and which MSDP variant (P or R) each number refers to, and whether the other variant also achieves success at that point.
  6. [Sec. IV-C] The claim that the method 'only introduces a few learnable parameters' is vague; please quantify the additional parameters added by the cross-attention layer relative to the baselines.

Circularity Check

0 steps flagged

No significant circularity; MSDP's pretrained representations are evaluated by downstream RL success, which is not used in pretraining.

full rationale

The paper's derivation chain is self-contained and empirically falsifiable: MSDP first trains a transformer encoder offline by masked autoencoding (reconstructing current or next multisensory observations from a subset of sensor embeddings), then freezes that encoder and trains an actor-critic policy on the frozen embeddings with SAC/RLPD. The downstream RL success rates reported in Figures 4, 5, 6, 7, 8, 10a, and 10b are not used to fit or select the pretraining weights in a way that would make the prediction equal to the input. Neither the abstract's claim of accelerated learning nor the 14% FT-sensor improvement reduces by construction to a fitted parameter; MSDP is compared against no-FT ablations and against Concat, PoE, and VTT baselines that are pretrained with the same observation-reconstruction objective and comparable decoders. The latent-bridging cross-attention for the critic is learned during RL from reward/Bellman error, not from the pretraining target, so 'task-specific feature extraction' is not an input renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The paper's own admissions that MSDP-P fails on real-world Push Cube (Sec. V-D) and that Close Drawer Gently yields 'no clear results, mainly due to simulation bottlenecks' (Sec. V-C) weaken the breadth of the robustness/transfer claim, but that is an overclaim or correctness risk, not circular reasoning: the failed tasks show the framework is not trivially guaranteed to succeed. Accordingly, no circular step can be quoted or exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central claim rests on three layers of empirical assumptions: (i) masked-autoencoding pretraining produces a transferable multisensory representation; (ii) the asymmetric cross-attention/pooling bridge is a better compact mapping than CLS, pooling, or concatenation; (iii) the chosen RL algorithms (SAC, RLPD) and budgets are sufficient. No exact masking ratio is reported, and per-task hyperparameters (pretraining steps, update ratios) are tuned, so the method as presented still contains several free parameters beyond the core pretraining objective.

free parameters (4)
  • Pretraining observation-reconstruction target = next-obs (MSDP-P) vs current-obs (MSDP-R) chosen per task
    Both objectives are evaluated, and the choice of which one to use is empirical and task-dependent (e.g., MSDP-P works in simulation and real peg insertion, but fails on real-world push; MSDP-R works there). This is a hand-chosen design decision, not derived.
  • Pretraining update steps / number of demos+play samples = 6,000 steps for peg insertion; 10,000 steps for push cube; ~3,000 total offline samples
    The paper says they use 6k pretraining steps for Peg Insertion and 10k for Push Cube 'due to higher vision variance'; these are task-specific tuning choices made to obtain good results, hence free parameters.
  • Critic/actor update ratios in RLPD = 2 critic+1 actor updates per online sample (peg); 4 critic+1 actor (push)
    The per-task update ratios are tuned for each real-world task; the paper states these explicitly, so they are honest but still effective free parameters of the successful configuration.
  • Masking scheme/ratio = not specified numerically
    The paper says 'randomly mask out a subset of sensor tokens' and 'Vision as a global sensing modality never gets fully masked out', but no exact masking ratio/strategy is given; this is an implicit, unreported free parameter that typical MAE implementations need.
axioms (3)
  • domain assumption The pretrained multisensory encoder, trained by masked autoencoding reconstruction, yields a representation whose features remain informative for the downstream RL objective after freezing.
    This is the load-bearing premise of the entire framework. It is empirically validated in some tasks and contradicted in others (MSDP-P on real-world Push Cube), so it is not a mathematical guarantee.
  • domain assumption The simulator (PyBullet/panda-gym) models contact-rich dynamics well enough that multisensory representations learned in simulation transfer to the real world without sim-to-real adjustment.
    Real-world experiments use only real-world data (20 demos + play data), so sim-to-real transfer is not actually claimed for the final policy; however, the simulation evaluation is presented as evidence of the method's generality, and the real-world tasks use the same sensor types. This is a standard but unproven assumption about environment fidelity.
  • domain assumption SAC/RLPD off-policy RL with frozen encoder converges to a high-success policy in the given step budgets (500k sim steps, 6k real steps).
    This is an empirical assumption from prior RL literature, invoked in Section IV-C and the experiments; the paper does not prove convergence and indeed relies on RLPD's offline-data efficiency.
invented entities (2)
  • MSDP encoder (multisensory transformer with learnable positional/modality embeddings) independent evidence
    purpose: Fuses all sensor embeddings into a latent representation used for pretraining and downstream RL.
    It is a new model component, but it does not postulate new physics or unobserved quantities; its behavior is testable through the RL success-rate curves, which are independent handles.
  • Asymmetric latent bridging (cross-attention critic query + pooled actor representation) independent evidence
    purpose: Maps frozen embeddings to compact actor/critic inputs; claimed to improve task-specific feature extraction and training stability.
    The 'learnable query' is an architectural entity, not an invented physical quantity; its usefulness is validated by ablation (Figure 7), so it has a falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 11904 in / 9095 out tokens · 75101 ms · 2026-08-03T21:35:02.967099+00:00 · methodology

0 comments
read the original abstract

Effective contact-rich manipulation requires robots to synergistically leverage vision, force, and proprioception. However, Reinforcement Learning agents struggle to learn in such multisensory settings, especially amidst sensory noise and dynamic changes. We propose MultiSensory Dynamic Pretraining (MSDP), a novel framework for learning expressive multisensory representations tailored for task-oriented policy learning. MSDP is based on masked autoencoding and trains a transformer-based encoder by reconstructing multisensory observations from only a subset of sensor embeddings, leading to cross-modal prediction and sensor fusion. For downstream policy learning, we introduce a novel asymmetric architecture, where a cross-attention mechanism allows the critic to extract dynamic, task-specific features from the frozen embeddings, while the actor receives a stable pooled representation to guide its actions. Our method demonstrates accelerated learning and robust performance under diverse perturbations, including sensor noise, and changes in object dynamics. Evaluations in multiple challenging, contact-rich robot manipulation tasks in simulation and the real world showcase the effectiveness of MSDP. Our approach exhibits strong robustness to perturbations and achieves high success rates on the real robot with as few as 6,000 online interactions, offering a simple yet powerful solution for complex multisensory robotic control. Website: https://msdp-pearl.github.io/

Figures

Figures reproduced from arXiv: 2511.14427 by Gabriele Tiboni, Georgia Chalvatzaki, Rickmer Krohn, Vignesh Prasad.

Figure 1
Figure 1. Figure 1: Multisensory Dynamic Pretraining fuses multiple sensors, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The MSDP framework with MSDP-Encoder (left), Pretraining (top right) and downstream RL (bottom right): The current multisensory [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Multisensory contact-rich robot environments [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison between MSDP-P and MSDP-R to the baselines in Peg Insertion, Push Cube, Close Drawer Gently and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Peg Insertion Sensor Ablation: Proprioception is crucial to [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Pretraining the representation for Peg Insertion with multiple [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Critic’s cross-attention maps in the Push Cube task. The [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Real world setup and experimental results. Our MSDP framework enables training RL policies directly in the real world, with first [PITH_FULL_IMAGE:figures/full_fig_p007_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots

    cs.RO 2026-07 conditional novelty 6.0

    A self-supervised multimodal encoder trained with vision, proprioception, and force yields a vision-only latent that recovers end-effector state and force above vision baselines on RH20T, with modest absolute force accuracy.

Reference graph

Works this paper leans on

55 extracted references · 26 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Playing Atari with Deep Reinforcement Learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier- stra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” Dec. 2013, arXiv:1312.5602 [cs]

    Pith/arXiv arXiv 2013

  2. [2]

    Robot Parkour Learning,

    Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao, “Robot Parkour Learning,” Sep. 2023, arXiv:2309.05665 [cs]

    Pith/arXiv arXiv 2023

  3. [3]

    Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control,

    F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke, “Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control,” Nov. 2015, arXiv:1511.03791 [cs]

    Pith/arXiv arXiv 2015

  4. [4]

    Making Sense of Vision and Touch: Self- Supervised Learning of Multimodal Representations for Contact-Rich Tasks,

    M. A. Lee, Y . Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making Sense of Vision and Touch: Self- Supervised Learning of Multimodal Representations for Contact-Rich Tasks,”CoRR, vol. abs/1810.10191, 2018, _eprint: 1810.10191

    Pith/arXiv arXiv 2018

  5. [5]

    The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning,

    C. Sferrazza, Y . Seo, H. Liu, Y . Lee, and P. Abbeel, “The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning,” Nov. 2023, arXiv:2311.00924 [cs]

    Pith/arXiv arXiv 2023

  6. [6]

    Sensor fusion for compliant robot motion control,

    J. G. García, A. Robertsson, J. G. Ortega, and R. Johansson, “Sensor fusion for compliant robot motion control,”IEEE Transactions on Robotics, vol. 24, no. 2, pp. 430–441, 2008

    2008

  7. [7]

    Dexterous robotic manipulation of deformable objects with multi-sensory feedback-a review,

    F. F. Khalil and P. Payeur, “Dexterous robotic manipulation of deformable objects with multi-sensory feedback-a review,”Robot Manipulators Trends and Development, no. March 2010, 2010

    2010

  8. [8]

    Development of sensory-motor fusion-based manipulation and grasping control for a robotic hand-eye system,

    Y . Hu, Z. Li, G. Li, P. Yuan, C. Yang, and R. Song, “Development of sensory-motor fusion-based manipulation and grasping control for a robotic hand-eye system,”IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 47, no. 7, pp. 1169–1180, 2016

    2016

  9. [9]

    A review on sensory perception for dexterous robotic manipulation,

    Z. Xia, Z. Deng, B. Fang, Y . Yang, and F. Sun, “A review on sensory perception for dexterous robotic manipulation,”International Journal of Advanced Robotic Systems, vol. 19, no. 2, p. 17298806221095974, 2022

    2022

  10. [10]

    A review on challenges of autonomous mobile robot and sensor fusion methods,

    M. B. Alatise and G. P. Hancke, “A review on challenges of autonomous mobile robot and sensor fusion methods,”IEEE Access, vol. 8, pp. 39 830–39 846, 2020

    2020

  11. [11]

    See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation,

    H. Li, Y . Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu, “See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation,” Dec. 2022, arXiv:2212.03858 [cs]

    Pith/arXiv arXiv 2022

  12. [12]

    Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations,

    Y . Hao, R. Wang, Z. Cao, Z. Wang, Y . Cui, and D. Sadigh, “Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations,” in2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct. 2023, pp. 1–7, iSSN: 2153-0866

    2023

  13. [13]

    MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models,

    S. Saxena, M. Sharma, and O. Kroemer, “MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models,” Jan. 2024, arXiv:2401.14502 [cs]

    Pith/arXiv arXiv 2024

  14. [14]

    Data quality in imitation learning,

    S. Belkhale, Y . Cui, and D. Sadigh, “Data quality in imitation learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  15. [15]

    Visuo-Tactile Transformers for Manipulation,

    Y . Chen, A. Sipos, M. Van der Merwe, and N. Fazeli, “Visuo-Tactile Transformers for Manipulation,” Sep. 2022, arXiv:2210.00121 [cs]

    Pith/arXiv arXiv 2022

  16. [16]

    Q. Liu, Z. Sun, Y . Cui, L. Gaofeng, Q. Ye, and J. Chen,Masked Visual-Tactile Pre-training for Robot Manipulation, Feb. 2024

    2024

  17. [17]

    MultiMAE: Multi-modal Multi-task Masked Autoencoders,

    R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, “MultiMAE: Multi-modal Multi-task Masked Autoencoders,” Apr. 2022

    2022

  18. [18]

    Multimodal Masked Autoencoders Learn Transferable Representations,

    X. Geng, H. Liu, L. Lee, D. Schuurmans, S. Levine, and P. Abbeel, “Multimodal Masked Autoencoders Learn Transferable Representations,” May 2022

    2022

  19. [19]

    Simple Masked Training Strategies Yield Control Policies That Are Robust to Sensor Failure,

    S. Skand, B. Pandit, C. Kim, L. Fuxin, and S. Lee, “Simple Masked Training Strategies Yield Control Policies That Are Robust to Sensor Failure,” Sep. 2024

    2024

  20. [20]

    Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation,

    G.-H. Liu, A. Siravuru, S. Prabhakar, M. Veloso, and G. Kantor, “Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation,” 2017

    2017

  21. [21]

    Real-World Robot Learning with Masked Visual Pre-training,

    I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell, “Real-World Robot Learning with Masked Visual Pre-training,” Oct. 2022, arXiv:2210.03109 [cs]

    Pith/arXiv arXiv 2022

  22. [22]

    Historical perspective and state of the art in robot force control,

    D. E. Whitney, “Historical perspective and state of the art in robot force control,”The International Journal of Robotics Research, vol. 6, no. 1, pp. 3–14, 1987

    1987

  23. [23]

    Quasi-static assembly of compliantly supported rigid parts,

    D. E. Whitneyet al., “Quasi-static assembly of compliantly supported rigid parts,”Journal of Dynamic Systems, Measurement, and Control, vol. 104, no. 1, pp. 65–77, 1982

    1982

  24. [24]

    Learning the peg-into-hole assembly operation with a connectionist reinforcement technique,

    M. Nuttin and H. Van Brussel, “Learning the peg-into-hole assembly operation with a connectionist reinforcement technique,”Computers in Industry, vol. 33, no. 1, pp. 101–109, 1997

    1997

  25. [25]

    End-to-end training of deep visuomotor policies,

    S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,”Journal of Machine Learning Research, vol. 17, no. 39, pp. 1–40, 2016

    2016

  26. [26]

    A review on reinforce- ment learning for contact-rich robotic manipulation tasks,

    Í. Elguea-Aguinaco, A. Serrano-Muñoz, D. Chrysostomou, I. Inziarte- Hidalgo, S. Bøgh, and N. Arana-Arexolaleiba, “A review on reinforce- ment learning for contact-rich robotic manipulation tasks,”Robotics and Computer-Integrated Manufacturing, vol. 81, p. 102517, 2023

    2023

  27. [27]

    A survey of robot manipulation in contact,

    M. Suomalainen, Y . Karayiannidis, and V . Kyrki, “A survey of robot manipulation in contact,”Robotics and Autonomous Systems, vol. 156, p. 104224, 2022

    2022

  28. [28]

    Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,

    R. Liu, F. Nageotte, P. Zanne, M. de Mathelin, and B. Dresp-Langley, “Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review,”Robotics, vol. 10, no. 1, p. 22, 2021

    2021

  29. [29]

    An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale,”CoRR, vol. abs/2010.11929, 2020, _eprint: 2010.11929

    Pith/arXiv arXiv 2010

  30. [30]

    Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model,

    A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine, “Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model,” Oct. 2020, arXiv:1907.00953 [cs]

    Pith/arXiv arXiv 2020

  31. [31]

    Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation,

    J. Mejia, V . Dean, T. Hellebrekers, and A. Gupta, “Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation,” May 2024, arXiv:2405.08576 [cs]

    Pith/arXiv arXiv 2024

  32. [32]

    Audio-Visual In- stance Discrimination with Cross-Modal Agreement,

    P. Morgado, N. Vasconcelos, and I. Misra, “Audio-Visual In- stance Discrimination with Cross-Modal Agreement,” Mar. 2021, arXiv:2004.12943 [cs]

    Pith/arXiv arXiv 2021

  33. [33]

    Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,

    M. A. Lee, Y . Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,”IEEE Transactions on Robotics, vol. 36, no. 3, pp. 582–596, 2020

    2020

  34. [34]

    See, hear, and feel: Smart sensory fusion for robotic manipulation,

    H. Li, Y . Zhang, J. Zhu, S. Wang, M. A. Lee, H. Xu, E. Adelson, L. Fei-Fei, R. Gao, and J. Wu, “See, hear, and feel: Smart sensory fusion for robotic manipulation,” inConference on Robot Learning. PMLR, 2023, pp. 1368–1378

    2023

  35. [35]

    Reinforcement learning strategy based on multimodal representations for high-precision assembly tasks,

    A. Li, R. Liu, X. Yang, and Y . Lou, “Reinforcement learning strategy based on multimodal representations for high-precision assembly tasks,” inIntelligent Robotics and Applications: 14th International Conference, ICIRA 2021, Yantai, China, October 22–25, 2021, Proceedings, Part I

    2021

  36. [36]

    Springer, 2021, pp. 56–66

    2021

  37. [37]

    Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play,

    I. Guzey, B. Evans, S. Chintala, and L. Pinto, “Dexterity from touch: Self-supervised pre-training of tactile representations with robotic play,” arXiv preprint arXiv:2303.12076, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer,

    Y . Han, K. Yu, R. Batra, N. Boyd, C. Mehta, T. Zhao, Y . She, S. Hutchinson, and Y . Zhao, “Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer,” IEEE/ASME Transactions on Mechatronics, 2024

    2024

  39. [39]

    Play to the score: Stage-guided dynamic multi-sensory fusion for robotic manipulation,

    R. Feng, D. Hu, W. Ma, and X. Li, “Play to the score: Stage-guided dynamic multi-sensory fusion for robotic manipulation,” in8th Annual Conference on Robot Learning, 2024

    2024

  40. [40]

    Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre- Training,

    V . Dave, F. Lygerakis, and E. Rueckert, “Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre- Training,” Jan. 2024, arXiv:2401.12024 [cs]

    Pith/arXiv arXiv 2024

  41. [41]

    M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representa- tion Learning for Robotic Manipulation,

    F. Lygerakis, V . Dave, and E. Rueckert, “M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representa- tion Learning for Robotic Manipulation,” Jun. 2024, arXiv:2401.17032 [cs]

    Pith/arXiv arXiv 2024

  42. [42]

    Partially Observable Markov Decision Processes (POMDPs) and Robotics,

    H. Kurniawati, “Partially Observable Markov Decision Processes (POMDPs) and Robotics,” Jul. 2021, arXiv:2107.07599 [cs]

    Pith/arXiv arXiv 2021

  43. [43]

    Soft Actor- Critic Algorithms and Applications,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft Actor- Critic Algorithms and Applications,” 2019, _eprint: 1812.05905

    Pith/arXiv arXiv 2019

  44. [44]

    Efficient Online Rein- forcement Learning with Offline Data,

    P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient Online Rein- forcement Learning with Offline Data,” May 2023, arXiv:2302.02948 [cs]

    Pith/arXiv arXiv 2023

  45. [45]

    Masked Au- toencoders Are Scalable Vision Learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick, “Masked Au- toencoders Are Scalable Vision Learners,” Dec. 2021, arXiv:2111.06377 [cs]

    Pith/arXiv arXiv 2021

  46. [46]

    Context Autoencoder for Self-Supervised Representation Learning,

    X. Chen, M. Ding, X. Wang, Y . Xin, S. Mo, Y . Wang, S. Han, P. Luo, G. Zeng, and J. Wang, “Context Autoencoder for Self-Supervised Representation Learning,” Aug. 2023, arXiv:2202.03026 [cs]

    Pith/arXiv arXiv 2023

  47. [47]

    Masked World Models for Visual Control,

    Y . Seo, D. Hafner, H. Liu, F. Liu, S. James, K. Lee, and P. Abbeel, “Masked World Models for Visual Control,” May 2023, arXiv:2206.14244 [cs]

    Pith/arXiv arXiv 2023

  48. [48]

    Early Convolutions Help Transformers See Better,

    T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Gir- shick, “Early Convolutions Help Transformers See Better,” Oct. 2021, arXiv:2106.14881 [cs]

    Pith/arXiv arXiv 2021

  49. [49]

    Attention is All you Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017

    2017

  50. [50]

    Masked Visual Pre-training for Motor Control,

    T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, “Masked Visual Pre-training for Motor Control,” Mar. 2022, arXiv:2203.06173

    Pith/arXiv arXiv 2022

  51. [51]

    Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

    A. Majumdar, K. Yadav, S. Arnaud, Y . J. Ma, C. Chen, S. Silwal, A. Jain, V .-P. Berges, P. Abbeel, J. Malik, D. Batra, Y . Lin, O. Maksymets, A. Rajeswaran, and F. Meier, “Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?” Feb. 2024, arXiv:2303.18240 [cs]

    Pith/arXiv arXiv 2024

  52. [52]

    Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning,

    S. Garcin, T. McInroe, P. S. Castro, P. Panangaden, C. G. Lucas, D. Abel, and S. V . Albrecht, “Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning,” Mar. 2025, arXiv:2503.06343 [cs]

    Pith/arXiv arXiv 2025

  53. [53]

    panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,

    Q. Gallouédec, N. Cazin, E. Dellandréa, and L. Chen, “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,” 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021

    2021

  54. [54]

    PyBullet, a Python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “PyBullet, a Python module for physics simulation for games, robotics and machine learning,” 2016

    2016

  55. [55]

    Serl: A software suite for sample-efficient robotic reinforcement learning,

    J. Luo, Z. Hu, C. Xu, Y . L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine, “Serl: A software suite for sample-efficient robotic reinforcement learning,” 2024

    2024