REVIEW 28 cited by
Masked Visual Pre-training for Motor Control
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We first train the visual representations by masked modeling of natural images. We then freeze the visual encoder and train neural network controllers on top with reinforcement learning. We do not perform any task-specific fine-tuning of the encoder; the same visual representations are used for all motor control tasks. To the best of our knowledge, this is the first self-supervised model to exploit real-world images at scale for motor control. To accelerate progress in learning from pixels, we contribute a benchmark suite of hand-designed tasks varying in movements, scenes, and robots. Without relying on labels, state-estimation, or expert demonstrations, we consistently outperform supervised encoders by up to 80% absolute success rate, sometimes even matching the oracle state performance. We also find that in-the-wild images, e.g., from YouTube or Egocentric videos, lead to better visual representations for various manipulation tasks than ImageNet images.
Forward citations
Cited by 28 Pith papers
-
Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies
Regularizing flow-VLA action-velocity predictions across action-equivalent camera views improves held-out camera success on LIBERO-Plus and a real robot while keeping single-scene-RGB inference.
-
BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
Adding stage-wise temporal and spatial memory to a heatmap-prediction 3D VLA policy yields strong results on memory-dependent manipulation benchmarks while keeping data efficiency.
-
VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances
From 204K egocentric human videos, the authors automatically extract visual, grasp, and trajectory affordances and train one vision-language model, VLAff, that predicts all three for robot manipulation.
-
Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.
-
RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
Conditioning a pretrained ViT on per-pixel Plücker camera rays — via a gated-cross-attention class token and patch-level ray embeddings — makes imitation-learned manipulation policies substantially more robust to came...
-
Robot-DIFT: Correspondence-Sensitive Diffusion Features for Contact-Rich Robot Manipulation
Distilling Stable Diffusion's decoder features into a deterministic student backbone with a multi-scale fusion network improves contact-rich manipulation success in simulation and on a real robot.
-
Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning
MSDP pre-trains a transformer encoder with masked multisensory autoencoding, then uses an asymmetric actor-critic bridge (cross-attention for critic, pooling for actor) to accelerate and robustify contact-rich RL acro...
-
Demonstrating Multi-Suction Item Picking at Scale via Multi-Modal Learning of Pick Success
An in-domain multimodal masked-autoencoder model predicts multi-suction pick success with higher test AUC than an engineered-feature baseline on large real-world warehouse picking datasets.
-
UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
UAD distills affordance knowledge from vision-language models and DINOv2 features into a lightweight task-conditioned model that predicts pixel-level manipulation regions and improves few-shot imitation learning gener...
-
Efficient Robotic Policy Learning via Latent Space Backward Planning
Planning backward from a predicted final latent goal, instead of forward into the future, reduces error accumulation in long-horizon robot manipulation and outperforms prior planning methods on LIBERO-LONG.
-
RoboBERT: An End-to-end Multimodal Robotic Manipulation Model
A two-stage trained vision-language-action diffusion policy with carefully selected data augmentations reaches mean episode lengths of 4.52 (ABCD to D) and 3.79 (ABC to D) on CALVIN.
-
On Learning Informative Trajectory Embeddings for Imitation, Classification and Regression
A variational autoencoder over skill sequences produces label-free trajectory embeddings that separate and imitate policies of different ability levels in MuJoCo control tasks.
-
Learning from Massive Human Videos for Universal Humanoid Pose Control
Humanoid-X contributes 163,800 text-annotated motion clips retargeted from human videos into humanoid robot poses, and UH-1 is an autoregressive transformer that maps text instructions to humanoid actions.
-
Reinforcement Learning from Wild Animal Videos
A quadruped robot acquires walking, jumping, running-like, and standing skills using only the output of a video classifier trained on wild-animal videos as its reinforcement learning reward.
-
The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
In model-based reinforcement learning, frozen pre-trained visual representations do not improve sample efficiency or out-of-distribution generalization over representations learned from scratch.
-
Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation
Lift3D-VLA integrates 3D point cloud encoding and temporal action modeling into Vision-Language-Action models, achieving higher success rates on simulated and real-world robotic manipulation tasks.
-
The Latent Color Subspace: Emergent Order in High-Dimensional Chaos
FLUX.1’s VAE latent space contains an interpretable Hue–Saturation–Lightness structure that enables training-free color prediction and control via closed-form latent edits.
-
4D Visual Pre-training for Robot Learning
A next-frame point-cloud diffusion pre-training method (FVP) improves DP3 and RDT-1B manipulation success rates on the paper's own tasks.
-
Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning
A lightweight attention module that weights embeddings from multiple pre-trained models achieves comparable Atari RL performance to end-to-end training, with improved robustness to visual changes.
-
GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulation
GLOVER++ predicts where to interact with objects from language instructions after training on the new 500K-image HOVA-500K affordance dataset, and reports gains in simulated and real manipulation.
-
Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder
APE pretrains a ResNet18 encoder with adaptively selected augmentations and freezes its early layers during policy learning, improving sample efficiency of DreamerV3 and DrQ-v2 on several visual RL benchmarks.
-
An Empirical Study of Autoregressive Pre-training from Videos
Autoregressive next-token prediction on video and image tokens yields competitive visual representations across recognition, tracking, and robotics benchmarks, with scaling laws that are slower than those of language models.
-
Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos
A video labeling model trained with visual shift contrast, latent future reconstruction, and ground-truth action prediction enables sample-efficient policy cloning from action-free videos on Procgen.
-
Mask-based Predictive Representations for Reinforcement Learning
Mask-based predictive representations (MPR) as an auxiliary self-supervised task improve sample efficiency of vision-based RL over prior SOTA on continuous and discrete control benchmarks.
-
Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
A survey that taxonomizes robotic manipulation policies trained by imitation learning, traces their evolution, and compiles benchmark comparisons.
-
A Survey of State Representation Learning for Deep Reinforcement Learning
A six-class taxonomy of state representation learning methods for model-free online deep reinforcement learning, with selection guidelines, evaluation metrics, and future directions.
-
Efficient Sensorimotor Learning for Open-world Robot Manipulation
A PhD dissertation argues that object, spatial, and behavioral regularities, extracted with foundation models, enable data-efficient, generalizable robot manipulation, and presents seven systems and a benchmark built ...
-
Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer
Manipulation-pretrained CNN encoders score highest on an offline benchmark of 23 vision encoders judged by action-probing accuracy and sim-real embedding alignment.
Discussion (0). Continue with ORCID to comment.