Pith. sign in

REVIEW 3 major objections 5 minor 83 references

The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On three visual control benchmarks, model-based RL agents with pre-trained visual representations are no more sample-efficient and no better at out-of-distribution generalization than agents that learn their encoder from scratch.

desk verdict A substantial first MBRL-specific PVR benchmark whose headline negative result is plausible but partly confounded by the DreamerV3 decoder-target asymmetry. read the letter →

arxiv 2411.10175 v2 pith:VKVMD6EW submitted 2024-11-15 cs.LG cs.AIcs.CV

Moritz Schneider , Robert Krug , Narunas Vaskevicius , Luigi Palmieri , Joschka Boedecker This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords model-basedreinforcementlearningpre-trainedvisualrepresentationssampleefficiencyout-of-distributiongeneralizationworldmodelsDreamerV3TD-MPC2controlbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether frozen pre-trained visual representations (PVRs) — vision models trained on large image and video datasets, such as CLIP, DINOv2, R3M, and VC-1 — give model-based reinforcement learning agents an edge in sample efficiency and out-of-distribution (OOD) generalization. The question matters because PVRs are widely assumed to pass their robustness on to downstream control, and model-based RL is a leading route to data-efficient agents. The authors benchmark thirteen PVR variants inside DreamerV3 and TD-MPC2 on ten tasks from three domains and find that representations learned from scratch are in most cases equally or more data-efficient, and that the from-scratch baseline outperforms every PVR in OOD evaluation. Even autoencoders pre-trained on the downstream task's own data fail to beat the scratch baseline. The paper attributes the gap to world-model quality: PVR-based agents learn comparable dynamics predictions but worse reward predictions, and their latent spaces entangle low- and high-reward states; among PVR properties, data diversity and a ViT architecture are the strongest contributors to OOD performance.

What carries the argument

The argument runs through a specific integration scheme: the frozen PVR acts as a fixed feature extractor whose output $x_t$ replaces the raw image as the input to the MBRL agent, with only a single linear layer trained on top to map $x_t$ into the latent state (for DreamerV3, the linear map feeds the discrete-latent encoder; for TD-MPC2, a stack of three embeddings is fed to its encoder). The paper then dissects results with three instruments: normalized ID/OOD return curves, a categorization of PVRs by training properties (ViT vs. non-ViT, diverse data, sequential data, language conditioning), and world-model diagnostics on a pendulum task measuring accumulated dynamics prediction error, accumulated reward error, and UMAP projections of the latent state colored by reward. Notably, in DreamerV3 the decoder reconstructs the frozen embedding $x_t$, not the original image observation.

What would settle it

Train a from-scratch DreamerV3 agent whose decoder reconstructs a low-dimensional embedding of the same size as a PVR output, rather than the raw $64\times64$ image; if performance drops to the PVR level, the reconstruction target is the cause. The reverse test, letting a PVR-based agent reconstruct the raw image through the frozen encoder and showing it beats from scratch, would overturn the paper's conclusion.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that for model-based RL the transfer benefits of pre-trained visual representations do not materialize. Across DreamerV3 and TD-MPC2 on DMC, ManiSkill2, and Miniworld, a representation learned from scratch is at least as data-efficient as any of the thirteen PVR variants and often better, and it also wins in held-out out-of-distribution settings. The paper attributes this to the objective mismatch of MBRL and to the information bottleneck of PVRs: frozen features trained without a reward signal do not preserve the reward information the world model needs, so PVR-based agents struggle to predict rewards accurately (reward prediction error correlates negatively with task performance, $r = -0.66$), while dynamics prediction quality is comparable across representations. The authors conclude that the common assumption that PVRs improve model-based agents — carried over from model-free RL and imitation learning — does not hold for current PVRs, and that data diversity and network architecture, not language conditioning or sequential pre-training data, are what drive OOD generalization.

Load-bearing premise

The load-bearing assumption is that decoding the frozen pre-trained features instead of the original image is an equivalent way to train the agent's internal predictor of how the environment evolves; if the reconstruction target changes what that predictor learns, the from-scratch advantage could come from the interface rather than from the representations.

Editorial extensions

If this is right

  • Model-based RL practitioners gain nothing, in these settings, from swapping in a frozen CLIP-, DINOv2-, or VC-1-class encoder instead of training a CNN from scratch; data efficiency is equal or worse.
  • OOD generalization of the agent is not inherited from the PVR: held-out visual shifts are handled better by the scratch encoder in the benchmark.
  • Because reward prediction error, not dynamics prediction error, tracks task performance, improving a PVR-based world model should focus on reward-relevant information, not on more accurate state prediction.
  • Data diversity and ViT backbone are the PVR properties that matter most for OOD performance; language conditioning and sequential pre-training data play minor roles.
  • Pre-training an encoder on the downstream task's own data distribution does not help either, so the failure is not simply domain mismatch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison is asymmetric: DreamerV3 with a PVR decodes the frozen embedding, while the from-scratch baseline reconstructs the raw $64\times64$ or $128\times128$ image; if the reconstruction target changes what the world model learns, part of the scratch advantage could be an artifact of this integration rather than of PVRs. This confound is stated in the appendix but never tested.
  • A direct test would be to let a from-scratch encoder also decode a low-dimensional embedding instead of pixels, or to train a PVR-based agent with a decoder that reconstructs pixels through the frozen features; either design would isolate the representation from the interface.
  • The reward-bottleneck result suggests a concrete recipe for future PVRs aimed at MBRL: pre-training objectives should include reward or value information, such as value-implicit or goal-conditioned losses, rather than only reconstruction or contrastive image-text alignment.
  • Because sequential training data helped in manipulation and navigation domains but not in DMC, the usefulness of video pre-training may depend on the structure of the downstream task; testing more control families could reveal when temporal priors matter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper benchmarks a set of pre-trained visual representations (PVRs), including CLIP, DINOv2, OpenCLIP, R3M, VIP, VC-1, Taskonomy variants, and an in-distribution autoencoder, in two model-based RL algorithms (DreamerV3 and TD-MPC2) across 10 tasks in DMC, ManiSkill2, and Miniworld. It measures in-distribution sample efficiency and out-of-distribution generalization under visual randomization shifts, and analyzes world-model quality in terms of accumulated dynamics and reward prediction errors. The headline finding is that frozen PVRs with a linear encoder do not improve sample efficiency or OOD generalization over a from-scratch CNN encoder, and that the from-scratch baseline is often better; the paper further attributes OOD success to data diversity and ViT architecture. The evaluation uses 4-6 seeds per condition and 200 evaluation rollouts per seed (1200 per representation-environment), with standard errors shown.

Significance. If the headline result holds, the paper provides a timely and useful negative result: it challenges the assumption that the benefits of PVRs documented in model-free RL transfer to model-based RL, and it does so with a relatively broad benchmark spanning three domains, two MBRL algorithms, and 14 representation variants. The OOD holdout protocol is principled, the hyperparameter details and code links are provided, and the authors transparently state limitations. However, the strength of the conclusion is moderated by a decoder-target asymmetry in the DreamerV3 integration and by the restriction to frozen backbones; the current phrasing of the abstract overstates the scope of what is actually tested.

major comments (3)
  1. [Section 3.1 / Appendix A.1] In the DreamerV3 PVR variant, the world model decodes the PVR embedding x_t instead of the full input image o_t, while the from-scratch baseline reconstructs 64x64 (DMC/Miniworld) or 128x128 (ManiSkill2) RGB images. DreamerV3's reconstruction loss is a primary learning signal for the world model, so this changes the training objective between conditions. The scratch advantage may therefore be an artifact of the reconstruction target rather than of the PVR itself. The in-domain autoencoder control uses the same embedding-decoding setup, so it cannot rule out this confound. Since the ManiSkill2 and Miniworld claims rest entirely on DreamerV3, the authors need either a pixel-decoding PVR variant or an explicit control that isolates the decoder target before the central negative claim can be supported.
  2. [Abstract / Section 4.1] The abstract and Section 4.1 claim that 'current PVRs are not more sample efficient' than representations learned from scratch, but the experiments only cover frozen PVRs with a single linear layer. The MLP comparison in Appendix C keeps the backbone frozen and is limited to VC-1 and scratch on a few tasks, so it does not address fine-tuning or other adapter architectures that are commonly used in the PVR-for-RL literature. Fine-tuning can change the objective mismatch dynamics that the paper emphasizes. The conclusions should be scoped to 'frozen PVRs' or supplemented by fine-tuning experiments to match the evidence.
  3. [Section 4.3 / Table 3] The property analysis groups at most four PVRs per category, with overlapping category memberships and no statistical test. The categories are also confounded: the ViT group largely overlaps with the diverse-data group (DINOv2, OpenCLIP, VC-1, CLIP-ViT), so the claim that data diversity and network architecture are 'the most important contributors' to OOD generalization is stronger than the descriptive IQM plots can support. A formal analysis, such as a regression or permutation test across the PVR-level scores, is needed to separate architecture from data diversity and to quantify uncertainty.
minor comments (5)
  1. [Appendix A.1] For ManiSkill2, the PVR-based DreamerV3 variants disable symlog while the from-scratch baseline keeps it ('For the ManiSkill environments we found that not applying symlog to the PVR-based instances performs better'); this is a second asymmetric treatment whose effect should be reported and ideally ablated.
  2. [Figure 4] The caption states that 'the baseline representation learned from scratch outperforms all PVRs, even in the OOD settings,' but the plotted standard errors may overlap with some PVRs. The claim would be strengthened by reporting bootstrap confidence intervals or a paired statistical comparison.
  3. [Section 4.3] The word 'suprising' should be 'surprising'.
  4. [Appendix A.1] It would be helpful to state explicitly that the PVR embeddings, rather than raw images, are stored in the replay buffer; this makes the decoder-target difference a design choice rather than a minor implementation detail.
  5. [Section 4.4] The world-model quality analysis is conducted only on the Pendulum-Swingup task, and the correlations (r = -0.22, p = 0.4; r = -0.66, p = 0.004) are computed over a small number of representation-level points; the text should acknowledge the single-task, small-sample nature of this analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central negative result is a direct empirical comparison, not a derived or fitted claim.

full rationale

The paper's central claims—that pre-trained visual representations do not improve MBRL sample efficiency or OOD generalization, and that from-scratch representations often outperform PVRs—are supported by measured returns from benchmark runs rather than by a derivation from fitted parameters or by definitions that presuppose the conclusion. The only equations in the paper are standard normalization and exploration-reward modifications (Eqs. 1 and 2), and neither encodes the PVR-ineffectiveness result. The property analysis in Section 4.3 groups PVRs after seeing the measured outcomes; this is a post hoc narrative reading of the same data, but it is explicitly an explanatory analysis and does not make the headline empirical claim circular. The sole self-citation appears to be Watter et al. [14] in Related Work, a historical MBRL reference that is not load-bearing for the benchmark conclusions. Appendix A.1 states that PVR-based DreamerV3 'decodes the encoding x only and not the whole input image o' while the from-scratch baseline reconstructs 64x64 or 128x128 pixel images; this asymmetry is a potential confound, but a confound is not circularity because the result does not reduce to its inputs by construction. No equation, fitted parameter, or self-citation chain forces the negative result, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters enter the central comparison; hyperparameters are shared across conditions. The property categorizations in Section 4.3 are post hoc groupings, not free parameters. World-model error metrics (KL divergence, MSE) are standard. No new entities are introduced.

assumptions (3)
  • domain assumption DreamerV3 and TD-MPC2 are representative MBRL algorithms; conclusions about 'MBRL' generalize from these two.
    Section 3.1 uses only these two algorithms; no other MBRL family is tested.
  • domain assumption Held-out randomizations of colors, sizes, and object configurations constitute out-of-distribution shifts similar to real-world visual shifts.
    Section 3.3 defines OOD as 20% held-out simulator randomizations; the limitations section notes real-world experiments are still needed.
  • ad hoc to paper Frozen PVR plus a linear layer is an adequate integration of pre-trained representations in MBRL.
    Section 3.1 justifies linear probing by citing linear-probe success for DINOv2, but no fine-tuned PVR baseline is included, which limits the generality of the negative result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/VKVMD6EW

@misc{pith2026241110175,
  author       = {Pith},
  title        = {Pith review of: The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKVMD6EW}},
  note         = {Machine review of arXiv:2411.10175}
}
read the original abstract

Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution (OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors to OOD generalization performance.

Figures

Figures reproduced from arXiv: 2411.10175 by the authors.

Figure 1
Figure 1. Components of our PVR-based DreamerV3 (left) and TD-MPC2 (right) architectures. In DreamerV3, the output xt of the frozen pre-trained vision module g❄ is given to the encoder enc(zt|xt) which maps its input to a discrete latent variable zt. In TD-MPC2 a stack xt−3:t of the last 3 PVR embeddings is given to the encoder enc(xt−3:t) which maps the inputs to fixed-dimensional simplices. The encoder of DreamerV3 addition… view at source ↗
Figure 2
Figure 2. Illustration of tasks ranging from DMC and ManiSkill2 to Miniworld with randomizations. Note that while DMC and Miniworld task images show the perspective of the agents, agents in ManiSkill2 tasks utilize the perspective of a wrist-mounted camera. 3.3 Domains We evaluate all representations across a total of 10 diverse control tasks from 3 different domains: DeepMind Control Suite (DMC) [59], ManiSkill2 [60] and Min… view at source ↗
Figure 3
Figure 3. Normalized ID performance and data-efficiency comparison on DMC, ManiSkill2 and Miniworld environments between the different representations. Each line represents the mean over all runs with a given representation, the shaded area represents the corresponding standard deviation. Solid lines represent DreamerV3 runs, whereas dashed lines indicate TD-MPC2 experiments. Espe￾cially in the DMC experiments, representation… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Average normalized performance on DMC, ManiSkill2 and Miniworld tasks in the OOD setting. The baseline representation learned from scratch outperforms all PVRs, even in the OOD settings. Thin black lines denote the standard error. in-distribution on our own task-specif…
Figure 5
Figure 5. Figure 5: IQM return of the different categorizations. Each marker represents the interquartile￾mean performance of an individual group. The x-axis shows the ID performance and the y-axis the OOD performance. Especially, ViT representations or representations trained on diverse …
Figure 6
Figure 6. Figure 6: Average Accumulated Dynamics Prediction Errors on the Pendulum Swingup task for 200 trajectories. For DreamerV3 we average the forward and backward KL divergence between the prior and posterior distributions of the latent state zt. For TD-MPC2 the MSE between the predi…
Figure 7
Figure 7. Figure 7: Average Accumulated Reward Errors on the Pendulum Swingup task for 200 trajecto￾ries. The error is calculated as the absolute difference between true and predicted reward |rt − rˆt|. Thin black lines denote the standard error. functions to update the policy. Since the …
Figure 8
Figure 8. Figure 8: UMAP projections of DreamerV3 (top row) and TD-MPC2 (bottom row) encodings using different representations as input. The points are color coded by the real perceived reward. Each point represents a visited state in the Pendulum Swingup environment of DMC. The represent…
Figure 9
Figure 9. Figure 9: Performance comparison with representations using linear layers versus multilayer perceptrons. Top row shows normalized ID and OOD performance on DMC environments whereas the bottom row shows the performance on ManiSkill2 environments. Differences between MLPs and line…
Figure 10
Figure 10. Figure 10: Performance and data-efficiency comparison for each task of ManiSkill2, DMC and Miniworld between the different representations. The solid/dashed line shows the mean over multiple runs for DreamerV3/TD-MPC2. The shaded area represents the standard deviation of the res…
Figure 11
Figure 11. Figure 11: IQM return of the different properties on task level. Each marker represents the interquartile-mean performance of an individual group. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Ablation of transformer blocks in VC-1. Using 2 ⁄3 of VC-1 results in similar performance compared to the full model. Transformer blocks near the final one seam to offer as much information as the final output. With only 1 ⁄3 of VC-1 the performance drops significantl…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 55 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748–8763. PMLR, 2021

    2021

  2. [2]

    Cliport: What and where pathways for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894–906. PMLR, 2022

    2022

  3. [3]

    Simple but effective: Clip embeddings for embodied ai

    Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14829–14838, 2022

    2022

  4. [4]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  5. [5]

    Objective mismatch in model-based reinforcement learning

    Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. In Learning for Dynamics and Control , pages 761–770. PMLR, 2020

    work page 2020

  6. [6]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  7. [7]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022

    2022

  8. [8]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    2016

Show all 83 references
  1. [9]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(64...

    2018

  2. [10]

    Mastering atari, go, chess and shogi by planning with a learned model

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  3. [11]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  4. [12]

    Deep rein- forcement learning in a handful of trials using probabilistic dynamics models

    Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep rein- forcement learning in a handful of trials using probabilistic dynamics models. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, edi- tors, Advances in Neural...

    2018

  5. [13]

    Deep dynamics models for learning dexterous manipulation

    Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning , pages 1101–1112. PMLR, 2020

    2020

  6. [14]

    Embed to control: A locally linear latent dynamics model for control from raw images

    Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing ...

    2015

  7. [15]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  8. [16]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555–2565. PMLR, 2019

    2019

  9. [17]

    Dream to control: Learning behaviors by latent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representa- tions, 2020. URL https://openreview.net/forum?id=S1lOTC4tDS

    2020

  10. [18]

    Storm: Efficient stochastic transformer based world models for reinforcement learning

    Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing S...

    2023

  11. [19]

    Models, pixels, and rewards: Evaluating design trade-offs in visual model-based reinforcement learning

    Mohammad Babaeizadeh, Mohammad Taghi Saffar, Danijar Hafner, Harini Kannan, Chelsea Finn, Sergey Levine, and Dumitru Erhan. Models, pixels, and rewards: Evaluating design trade-offs in visual model-based reinforcement learning. arXiv preprint arXiv:2012.04603 , 2020

    2012 arXiv

  12. [20]

    Reinforcement learning with action-free pre-training from videos

    Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning , pages 19561–19579. PMLR, 2022

    2022

  13. [21]

    Pre-training contextual- ized world models with in-the-wild videos for reinforcement learning

    Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextual- ized world models with in-the-wild videos for reinforcement learning. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing ...

    2023

  14. [22]

    Morel: Model-based offline reinforcement learning

    Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages ...

    2020

  15. [23]

    Mopo: Model-based offline policy optimization

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing ...

    2020

  16. [24]

    Combo: Conservative offline model-based policy optimization

    Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information P...

    2021

  17. [25]

    Re- inforcement learning with augmented data

    Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, p...

    2020

  18. [26]

    Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=GY6-6sTvGaf. 13

    2021

  19. [27]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_SJ-_yyes8

    2022

  20. [28]

    Curl: Contrastive unsupervised represen- tations for reinforcement learning

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning , pages 5639–5650. PMLR, 2020

    2020

  21. [29]

    Pre-trained image encoder for generalizable visual reinforcement learning

    Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, YI WU, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Proce...

    2022

  22. [30]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , pages 448–456. PMLR, 2015

    2015

  23. [31]

    R3m: A universal visual representation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In Conference on Robot Learning, pages 892–909. PMLR, 2023

    2023

  24. [32]

    VIP: Towards universal visual reward and representation via value-implicit pre- training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre- training. In The Eleventh International Conference on Learning Representations , 2023. URL https://o...

    2023

  25. [33]

    Real-world robot learning with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416–426. PMLR, 2023

    2023

  26. [34]

    Masked visual pre-training for motor control

    Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022

    2022 arXiv

  27. [35]

    Where are we in the search for an artificial visual cortex for embodied intelligence? In A

    Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, Pieter Abbeel, Jitendra Malik, Dhruv Batra, Yixin Lin, Oleksandr Maksymets, Aravind Rajeswaran, and Franziska Meier. Where are we in the...

    2023

  28. [36]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000–16009, 2022

    2022

  29. [37]

    Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies

    Alexander Sax, Bradley Emi, Amir R Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXiv preprint arXiv:1812.11971, 2018

    2018 arXiv

  30. [38]

    Robust policies via mid-level visual representations: An experimental study in manipulation and navigation

    Bryan Chen, Alexander Sax, Francis Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. In Conference on Robot Learning , pages 2328–2346. P...

    2021

  31. [39]

    Representation matters: Improving perception and exploration for robotics

    Markus Wulfmeier, Arunkumar Byravan, Tim Hertweck, Irina Higgins, Ankush Gupta, Te- jas Kulkarni, Malcolm Reynolds, Denis Teplyashin, Roland Hafner, Thomas Lampe, et al. Representation matters: Improving perception and exploration for robotics. In 2021 IEEE International Confe...

    2021

  32. [40]

    The unsur- prising effectiveness of pre-trained vision models for control

    Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, pages 17359–17371. PMLR, 2022

    2022

  33. [41]

    For pre-trained vision models in motor control, not all policy learning methods are created equal

    Yingdong Hu, Renhao Wang, Li Erran Li, and Yang Gao. For pre-trained vision models in motor control, not all policy learning methods are created equal. In International Conference on Machine Learning, pages 13628–13651. PMLR, 2023

    2023

  34. [42]

    Manan Tomar, Utkarsh Aashu Mishra, Amy Zhang, and Matthew E. Taylor. Learning represen- tations for pixel-based control: What matters and why? Transactions on Machine Learning Re- search, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=wIXHG8LZ2w

    2023

  35. [43]

    The role of pretrained representations for the OOD generalization of RL agents

    Frederik Tr¨auble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Sch¨olkopf, and Stefan Bauer. The role of pretrained representations for the OOD generalization of RL agents. In International C...

    2022

  36. [44]

    What makes pre-trained visual representations successful for robust manipulation? In 8th Annual Conference on Robot Learning , 2024

    Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? In 8th Annual Conference on Robot Learning , 2024. URL https://openreview.net/forum?id= A1hpY5RNiH

    2024

  37. [45]

    On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline

    Nicklas Hansen, Zhecheng Yuan, Yanjie Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline. In International Conference on Machine Learning , pages 12511–12526. PMLR, 2023

    2023

  38. [46]

    TD-MPC2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Oxh5CstDJU

    2024

  39. [47]

    Video prediction models as rewards for re- inforcement learning

    Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for re- inforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, ed...

    2023

  40. [48]

    Learning to model the world with language

    Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. In International Conference on Machine Learning, pages 29992–30017. PMLR, 2024

    2024

  41. [49]

    Transformers are sample-efficient world models

    Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb

    2023

  42. [50]

    Modem: Accelerating visual model-based reinforcement learning with demonstrations

    Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-based reinforcement learning with demonstrations. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/f...

    2023

  43. [51]

    Modem-v2: Visuo- motor world models for real-world robot manipulation

    Patrick Lancaster, Nicklas Hansen, Aravind Rajeswaran, and Vikash Kumar. Modem-v2: Visuo- motor world models for real-world robot manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 7530–7537. IEEE, 2024

    2024

  44. [52]

    Finetuning offline world models in the real world

    Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. In Conference on Robot Learning , pages 425–445. PMLR, 2023

    2023

  45. [53]

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, 15 Ishan Misra, Mich...

    2024

  46. [54]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  47. [55]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778, 2016

    2016

  48. [56]

    Taskonomy: Disentangling task transfer learning

    Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3712–3722, 2018

    2018

  49. [57]

    Open- CLIP

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, and others. Open- CLIP. Zenodo, 4:5, 2021. doi: 10.5281/zenodo.5143773. URL https://doi.org/10.5281/ zenodo.5143773

    2021 doi

  50. [58]

    R2d2: Reliable and repeatable detector and descriptor

    Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Sys- te...

    2019

  51. [59]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  52. [60]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh Internat...

    2023

  53. [61]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In A. Oh, T. Na...

    2023

  54. [62]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026–5033. IEEE, 2012

    2012

  55. [63]

    Generalization in reinforcement learning by soft data augmentation

    Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13611–13617. IEEE, 2021

    2021

  56. [64]

    Sapien: A simulated part-based interactive environ- ment

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environ- ment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages ...

    2020

  57. [65]

    Vision-based manipula- tors need to also see from their hands

    Kyle Hsu, Moo Jin Kim, Rafael Rafailov, Jiajun Wu, and Chelsea Finn. Vision-based manipula- tors need to also see from their hands. In International Conference on Learning Representations,

  58. [66]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Infor...

    2021

  59. [67]

    Foun- dation models for decision making: Problems, methods, and opportunities

    Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023

    2023 arXiv

  60. [68]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio G´omez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Gim´enez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol V...

    2022

  61. [69]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15180–15190, 2023

    2023

  62. [70]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  63. [71]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  64. [72]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning , pages 8469–

  65. [73]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=0oabwyZbOu

    2021

  66. [74]

    Temporal difference learning for model predictive control

    Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pages 8387–8406. PMLR, 2022

    2022

  67. [75]

    Umap: Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018

    2018

  68. [76]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583–8592. PMLR, 2020

    2020

  69. [77]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

  70. [78]

    L2-net: Deep learning of discriminative patch descriptor in euclidean space

    Yurun Tian, Bin Fan, and Fuchao Wu. L2-net: Deep learning of discriminative patch descriptor in euclidean space. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 661–669, 2017

    2017

  71. [79]

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ a1859debfb3b59d094f3504d5ebb6c25-Paper-Datasets_and_Benchmarks.pdf. 17

    2022

  72. [80]

    Benchmarking 6dof outdoor visual localization in changing conditions

    Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In Proceedings of the IEEE conference on compu...

    2018

  73. [81]

    Revisiting oxford and paris: Large-scale image retrieval benchmarking

    Filip Radenovi´c, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondˇrej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5706–5715, 2018

    2018

  74. [83]

    Keypoints into the future: Self- supervised correspondence in model-based reinforcement learning

    Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self- supervised correspondence in model-based reinforcement learning. In Conference on Robot Learning, pages 693–710. PMLR, 2021. 18 A Implementation Details We run all our experiments on a...

    2021

  75. [2022]

    URL https://openreview.net/forum?id=RJkAHKp7kNZ. 16

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.