{"id":"106f03de-8398-47f6-9ec8-be28b76b2ac0","arxiv_id":"2411.10175","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In model-based reinforcement learning, frozen pre-trained visual representations do not improve sample efficiency or out-of-distribution generalization over representations learned from scratch.","lead":"This paper tests whether pre-trained visual features, like those from CLIP or DINOv2, make reinforcement learning agents that plan ahead learn faster and handle unfamiliar scenes better. Across three simulated robot environments, agents that learned their own visual features from scratch matched or beat all pre-trained features.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DreamerV3 PVR comparison changes the decoder target from pixels to the frozen embedding; the negative result may be an artifact of this asymmetric world-model objective.","rationale":"The reader's weakest assumption correctly identifies the DreamerV3 decoder-target asymmetry. I considered other confounds—input resolution (scratch at 64x64 vs PVR at 224x224), unbalanced property groups, and absent statistical tests—but the decoder target is the one most capable of invalidating the central claim even for the paper's own frozen-PVR/linear-layer setup. DreamerV3's reconstruction loss shapes the latent space; predicting a lossy embedding is not equivalent to predicting pixels. This asymmetry affects all DreamerV3 results, including the ManiSkill2 and Miniworld experiments that support the strongest statements, while the decoder-free TD-MPC2 results cover only DMC with a subset of PVRs. The proposed pixel-decoder ablation holds the PVR integration fixed and changes only the reconstruction target, cleanly separating the confound. Because the paper provides no formal verification, an empirical ablation is the appropriate arbiter. If the pixel-decoder variant does not close the gap, the negative result is robust; if it does, the paper's headline claim must be substantially narrowed. This is consistent with the reader's conditional verdict.","tokens_in":22980,"tokens_out":10275,"duration_ms":111634,"concrete_test":"Re-run the DreamerV3 PVR variant (e.g., VC-1 or the in-domain autoencoder) with an additional pixel decoder that reconstructs the original RGB observation from the latent state, keeping the frozen PVR and linear encoder unchanged; compare ID and OOD returns with the current embedding-decoding variant and the from-scratch baseline. If the pixel-decoding variant closes or reverses the gap, the decoder-target asymmetry is the cause; if it remains worse, the negative result is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that frozen PVRs fail to improve MBRL sample efficiency and OOD generalization—rests heavily on a DreamerV3 comparison in which the world model's reconstruction target is not held constant. In Appendix A.1, the PVR variant 'decodes the encoding x only and not the whole input image o', while the from-scratch baseline reconstructs 64x64 (DMC/Miniworld) or 128x128 (ManiSkill2) RGB images. DreamerV3's decoder loss is a primary learning signal; reconstructing a low-dimensional, invariance-trained PVR embedding is a different objective from reconstructing pixels. The scratch advantage could therefore stem from the richer pixel-reconstruction signal, not from the absence of PVR benefit. The in-domain autoencoder control is also affected because it uses the same embedding-decoding setup, so it cannot rule out the confound. TD-MPC2 has no decoder and avoids this issue, but it was run only on DMC with a subset of PVRs; the ManiSkill2 and Miniworld results, which support the strong 'scratch outperforms all PVRs' statement, rely solely on DreamerV3 with this asymmetry. The paper reports the design choice but never tests or discusses it as a potential confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks a set of pre-trained visual representations (PVRs), including CLIP, DINOv2, OpenCLIP, R3M, VIP, VC-1, Taskonomy variants, and an in-distribution autoencoder, in two model-based RL algorithms (DreamerV3 and TD-MPC2) across 10 tasks in DMC, ManiSkill2, and Miniworld. It measures in-distribution sample efficiency and out-of-distribution generalization under visual randomization shifts, and analyzes world-model quality in terms of accumulated dynamics and reward prediction errors. The headline finding is that frozen PVRs with a linear encoder do not improve sample efficiency or OOD generalization over a from-scratch CNN encoder, and that the from-scratch baseline is often better; the paper further attributes OOD success to data diversity and ViT architecture. The evaluation uses 4-6 seeds per condition and 200 evaluation rollouts per seed (1200 per representation-environment), with standard errors shown.","tokens_in":23233,"tokens_out":5487,"duration_ms":51155,"significance":"If the headline result holds, the paper provides a timely and useful negative result: it challenges the assumption that the benefits of PVRs documented in model-free RL transfer to model-based RL, and it does so with a relatively broad benchmark spanning three domains, two MBRL algorithms, and 14 representation variants. The OOD holdout protocol is principled, the hyperparameter details and code links are provided, and the authors transparently state limitations. However, the strength of the conclusion is moderated by a decoder-target asymmetry in the DreamerV3 integration and by the restriction to frozen backbones; the current phrasing of the abstract overstates the scope of what is actually tested.","major_comments":[{"comment":"In the DreamerV3 PVR variant, the world model decodes the PVR embedding x_t instead of the full input image o_t, while the from-scratch baseline reconstructs 64x64 (DMC/Miniworld) or 128x128 (ManiSkill2) RGB images. DreamerV3's reconstruction loss is a primary learning signal for the world model, so this changes the training objective between conditions. The scratch advantage may therefore be an artifact of the reconstruction target rather than of the PVR itself. The in-domain autoencoder control uses the same embedding-decoding setup, so it cannot rule out this confound. Since the ManiSkill2 and Miniworld claims rest entirely on DreamerV3, the authors need either a pixel-decoding PVR variant or an explicit control that isolates the decoder target before the central negative claim can be supported.","section":"Section 3.1 / Appendix A.1"},{"comment":"The abstract and Section 4.1 claim that 'current PVRs are not more sample efficient' than representations learned from scratch, but the experiments only cover frozen PVRs with a single linear layer. The MLP comparison in Appendix C keeps the backbone frozen and is limited to VC-1 and scratch on a few tasks, so it does not address fine-tuning or other adapter architectures that are commonly used in the PVR-for-RL literature. Fine-tuning can change the objective mismatch dynamics that the paper emphasizes. The conclusions should be scoped to 'frozen PVRs' or supplemented by fine-tuning experiments to match the evidence.","section":"Abstract / Section 4.1"},{"comment":"The property analysis groups at most four PVRs per category, with overlapping category memberships and no statistical test. The categories are also confounded: the ViT group largely overlaps with the diverse-data group (DINOv2, OpenCLIP, VC-1, CLIP-ViT), so the claim that data diversity and network architecture are 'the most important contributors' to OOD generalization is stronger than the descriptive IQM plots can support. A formal analysis, such as a regression or permutation test across the PVR-level scores, is needed to separate architecture from data diversity and to quantify uncertainty.","section":"Section 4.3 / Table 3"}],"minor_comments":[{"comment":"For ManiSkill2, the PVR-based DreamerV3 variants disable symlog while the from-scratch baseline keeps it ('For the ManiSkill environments we found that not applying symlog to the PVR-based instances performs better'); this is a second asymmetric treatment whose effect should be reported and ideally ablated.","section":"Appendix A.1"},{"comment":"The caption states that 'the baseline representation learned from scratch outperforms all PVRs, even in the OOD settings,' but the plotted standard errors may overlap with some PVRs. The claim would be strengthened by reporting bootstrap confidence intervals or a paired statistical comparison.","section":"Figure 4"},{"comment":"The word 'suprising' should be 'surprising'.","section":"Section 4.3"},{"comment":"It would be helpful to state explicitly that the PVR embeddings, rather than raw images, are stored in the replay buffer; this makes the decoder-target difference a design choice rather than a minor implementation detail.","section":"Appendix A.1"},{"comment":"The world-model quality analysis is conducted only on the Pendulum-Swingup task, and the correlations (r = -0.22, p = 0.4; r = -0.66, p = 0.004) are computed over a small number of representation-level points; the text should acknowledge the single-task, small-sample nature of this analysis.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical benchmarking effort, and the central negative result would be valuable if supported. However, the decoder-target asymmetry in DreamerV3 is load-bearing for the ManiSkill2 and Miniworld claims, and the abstract overstates the frozen-PVR scope. A pixel-decoding control or a clearly scoped formulation, plus a more rigorous property analysis, would bring the manuscript in line with its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely new: it benchmarks 14 pre-trained visual representations against a from-scratch CNN across DMC, ManiSkill2, and Miniworld, with OOD splits, multiple seeds, and 1200 evaluation episodes per representation. That is real work, and the question—do frozen PVRs help model-based RL the way they help model-free RL—is worth asking. The finding that from-scratch representations are at least as data-efficient and often better even OOD is consistent with the from-scratch-is-strong result from Hansen et al., and extends it to MBRL. Credit where due: the world-model analysis is a nice addition, and the correlational link between reward prediction error and task performance is informative, even if post hoc.\n\nThe soft spot is the DreamerV3 comparison. Appendix A.1 states that the PVR variant decodes the frozen embedding only, not the original image, while the from-scratch baseline reconstructs pixels. DreamerV3's decoder loss is a primary learning signal; decoding an invariance-trained embedding is a different objective. The in-domain autoencoder control uses the same embedding-decoding setup, so it cannot rule out the confound. TD-MPC2 has no decoder and avoids the issue, but it was only run on DMC with a subset of PVRs. The ManiSkill2 and Miniworld results—the ones that support the strong 'scratch outperforms all PVRs' statement—rest solely on the confounded DreamerV3 comparison. The paper reports the design choice but never tests or discusses it as a confound, which is a genuine omission, though not a hidden one.\n\nThe property analysis (data diversity, ViT architecture) is explicitly post hoc grouping of the same data, so I read it as exploratory rather than decisive. The abstract also overgeneralizes: the experiments only cover frozen PVRs with a linear layer, not fine-tuned PVRs, and the claims should be scoped accordingly. The limitations section is honest but does not mention the decoder asymmetry.\n\nWould I trust the central negative result as stated? Not fully. The DreamerV3 evidence is weakened by the confound, and the cross-domain generality currently depends on it. But the benchmark itself is valuable, the question is important, and the flaws are addressable. This paper deserves a serious referee, not a desk reject, and a careful revision could make it a solid reference point.\n\nFor peer review: yes, send it out. For a reading group: yes, it will provoke discussion about representation integration and evaluation design. I would cite it, cautiously, as the first MBRL-specific PVR benchmark.","headline":"A substantial first MBRL-specific PVR benchmark whose headline negative result is plausible but partly confounded by the DreamerV3 decoder-target asymmetry.","tokens_in":23764,"tokens_out":1541,"would_cite":true,"duration_ms":16071,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On three visual control benchmarks, model-based RL agents with pre-trained visual representations are no more sample-efficient and no better at out-of-distribution generalization than agents that learn their encoder from scratch.","keywords":["model-based reinforcement learning","pre-trained visual representations","sample efficiency","out-of-distribution generalization","world models","DreamerV3","TD-MPC2","visual control benchmarks"],"falsifier":"Train a from-scratch DreamerV3 agent whose decoder reconstructs a low-dimensional embedding of the same size as a PVR output, rather than the raw $64\\times64$ image; if performance drops to the PVR level, the reconstruction target is the cause. The reverse test, letting a PVR-based agent reconstruct the raw image through the frozen encoder and showing it beats from scratch, would overturn the paper's conclusion.","tokens_in":22793,"feed_emoji":"🤖","tokens_out":10745,"duration_ms":91724,"temperature":0.7,"pith_summary":"This paper asks whether frozen pre-trained visual representations (PVRs) — vision models trained on large image and video datasets, such as CLIP, DINOv2, R3M, and VC-1 — give model-based reinforcement learning agents an edge in sample efficiency and out-of-distribution (OOD) generalization. The question matters because PVRs are widely assumed to pass their robustness on to downstream control, and model-based RL is a leading route to data-efficient agents. The authors benchmark thirteen PVR variants inside DreamerV3 and TD-MPC2 on ten tasks from three domains and find that representations learned from scratch are in most cases equally or more data-efficient, and that the from-scratch baseline outperforms every PVR in OOD evaluation. Even autoencoders pre-trained on the downstream task's own data fail to beat the scratch baseline. The paper attributes the gap to world-model quality: PVR-based agents learn comparable dynamics predictions but worse reward predictions, and their latent spaces entangle low- and high-reward states; among PVR properties, data diversity and a ViT architecture are the strongest contributors to OOD performance.","feed_headline":"Frozen vision features fail to help model-based RL","feed_subtitle":"Scratch encoders match or beat CLIP, DINOv2, VC-1 on sample efficiency and OOD generalization across three benchmarks.","key_machinery":"The argument runs through a specific integration scheme: the frozen PVR acts as a fixed feature extractor whose output $x_t$ replaces the raw image as the input to the MBRL agent, with only a single linear layer trained on top to map $x_t$ into the latent state (for DreamerV3, the linear map feeds the discrete-latent encoder; for TD-MPC2, a stack of three embeddings is fed to its encoder). The paper then dissects results with three instruments: normalized ID/OOD return curves, a categorization of PVRs by training properties (ViT vs. non-ViT, diverse data, sequential data, language conditioning), and world-model diagnostics on a pendulum task measuring accumulated dynamics prediction error, accumulated reward error, and UMAP projections of the latent state colored by reward. Notably, in DreamerV3 the decoder reconstructs the frozen embedding $x_t$, not the original image observation.","core_discovery":"The central discovery, on the paper's own terms, is that for model-based RL the transfer benefits of pre-trained visual representations do not materialize. Across DreamerV3 and TD-MPC2 on DMC, ManiSkill2, and Miniworld, a representation learned from scratch is at least as data-efficient as any of the thirteen PVR variants and often better, and it also wins in held-out out-of-distribution settings. The paper attributes this to the objective mismatch of MBRL and to the information bottleneck of PVRs: frozen features trained without a reward signal do not preserve the reward information the world model needs, so PVR-based agents struggle to predict rewards accurately (reward prediction error correlates negatively with task performance, $r = -0.66$), while dynamics prediction quality is comparable across representations. The authors conclude that the common assumption that PVRs improve model-based agents — carried over from model-free RL and imitation learning — does not hold for current PVRs, and that data diversity and network architecture, not language conditioning or sequential pre-training data, are what drive OOD generalization.","pith_inferences":["The comparison is asymmetric: DreamerV3 with a PVR decodes the frozen embedding, while the from-scratch baseline reconstructs the raw $64\\times64$ or $128\\times128$ image; if the reconstruction target changes what the world model learns, part of the scratch advantage could be an artifact of this integration rather than of PVRs. This confound is stated in the appendix but never tested.","A direct test would be to let a from-scratch encoder also decode a low-dimensional embedding instead of pixels, or to train a PVR-based agent with a decoder that reconstructs pixels through the frozen features; either design would isolate the representation from the interface.","The reward-bottleneck result suggests a concrete recipe for future PVRs aimed at MBRL: pre-training objectives should include reward or value information, such as value-implicit or goal-conditioned losses, rather than only reconstruction or contrastive image-text alignment.","Because sequential training data helped in manipulation and navigation domains but not in DMC, the usefulness of video pre-training may depend on the structure of the downstream task; testing more control families could reveal when temporal priors matter."],"forward_implications":["Model-based RL practitioners gain nothing, in these settings, from swapping in a frozen CLIP-, DINOv2-, or VC-1-class encoder instead of training a CNN from scratch; data efficiency is equal or worse.","OOD generalization of the agent is not inherited from the PVR: held-out visual shifts are handled better by the scratch encoder in the benchmark.","Because reward prediction error, not dynamics prediction error, tracks task performance, improving a PVR-based world model should focus on reward-relevant information, not on more accurate state prediction.","Data diversity and ViT backbone are the PVR properties that matter most for OOD performance; language conditioning and sequential pre-training data play minor roles.","Pre-training an encoder on the downstream task's own data distribution does not help either, so the failure is not simply domain mismatch."],"supporting_citations":[{"why":"DreamerV3 is the primary model-based RL algorithm whose encoder is replaced by the PVRs in the benchmark.","marker":"[11]"},{"why":"TD-MPC2 is the second model-based algorithm used to confirm the pattern on a subset of PVRs.","marker":"[46]"},{"why":"The objective-mismatch account is the paper's proposed explanation for why adapting an existing representation is harder than learning one from scratch in MBRL.","marker":"[5]"},{"why":"Closest prior work that compared PVRs against a learning-from-scratch baseline; this paper extends that comparison to model-based RL.","marker":"[45]"},{"why":"Provides the VC-1 PVR and the Ego4D-MNI pre-training data used as one of the benchmarked representations.","marker":"[35]"},{"why":"Provides the CLIP vision encoders (ViT and ResNet variants) that are among the strongest PVRs tested.","marker":"[1]"},{"why":"Provides R3M, the robot-manipulation PVR trained on Ego4D with time-contrastive and language-alignment objectives.","marker":"[31]"},{"why":"Provides DINOv2 and motivates the linear-encoder integration by showing its features can solve downstream tasks with a single linear layer.","marker":"[53]"}],"fun_headline_variants":["Pre-trained visual features don't boost model-based RL","Scratch beats CLIP, DINOv2 for model-based RL","PVRs fail to improve sample efficiency in MBRL","Frozen visual encoders hurt MBRL reward prediction","Data diversity matters more than pre-training in MBRL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that decoding the frozen pre-trained features instead of the original image is an equivalent way to train the agent's internal predictor of how the environment evolves; if the reconstruction target changes what that predictor learns, the from-scratch advantage could come from the interface rather than from the representations.","fun_headline_variants_meta":{"raw":{"variants":["Pre-trained visual features don't boost model-based RL","Scratch beats CLIP, DINOv2 for model-based RL","PVRs fail to improve sample efficiency in MBRL","Frozen visual encoders hurt MBRL reward prediction","Data diversity matters more than pre-training in MBRL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1640,"prompt_tokens":972,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":587}},"tokens_in":588,"tokens_out":668,"duration_ms":6029,"temperature":1.0,"reasoning_tokens":587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:52:46.812763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a from-scratch DreamerV3 agent whose decoder reconstructs a low-dimensional embedding of the same size as a PVR output, rather than the raw $64\\times64$ image; if performance drops to the PVR level, the reconstruction target is the cause. The reverse test, letting a PVR-based agent reconstruct the raw image through the frozen encoder and showing it beats from scratch, would overturn the paper's conclusion.","supporting_citations":[{"cited_title":"Objective mismatch in model-based reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The objective-mismatch account is the paper's proposed explanation for why adapting an existing representation is harder than learning one from scratch in MBRL."},{"cited_title":"On pre-training for visuo-motor control: Revisiting a learning- from-scratch baseline","cited_arxiv_id":null,"evidence_quote":"Closest prior work that compared PVRs against a learning-from-scratch baseline; this paper extends that comparison to model-based RL."},{"cited_title":"Where are we in the search for an artificial visual cortex for embodied intelligence? In A","cited_arxiv_id":null,"evidence_quote":"Provides the VC-1 PVR and the Ego4D-MNI pre-training data used as one of the benchmarked representations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides DINOv2 and motivates the linear-encoder integration by showing its features can solve downstream tasks with a single linear layer."}],"review_version":1}