{"id":"46e6fa5c-2c2e-444e-a7ca-88e5ceaf5862","arxiv_id":"2412.14312","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Dyna-style model-based RL methods (MBPO, ALM) that add synthetic rollouts help in OpenAI Gym but significantly degrade performance on most DeepMind Control Suite tasks.","lead":"Two popular robot-learning methods that use imagined experience work well on one benchmark, OpenAI Gym, but often fail to learn on a similar benchmark, DeepMind Control Suite. The study shows this gap persists even with better models and other fixes, and provides a faster open-source implementation for testing these algorithms.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's MBPO attribution rests on equating 1-step and original n-step MBPO from Gym-only evidence; if n-step behaves differently in DMC, the central degradation claim does not generalize to MBPO as published.","rationale":"The reader's weakest assumption is the one I would also flag as most load-bearing. The central empirical result is genuinely informative: two independent Dyna-style algorithms (1-step MBPO and ALM) underperform their model-free bases in most DMC tasks, and the controls -- synthetic-to-real ratio sweeps, a perfect-model condition, periodic resets, layer normalization, and large hyperparameter sweeps -- suggest the synthetic-data mechanism matters. Those controls make the observation robust as a phenomenon of these specific configurations. What is not robust is the label 'MBPO': the only evidence that 1-step MBPO is the original MBPO comes from matching final returns in Gym, and the paper explicitly needs this identity to make the DMC results speak to the published algorithm and to the Dyna subclass. DMC is exactly where rollout length could matter, because model errors are much larger there (Section 4.1). Thus the condition that must be true for the central claim -- that the tested arrangement faithfully represents MBPO in DMC -- is the least secure. The conclusion's 'all 15 tasks' overstatement is real but does not threaten the 'most environments' version of the claim; the variant-equivalence gap does. I therefore agree with the reader's conditional verdict and would keep it unchanged pending the n-step test or a scoped rewrite.","tokens_in":18755,"tokens_out":8070,"duration_ms":73102,"concrete_test":"Run the original PyTorch n-step MBPO (Janner, 2019) on the six DMC tasks in Figure 4, with its unmodified adaptive rollout-length schedule and original hyperparameters, for at least 500k environment steps and 6 seeds. Compare final returns against (i) the paper's JAX 1-step MBPO and (ii) the paper's SAC baseline. If n-step MBPO also cannot improve over a random policy, the paper's attribution to MBPO holds; if n-step MBPO matches or exceeds SAC, or substantially outperforms the 1-step variant, then the negative result must be scoped to the 1-step variant and the abstract's 'MBPO' claim revised. As a cheaper secondary check, enable the original MBPO rollout-length schedule within the released JAX code and repeat on hopper-stand and humanoid-stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.1 the authors write: \"Combining the results in Figure 2 and Figure 3 we can conclude that our JAX-based implementation of 1-step MBPO matches the original implementation's n-step MBPO's performance...\" and then say \"we will henceforth refer to 1-step MBPO as simply MBPO.\" This is the bridge that lets the DMC results in Sections 3.2 through 5 speak about MBPO, and the bridge is built only on OpenAI Gym final returns. DMC is not a small perturbation of Gym: Section 4.1 reports model errors above 100% on hopper tasks and above 25% on other failure tasks, and the original MBPO's n-step rollouts are specifically designed to adapt to compounding error via a rollout-length threshold. A hardcoded 1-step rollout removes exactly the length dimension that could change how synthetic data interacts with high-error dynamics. Consequently, even the perfect-model experiment in Section 4.2 is a perfect-model version of the 1-step algorithm, not of the original MBPO. If n-step MBPO succeeds in DMC, the paper's central claim that synthetic rollouts degrade performance is an artifact of the 1-step configuration, and only the ALM results would support the broader Dyna-family claim. The conclusion's \"all 15 DMC tasks ... consistently undermined\" is also stronger than the body's own \"six out of fifteen\" failure language, but the variant-equivalence gap is the more load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of Dyna-style model-based reinforcement learning (DMBRL), centered on MBPO and ALM, across OpenAI Gym and DeepMind Control Suite (DMC) tasks with proprioceptive observations. The main observation is that MBPO and ALM, which perform well in Gym, largely fail to improve over their model-free base algorithms in DMC, despite similar task structure and the same physics engine. The authors then investigate possible causes—predictive model error, critic divergence, and plasticity loss—and show that standard remedies (better model utilization, layer norm, periodic resets) do not restore competitive performance. They also introduce a JAX-based MBPO implementation with reported speedups and release code. The paper concludes that synthetic rollouts, the backbone of Dyna-style methods, can degrade performance and that the community's benchmark reliance has inflated confidence in these methods.","tokens_in":18992,"tokens_out":2979,"duration_ms":27666,"significance":"If the core claims hold, the paper is a useful and timely cautionary result: it documents a concrete generalization failure of two influential DMBRL algorithms, reproduces across 15 DMC tasks with 6 seeds, includes full training curves in the appendix, and provides an oracle-model experiment that separates model error from other failure sources. The released JAX implementation is a practical contribution that lowers the computational barrier for follow-up work. However, the strength of the central attribution to 'MBPO' and the broad 'Dyna-style methods' conclusion depends on an equivalence claim between the implemented 1-step variant and the original n-step MBPO, which is only established on Gym final returns. That gap, plus a conclusion that overstates the body's own results, currently tempers the significance of the headline message.","major_comments":[{"comment":"The paper's core attribution to MBPO rests on the statement, 'Combining the results in Figure 2 and Figure 3 we can conclude that our JAX-based implementation of 1-step MBPO matches the original implementation's n-step MBPO's performance... and as such we will henceforth refer to 1-step MBPO as simply MBPO.' This equivalence is established only on six OpenAI Gym final returns. Since the DMC results in Sections 3.2 through 5 are the load-bearing evidence for the paper's central claim, the 1-step to n-step bridge needs to be validated in DMC, where the paper itself reports model errors above 100% on hopper tasks and above 25% on other failure tasks. A hardcoded rollout length of 1 removes exactly the adaptive horizon mechanism that n-step MBPO uses to balance model bias and compounding error. As written, the paper characterizes 'MBPO' as known in the literature, but the evidence supports a claim about a 1-step MBPO configuration. Please either run the original n-step MBPO on the DMC tasks or explicitly re-scope the claims to the 1-step variant and adjust the title, abstract, and conclusion accordingly.","section":"Section 3.1"},{"comment":"The conclusion states that 'for all 15 DMC tasks we examined, adding model-generated samples consistently undermined both sample efficiency and wall-clock performance relative to simpler, model-free off-policy algorithms with identical hyperparameters.' This is inconsistent with the body. Section 3.2 says MBPO fails to improve the policy in 'six out of fifteen' environments, and the full results in Figure 11 and Figure 17 explicitly include scenarios where MBPO 'matches SAC's sample efficiency' or solves tasks more slowly. 'Consistently undermined' is therefore too strong for the presented evidence. The conclusion should be reworded to reflect the actual distribution: a subset of tasks shows complete failure, a larger set shows underperformance, and a few show matching or slower improvement. This is not merely a wording issue because the abstract and conclusion are what readers will cite.","section":"Section 6"},{"comment":"The perfect-model experiment is well designed as a diagnostic, but it is a perfect-model version of the 1-step algorithm, not of the original n-step MBPO. With a perfect model, longer rollouts are no longer penalized by compounding error, so the natural comparison would include an n-step perfect-model condition. The current experiment therefore cannot rule out the possibility that the original n-step MBPO, even with a learned model, would behave differently in DMC. Please either report an n-step perfect-model condition or explicitly state that this experiment only tests the 1-step variant.","section":"Section 4.2"}],"minor_comments":[{"comment":"The abstract says the gap 'significantly degrades performance across most DMC environments,' while Section 3.2 says MBPO fails in 'six out of fifteen' environments. The paper should harmonize these counts and use a consistent classification of failure, underperformance, and matching behavior.","section":"Abstract and Section 3.2"},{"comment":"The percent model error formula is presented without an equation number; numbering it would make the later references in Sections 4.2 and Appendix G easier to follow.","section":"Section 4.1"},{"comment":"The caption says 'the hopper-stand and humanoid-stand environment' (singular); it should say 'environments.' In addition, the two panels (a) and (b) should be labeled clearly so readers know which panel corresponds to which environment.","section":"Figure 10 caption"},{"comment":"The reference to Voelcker et al. appears with an extra space ('V oelcker') in several places; please fix the formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is reasonable. The empirical core—6-seed runs, full appendix curves, oracle model, and released code—is solid and worth publishing after revision. The main risk is the 1-step to n-step equivalence claim, which is load-bearing for the paper's attribution to MBPO as published. I would encourage the editor to ask for either an additional n-step DMC experiment or a careful rescoping of all claims to the 1-step variant. The conclusion overstates the body's results and should be corrected as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is a useful negative result with good experimental hygiene, but it overreaches on two fronts. The headline finding—that a JAX-based 1-step MBPO and ALM fail to beat their model-free bases across most DMC tasks—is credible and worth knowing. The paper generalizes the hopper-hop observation to 15 DMC tasks, tests ALM as a second Dyna-style method, includes a perfect-model oracle, layer norm, resets, and hyperparameter sweeps, and ships code that runs dramatically faster than the original MBPO. The perfect-model experiment is a genuinely useful control: even with oracle rollouts, the 1-step method doesn't reliably beat SAC. That's a real contribution.\n\nThe soft spots are in the attribution and the conclusion. The paper equates its 1-step MBPO with the original n-step MBPO based only on Gym final returns (Section 3.1), then refers to the variant as \"MBPO\" throughout. Original MBPO's rollout length is an adaptive mechanism for handling compounding model error, and DMC has much higher model error (Section 4.1). So the central claim that \"MBPO\" degrades performance in DMC is not actually established for the algorithm as published—only for the 1-step variant. The ALM results, which use the original implementation, do support a broader Dyna-family concern, but the MBPO-specific attribution is unproven. The authors should run n-step MBPO in DMC or explicitly scope the title and abstract to 1-step rollouts.\n\nSecond, the conclusion says synthetic rollouts \"consistently undermined\" performance on all 15 DMC tasks, but the body says six out of fifteen failed and the rest matched or solved slower. That's an internal inconsistency and should be fixed. The \"no free lunch\" framing is also rhetorical: two algorithms don't prove a subclass-level limitation, and DreamerV3 succeeds in DMC.\n\nNone of this sinks the core observation. The experimental work is careful, the code is released, and the paper is honest about what it didn't test. It deserves a serious referee. I'd push for revision rather than rejection, and I'd want the authors to address the n-step equivalence explicitly. I'd cite it as evidence that 1-step Dyna-style methods transfer poorly to DMC, not as a general verdict on MBPO.","headline":"Useful negative result, but the MBPO-specific conclusion overreaches: the DMC experiments run a 1-step variant, and the equivalence to original n-step MBPO is only shown in Gym.","tokens_in":19599,"tokens_out":3170,"would_cite":true,"duration_ms":26728,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that adding synthetic rollouts—the defining step of Dyna-style model-based reinforcement learning—significantly degrades performance in most DeepMind Control Suite tasks, even though the same algorithms gain sample…","keywords":["Dyna-style model-based RL","synthetic rollouts","sample efficiency","benchmark generalization","MBPO","ALM","DeepMind Control Suite","replay ratio"],"falsifier":"Run the original MBPO implementation with its original multi-step rollouts and hyperparameters on the six DMC tasks where the paper's 1-step MBPO shows no improvement, and compare final returns to SAC. If the original MBPO matches or beats SAC on those tasks, the paper's claim that synthetic rollouts degrade performance in DMC would not hold for MBPO as published.","tokens_in":18469,"feed_emoji":"🤖","tokens_out":9022,"duration_ms":71326,"temperature":0.7,"pith_summary":"This paper reports that a family of reinforcement learning methods known as Dyna-style model-based RL—algorithms that supplement real experience with imagined transitions from a learned world model—can lose their sample-efficiency advantage on a different benchmark of the same physical tasks. In OpenAI Gym, MBPO and ALM beat their model-free base algorithms; in the DeepMind Control Suite, with identical hyperparameters, they often fail to improve beyond a randomly initialized policy. The authors test whether model accuracy, critic divergence, network plasticity, or hyperparameter tuning explain the gap, and find that none of these fixes restores consistent gains. The central claim is that adding synthetic rollouts, the defining step of Dyna-style methods, can actively degrade performance in some environments, so there is no free lunch across benchmarks.","feed_headline":"Adding synthetic rollouts stalls RL agents in DMC tasks","feed_subtitle":"Imagined transitions that speed up MBPO in OpenAI Gym keep it stuck near random-policy level in most DeepMind Control Suite tasks.","key_machinery":"The central object is the synthetic rollout: a transition sampled from a learned dynamics model, branched from a real state in the replay buffer, and mixed into the off-policy training batch at a synthetic-to-real ratio of 0.95 (MBPO's default). The diagnostic machinery includes percent model error (normalized prediction error measured on the training distribution), sweeping the synthetic-to-real ratio, measuring average critic Q-values to expose divergence, and testing replay ratio and plasticity via periodic parameter resets. The argument runs through this machinery: synthetic transitions dominate the batch, their inaccuracy (and, in the perfect-model experiment, even their accuracy) interacts with MBPO's twenty updates per environment interaction to push the critic's Q-values out of line, and the resulting divergence blocks policy improvement.","core_discovery":"The paper's core discovery is that the value of synthetic rollouts is environment-dependent in a way that breaks the standard sample-efficiency narrative. The authors implement MBPO with single-step rollouts and reproduce its Gym advantage over SAC, then show the same algorithm with the same hyperparameters fails to improve in six of fifteen DMC tasks and lags SAC across most of the remaining ones. ALM, a latent-space Dyna-style method built on a deterministic DDPG base, shows the same pattern, which rules out an artifact of SAC or of MBPO's specific design. Using an augmented simulator that supplies perfect one-step transitions, they find that even a perfect model does not allow MBPO to consistently match SAC in DMC. The paper concludes that the Dyna-style augmentation loop itself—imagined data generated from a learned model and replayed at high frequency—can arrest policy improvement, and that this limitation applies to a subclass of Dyna-style algorithms rather than to model-based RL as a whole.","pith_inferences":["Editorial: If a direct n-step MBPO run in DMC fails to reproduce the gap, the paper's characterization of MBPO as such would need to be narrowed to 1-step rollouts; this is the sharpest test the paper leaves open.","Editorial: The dose-response pattern in the paper's synthetic-to-real ratio sweeps suggests a testable prediction: other Dyna-style methods will show monotone performance collapse as the fraction of imagined data in each batch increases in DMC.","Editorial: The results push toward reporting per-environment results rather than averaged returns, since averaged curves can hide environments where an algorithm never leaves random-policy level.","Editorial: A broader implication is that sample-efficiency gains in model-based RL may be less a property of the algorithm than of the benchmark's reward, termination, and integration structure; the paper documents these differences but does not isolate a cause."],"forward_implications":["Published sample-efficiency claims for MBPO and related Dyna-style methods should be treated as benchmark-specific until re-run in DMC or similar suites.","Because removing synthetic rollouts restores strong performance, the failure is in the Dyna-style augmentation loop, not in the underlying off-policy learner.","The standard remedies examined—better or tuned model, perfect model, layer normalization, periodic resets—do not consistently recover Gym-level gains, so the gap is not a simple tuning artifact.","The results distinguish a subclass of Dyna-style algorithms that struggle in DMC from model-based RL as a whole; DreamerV3's DMC success is consistent with this subclass-level conclusion.","The large wall-clock speedup of the paper's implementation makes broad re-evaluation of other Dyna-style algorithms practical on a single GPU."],"supporting_citations":[{"why":"Defines the Dyna architecture whose synthetic-rollout training loop is the object under test.","marker":"(Sutton, 1991)"},{"why":"Introduces MBPO and reports the Gym sample-efficiency gains that the paper re-implements and then tests in DMC.","marker":"(Janner et al., 2019)"},{"why":"Introduces ALM, the second Dyna-style algorithm used to show the gap is not specific to MBPO.","marker":"(Ghugare et al., 2022)"},{"why":"Defines the DeepMind Control Suite benchmark where the performance gap appears.","marker":"(Tassa et al., 2020)"},{"why":"Previous report that MBPO cannot reliably solve hopper-hop in DMC; the paper generalizes this to many tasks.","marker":"V oelcker et al., 2024"},{"why":"Supplies the high-performance SAC base used in the JAX reimplementation, enabling the paper's large DMC experiments.","marker":"(D'Oro et al., 2023)"},{"why":"Provides the six challenging DMC tasks and the primacy-bias/plasticity framing used in Section 5.","marker":"(Nikishin et al., 2022)"},{"why":"Motivates periodic model reset experiments that the paper runs and finds insufficient.","marker":"(Qiao et al., 2023)"},{"why":"Supplies the critic-divergence analysis that motivates the Q-value measurements and layer-normalization test.","marker":"(Nauman et al., 2024)"}],"fun_headline_variants":["Imagined rollouts that boost Gym RL fail on DeepMind tasks","Dyna-style RL free lunch vanishes on DeepMind Control Suite","Synthetic transitions help OpenAI Gym but hurt DeepMind Control","RL's imagined data: good in Gym, bad in DeepMind tasks","No free lunch: Dyna-style RL agents stall on DMC benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dividing assumption is that the paper's version of MBPO, which generates one imagined step per rollout, is equivalent to the original MBPO, which generates several; that match is established only in OpenAI Gym, so if longer imagined rollouts behave differently in the DeepMind Control Suite, the reported failure may belong to this particular variant.","fun_headline_variants_meta":{"raw":{"variants":["Imagined rollouts that boost Gym RL fail on DeepMind tasks","Dyna-style RL free lunch vanishes on DeepMind Control Suite","Synthetic transitions help OpenAI Gym but hurt DeepMind Control","RL's imagined data: good in Gym, bad in DeepMind tasks","No free lunch: Dyna-style RL agents stall on DMC benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2733,"prompt_tokens":927,"completion_tokens":1806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1715}},"tokens_in":543,"tokens_out":1806,"duration_ms":12938,"temperature":1.0,"reasoning_tokens":1715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:20:37.800014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original MBPO implementation with its original multi-step rollouts and hyperparameters on the six DMC tasks where the paper's 1-step MBPO shows no improvement, and compare final returns to SAC. If the original MBPO matches or beats SAC on those tasks, the paper's claim that synthetic rollouts degrade performance in DMC would not hold for MBPO as published.","supporting_citations":[{"cited_title":"Simplifying model-based rl: learning representations, latent-space models, and policies with one objective","cited_arxiv_id":null,"evidence_quote":"Introduces ALM, the second Dyna-style algorithm used to show the gap is not specific to MBPO."},{"cited_title":"G., and Courville, A","cited_arxiv_id":null,"evidence_quote":"Supplies the high-performance SAC base used in the JAX reimplementation, enabling the paper's large DMC experiments."},{"cited_title":"The primacy bias in deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the six challenging DMC tasks and the primacy-bias/plasticity framing used in Section 5."},{"cited_title":"Mind the model, not the agent: the primacy bias in model-based RL","cited_arxiv_id":null,"evidence_quote":"Motivates periodic model reset experiments that the paper runs and finds insufficient."},{"cited_title":"Overestimation, overfitting, and plasticity in actor-critic: the Bitter lesson of Reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the critic-divergence analysis that motivates the Q-value measurements and layer-normalization test."}],"review_version":1}