Pith. sign in

REVIEW 3 major objections 4 minor 57 references

Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that adding synthetic rollouts—the defining step of Dyna-style model-based reinforcement learning—significantly degrades performance in most DeepMind Control Suite tasks, even though the same algorithms gain sample…

desk verdict Useful negative result, but the MBPO-specific conclusion overreaches: the DMC experiments run a 1-step variant, and the equivalence to original n-step MBPO is only shown in Gym. read the letter →

arxiv 2412.14312 v3 pith:EVN5LVHA submitted 2024-12-18 cs.LG

Brett Barkley , David Fridovich-Keil This is my paper · ORCID
classification cs.LG
keywords Dyna-stylemodel-basedRLsyntheticrolloutssampleefficiencybenchmarkgeneralizationMBPOALMDeepMindControlSuitereplayratio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports that a family of reinforcement learning methods known as Dyna-style model-based RL—algorithms that supplement real experience with imagined transitions from a learned world model—can lose their sample-efficiency advantage on a different benchmark of the same physical tasks. In OpenAI Gym, MBPO and ALM beat their model-free base algorithms; in the DeepMind Control Suite, with identical hyperparameters, they often fail to improve beyond a randomly initialized policy. The authors test whether model accuracy, critic divergence, network plasticity, or hyperparameter tuning explain the gap, and find that none of these fixes restores consistent gains. The central claim is that adding synthetic rollouts, the defining step of Dyna-style methods, can actively degrade performance in some environments, so there is no free lunch across benchmarks.

What carries the argument

The central object is the synthetic rollout: a transition sampled from a learned dynamics model, branched from a real state in the replay buffer, and mixed into the off-policy training batch at a synthetic-to-real ratio of 0.95 (MBPO's default). The diagnostic machinery includes percent model error (normalized prediction error measured on the training distribution), sweeping the synthetic-to-real ratio, measuring average critic Q-values to expose divergence, and testing replay ratio and plasticity via periodic parameter resets. The argument runs through this machinery: synthetic transitions dominate the batch, their inaccuracy (and, in the perfect-model experiment, even their accuracy) interacts with MBPO's twenty updates per environment interaction to push the critic's Q-values out of line, and the resulting divergence blocks policy improvement.

What would settle it

Run the original MBPO implementation with its original multi-step rollouts and hyperparameters on the six DMC tasks where the paper's 1-step MBPO shows no improvement, and compare final returns to SAC. If the original MBPO matches or beats SAC on those tasks, the paper's claim that synthetic rollouts degrade performance in DMC would not hold for MBPO as published.

Watch

Extended reading notes

Core claim

The paper's core discovery is that the value of synthetic rollouts is environment-dependent in a way that breaks the standard sample-efficiency narrative. The authors implement MBPO with single-step rollouts and reproduce its Gym advantage over SAC, then show the same algorithm with the same hyperparameters fails to improve in six of fifteen DMC tasks and lags SAC across most of the remaining ones. ALM, a latent-space Dyna-style method built on a deterministic DDPG base, shows the same pattern, which rules out an artifact of SAC or of MBPO's specific design. Using an augmented simulator that supplies perfect one-step transitions, they find that even a perfect model does not allow MBPO to consistently match SAC in DMC. The paper concludes that the Dyna-style augmentation loop itself—imagined data generated from a learned model and replayed at high frequency—can arrest policy improvement, and that this limitation applies to a subclass of Dyna-style algorithms rather than to model-based RL as a whole.

Load-bearing premise

The dividing assumption is that the paper's version of MBPO, which generates one imagined step per rollout, is equivalent to the original MBPO, which generates several; that match is established only in OpenAI Gym, so if longer imagined rollouts behave differently in the DeepMind Control Suite, the reported failure may belong to this particular variant.

Editorial extensions

If this is right

  • Published sample-efficiency claims for MBPO and related Dyna-style methods should be treated as benchmark-specific until re-run in DMC or similar suites.
  • Because removing synthetic rollouts restores strong performance, the failure is in the Dyna-style augmentation loop, not in the underlying off-policy learner.
  • The standard remedies examined—better or tuned model, perfect model, layer normalization, periodic resets—do not consistently recover Gym-level gains, so the gap is not a simple tuning artifact.
  • The results distinguish a subclass of Dyna-style algorithms that struggle in DMC from model-based RL as a whole; DreamerV3's DMC success is consistent with this subclass-level conclusion.
  • The large wall-clock speedup of the paper's implementation makes broad re-evaluation of other Dyna-style algorithms practical on a single GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If a direct n-step MBPO run in DMC fails to reproduce the gap, the paper's characterization of MBPO as such would need to be narrowed to 1-step rollouts; this is the sharpest test the paper leaves open.
  • Editorial: The dose-response pattern in the paper's synthetic-to-real ratio sweeps suggests a testable prediction: other Dyna-style methods will show monotone performance collapse as the fraction of imagined data in each batch increases in DMC.
  • Editorial: The results push toward reporting per-environment results rather than averaged returns, since averaged curves can hide environments where an algorithm never leaves random-policy level.
  • Editorial: A broader implication is that sample-efficiency gains in model-based RL may be less a property of the algorithm than of the benchmark's reward, termination, and integration structure; the paper documents these differences but does not isolate a cause.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical study of Dyna-style model-based reinforcement learning (DMBRL), centered on MBPO and ALM, across OpenAI Gym and DeepMind Control Suite (DMC) tasks with proprioceptive observations. The main observation is that MBPO and ALM, which perform well in Gym, largely fail to improve over their model-free base algorithms in DMC, despite similar task structure and the same physics engine. The authors then investigate possible causes—predictive model error, critic divergence, and plasticity loss—and show that standard remedies (better model utilization, layer norm, periodic resets) do not restore competitive performance. They also introduce a JAX-based MBPO implementation with reported speedups and release code. The paper concludes that synthetic rollouts, the backbone of Dyna-style methods, can degrade performance and that the community's benchmark reliance has inflated confidence in these methods.

Significance. If the core claims hold, the paper is a useful and timely cautionary result: it documents a concrete generalization failure of two influential DMBRL algorithms, reproduces across 15 DMC tasks with 6 seeds, includes full training curves in the appendix, and provides an oracle-model experiment that separates model error from other failure sources. The released JAX implementation is a practical contribution that lowers the computational barrier for follow-up work. However, the strength of the central attribution to 'MBPO' and the broad 'Dyna-style methods' conclusion depends on an equivalence claim between the implemented 1-step variant and the original n-step MBPO, which is only established on Gym final returns. That gap, plus a conclusion that overstates the body's own results, currently tempers the significance of the headline message.

major comments (3)
  1. [Section 3.1] The paper's core attribution to MBPO rests on the statement, 'Combining the results in Figure 2 and Figure 3 we can conclude that our JAX-based implementation of 1-step MBPO matches the original implementation's n-step MBPO's performance... and as such we will henceforth refer to 1-step MBPO as simply MBPO.' This equivalence is established only on six OpenAI Gym final returns. Since the DMC results in Sections 3.2 through 5 are the load-bearing evidence for the paper's central claim, the 1-step to n-step bridge needs to be validated in DMC, where the paper itself reports model errors above 100% on hopper tasks and above 25% on other failure tasks. A hardcoded rollout length of 1 removes exactly the adaptive horizon mechanism that n-step MBPO uses to balance model bias and compounding error. As written, the paper characterizes 'MBPO' as known in the literature, but the evidence supports a claim about a 1-step MBPO configuration. Please either run the original n-step MBPO on the DMC tasks or explicitly re-scope the claims to the 1-step variant and adjust the title, abstract, and conclusion accordingly.
  2. [Section 6] The conclusion states that 'for all 15 DMC tasks we examined, adding model-generated samples consistently undermined both sample efficiency and wall-clock performance relative to simpler, model-free off-policy algorithms with identical hyperparameters.' This is inconsistent with the body. Section 3.2 says MBPO fails to improve the policy in 'six out of fifteen' environments, and the full results in Figure 11 and Figure 17 explicitly include scenarios where MBPO 'matches SAC's sample efficiency' or solves tasks more slowly. 'Consistently undermined' is therefore too strong for the presented evidence. The conclusion should be reworded to reflect the actual distribution: a subset of tasks shows complete failure, a larger set shows underperformance, and a few show matching or slower improvement. This is not merely a wording issue because the abstract and conclusion are what readers will cite.
  3. [Section 4.2] The perfect-model experiment is well designed as a diagnostic, but it is a perfect-model version of the 1-step algorithm, not of the original n-step MBPO. With a perfect model, longer rollouts are no longer penalized by compounding error, so the natural comparison would include an n-step perfect-model condition. The current experiment therefore cannot rule out the possibility that the original n-step MBPO, even with a learned model, would behave differently in DMC. Please either report an n-step perfect-model condition or explicitly state that this experiment only tests the 1-step variant.
minor comments (4)
  1. [Abstract and Section 3.2] The abstract says the gap 'significantly degrades performance across most DMC environments,' while Section 3.2 says MBPO fails in 'six out of fifteen' environments. The paper should harmonize these counts and use a consistent classification of failure, underperformance, and matching behavior.
  2. [Section 4.1] The percent model error formula is presented without an equation number; numbering it would make the later references in Sections 4.2 and Appendix G easier to follow.
  3. [Figure 10 caption] The caption says 'the hopper-stand and humanoid-stand environment' (singular); it should say 'environments.' In addition, the two panels (a) and (b) should be labeled clearly so readers know which panel corresponds to which environment.
  4. [References] The reference to Voelcker et al. appears with an extra space ('V oelcker') in several places; please fix the formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical ablation, not a derivation; the 1-step/n-step attribution gap is an external-validity concern, not a self-referential reduction.

full rationale

This paper makes no formal derivation whose output is equivalent to its input by construction. The central comparison is a controlled ablation: MBPO versus its own 'no Dyna' base (SAC) with identical hyperparameters, so the difference isolates the synthetic-rollout component. The perfect-model experiment in Section 4.2 is an independent benchmark that does not reuse fitted values from the learned-model experiments. No parameter is fitted to a subset of data and then reported as a prediction of a closely related quantity. The cited prior work (Janner et al., 2019; Ghugare et al., 2022; Voelcker et al., 2024) is external to the present authors and is used as evidence or as a baseline, not as a self-citation chain that forces the conclusion. The one substantive validity concern is the paper's Section 3.1 decision to equate its 1-step MBPO implementation with the original n-step MBPO based on Gym results and 'henceforth refer to 1-step MBPO as simply MBPO.' That is an attribution or generalization question: if n-step rollouts behave differently in DMC, the conclusions may not transfer to MBPO as published. But this is not circularity under the enumerated patterns, because the equivalence is asserted from empirical evidence, not defined into existence, and the DMC results are generated by the 1-step algorithm itself rather than being derived from that equivalence. The claim that Dyna-style enhancements degrade performance is therefore self-contained as an experimental finding about the tested configurations; its scope, not its logical structure, is what is contested.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fit; all hyperparameters come from original implementations or from sweeps used only to test robustness. The central claim relies on two domain assumptions, benchmark similarity and seed sufficiency, and one ad hoc assumption: the 1-step MBPO variant can stand in for the original n-step MBPO in DMC based on Gym-only validation. No invented entities are introduced.

assumptions (3)
  • domain assumption OpenAI Gym and DMC are sufficiently comparable that a robust algorithm should transfer.
    The 'surprising gap' framing depends on the similarity of tasks and the shared MuJoCo backend. Appendix C lists differences in reward normalization, termination conditions, physical parameters, and integration schemes.
  • ad hoc to paper The 1-step MBPO variant is equivalent to the original n-step MBPO for DMC conclusions.
    Section 3.1 validates the 1-step variant against n-step MBPO only in Gym, then uses 1-step MBPO for all DMC experiments and refers to it as 'MBPO'.
  • domain assumption Six random seeds are sufficient to establish relative sample efficiency and final return comparisons.
    All main comparisons use 6 seeds with mean and standard deviation; no bootstrap or aggregate statistical method such as rliable is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning." pith.science (2026). https://pith.science/paper/EVN5LVHA

@misc{pith2026241214312,
  author       = {Pith},
  title        = {Pith review of: Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVN5LVHA}},
  note         = {Machine review of arXiv:2412.14312}
}
read the original abstract

Dyna-style off-policy model-based reinforcement learning (DMBRL) algorithms are a family of techniques for generating synthetic state transition data and thereby enhancing the sample efficiency of off-policy RL algorithms. This paper identifies and investigates a surprising performance gap observed when applying DMBRL algorithms across different benchmark environments with proprioceptive observations. We show that, while DMBRL algorithms perform well in OpenAI Gym, their performance can drop significantly in DeepMind Control Suite (DMC), even though these settings offer similar tasks and identical physics backends. Modern techniques designed to address several key issues that arise in these settings do not provide a consistent improvement across all environments, and overall our results show that adding synthetic rollouts to the training process -- the backbone of Dyna-style algorithms -- significantly degrades performance across most DMC environments. Our findings contribute to a deeper understanding of several fundamental challenges in model-based RL and show that, like many optimization fields, there is no free lunch when evaluating performance across diverse benchmarks in RL.

Figures

Figures reproduced from arXiv: 2412.14312 by the authors.

Figure 1
Figure 1. Comparison of normalized final return for two different DMBRL algorithms with and without Dyna-style enhancements. Results are averaged across 6 random seeds per task, with 6 tasks from OpenAI Gym (Brockman et al., 2016) and 15 from DMC (Tassa et al., 2020). For training curves, cf. Figures 3 and 17. Beyond robustness, these discrepancies raise critical ques￾tions for us in the RL community. First, is the perfor￾man… view at source ↗
Figure 2
Figure 2. Comparison of seconds per environment step across mul￾tiple DMBRL implementations when deployed on 6 OpenAI Gym environments. Compared to Pytorch MBPO and ALM, our imple￾mentation takes ∼40× and ∼4× less time, respectively. In summary, our main contributions are: 1. Demonstrating that DMBRL methods can suffer from a significant performance gap when training from scratch in OpenAI Gym versus DMC environments. 2. Anal… view at source ↗
Figure 3
Figure 3. 1-step MBPO consistently achieves higher final episodic returns and demonstrates faster convergence compared to SAC. Likewise, 1-step MBPO either matches or surpasses ALM in terms of sample efficiency across all but one task. Solid curves cor￾respond to the mean and shaded regions represent the standard deviation across six trials. The dotted line represents the episodic return of the original n-step MBPO at the end… view at source ↗
Figures from the paper (16 more)
Figure 5
Figure 5. Figure 5: Performance of ALM with and without Dyna-style en￾hancements on six DMC benchmarks. Like MBPO, ALM shows no policy improvement with Dyna-style enhancements, but per￾forms strongly without them (6 trials, ± std). ALM is built on top of DDPG, which allows us to test Dyna…
Figure 6
Figure 6. Figure 6: Comparing MBPO with a perfect predictive model to SAC and MBPO with a learned model. Solid curves correspond to the mean and shaded regions represent the standard deviation across six trials. the default hyperparameters and access to perfect rollouts can improve upon i…
Figure 7
Figure 7. Figure 7: Mean Q-values (6 seeds) with shaded regions showing min-max range. MBPO’s critic either massively underestimates or predicts no return when compared to SAC. Overestimation is magnified in quadruped tasks with Dyna-style enhancements. Results across all 15 DMC tasks are…
Figure 8
Figure 8. Figure 8: , SAC with periodic resets significantly outperforms MBPO with the same resets on most tasks. This highlights 5Training curves for model resets every 2 × 104 environment steps may be found in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Hyperparameter sweep for MBPO trained on the hopper-stand environment across 93 hyperparameter configurations (3 trials each, ± std). As none of these configurations yielded meaningful or consistent performance, we omit per-parameter plots for brevity. Despite one seed…
Figure 10
Figure 10. Figure 10: Hyperparameter sweep for MBPO trained on the hopper-stand and humanoid-stand environment across 81 hyperpa￾rameter configurations each (1 trials each). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Full performance of MBPO and SAC for 15 challenging DMC benchmark tasks as referenced in Section 3.2. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Illustration of the change in percent model error during RL training for OpenAI Gym, as measured on the training distribution. (6 trials, ± std) [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Illustration of the change in percent model error during RL training for DMC, as measured on the training distribution. (6 trials, ± std). (a) (b) [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Illustration of the relationship between final episodic return and synthetic-to-real ratio used for MBPO training for (a) hopper-stand for 6 seeds and 100k steps each, and (b) humanoid-stand for 3 seeds and 400k steps each. These results show that more synthetic data …
Figure 15
Figure 15. Figure 15: Comparing full performance of MBPO, SAC, and MBPO with actor and critic resets for 15 challenging DMC benchmark tasks as referenced in Section 5. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Mean Q-values for 15 challenging DMC benchmark tasks as referenced in Section 4.3. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Comparing performance of MBPO, SAC, ALM, and ALM without Dyna for 15 challenging DMC benchmark tasks. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Mean Q-values across six seeds each for 6 challenging DMC benchmark tasks. Comparisons are of MBPO and SAC to MBPO with a perfect predictive model as discussed in Section 4.3. Shaded regions correspond to minimum and maximum Q values across trials and solid lines are …
Figure 19
Figure 19. Figure 19: Mean Q-values for 6 challenging DMC benchmark tasks across six seeds each. Comparisons are of MBPO and SAC to MBPO with layer norm applied as discussed in Section 4.3. Shaded regions correspond to minimum and maximum Q values across trials and solid lines are the mean…
Figure 20
Figure 20. Figure 20: Comparing performance of MBPO, SAC, MBPO with layer norm applied to the critic, and MBPO with periodic model resets for 6 challenging DMC benchmark tasks as discussed in Section 5. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34: 0 29304--29320, 2021

    work page 2021

  3. [3]

    Ahmed, N., Wahed, M., and Thompson, N. C. The growing influence of industry in ai research. Science, 379 0 (6635): 0 884--886, 2023. doi:10.1126/science.ade2420. URL https://www.science.org/doi/abs/10.1126/science.ade2420

    doi:10.1126/science.ade2420 2023

  4. [4]

    Amos, B., Stanton, S., Yarats, D., and Wilson, A. G. On the model-based stochastic value gradient for continuous reinforcement learning. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 6--20. PMLR, 2021

    work page 2021

  5. [5]

    and Schaal, S

    Atkeson, C. and Schaal, S. Learning tasks from a single demonstration. In Proceedings of International Conference on Robotics and Automation, volume 2, pp.\ 1706--1712 vol.2, 1997. doi:10.1109/ROBOT.1997.614389

    work page arXiv 1997

  6. [6]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016 a . URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  7. [7]

    J., Smith, L., Kostrikov, I., and Levine, S

    Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In Proceedings of the International Conference on Machine Learning, pages 1577--1594. PMLR, 2023

    work page 2023

  8. [8]

    M., Gebru, T., McMillan-Major, A., and Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp.\ 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi:10.1145/3442188.344592...

    arXiv 2021

Show all 57 references
  1. [9]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  2. [10]

    Openai gym, 2016

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016. URL https://arxiv.org/abs/1606.01540

    2016 arXiv

  3. [11]

    Sample-efficient reinforcement learning with stochastic ensemble value expansion

    Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in Neural Information Processing Systems, 31, 2018

    2018

  4. [12]

    G., and Silver, D

    Dabney, W., Barreto, A., Rowland, M., Dadashi, R., Quan, J., Bellemare, M. G., and Silver, D. The value-improvement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, number 8, pages 716...

    2021

  5. [13]

    Deisenroth, M. P. and Rasmussen, C. E. Pilco: a model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp.\ 465–472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195

    2011

  6. [14]

    Dyna-style model-based reinforcement learning with model-free policy optimization

    Dong, K., Luo, Y., Wang, Y., Liu, Y., Qu, C., Zhang, Q., Cheng, E., Sun, Z., and Song, B. Dyna-style model-based reinforcement learning with model-free policy optimization. Knowledge-Based Systems, 287: 0 111428, 2024. ISSN 0950-7051. doi:https://doi.org/10.1016/j.knosys.2024....

    2024

  7. [15]

    G., and Courville, A

    D’Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Proceedings of the Eleventh International Conference on Learning Representations, 2023

    2023

  8. [16]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning, pages 1587--1596. PMLR, 2018

    2018

  9. [17]

    Simplifying model-based rl: learning representations, latent-space models, and policies with one objective

    Ghugare, R., Bharadhwaj, H., Eysenbach, B., Levine, S., and Salakhutdinov, R. Simplifying model-based rl: learning representations, latent-space models, and policies with one objective. In International Conference on Learning Representations, 2022

    2022

  10. [18]

    Continuous deep q-learning with model-based acceleration

    Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Continuous deep q-learning with model-based acceleration. In Proceedings of the International Conference on Machine Learning, pages 2829--2838. PMLR, 2016

    2016

  11. [19]

    Soft actor-critic algorithms and applications, 2019 a

    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications, 2019 a . URL https://arxiv.org/abs/1812.05905

    2019 arXiv

  12. [20]

    Dream to control: learning behaviors by latent imagination

    Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: learning behaviors by latent imagination. In Proceedings of the International Conference on Learning Representations, 2020

    2020

  13. [21]

    Mastering diverse control tasks through world models

    Hafner, D., Pasukonis, J., Ba, J., et al. Mastering diverse control tasks through world models. Nature, 640: 0 647--653, 2025. doi:10.1038/s41586-025-08744-2

    2025 doi

  14. [22]

    A., Hosny, A., Khodakarami, F., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C

    Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Waldron, L., Wang, B., McIntosh, C., Goldenberg, A., Kundaje, A., Greene, C. S., et al. Transparency and reproducibility in artificial intelligence. Nature, 586 0 (7829): 0 E14--E16, 2020. doi:10.1038/s41586-020-2766-y

    2020 doi

  15. [23]

    A., Su, H., and Wang, X

    Hansen, N. A., Su, H., and Wang, X. Temporal difference learning for model predictive control. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedin...

    2022

  16. [24]

    Z., Talvitie, E

    Holland, G. Z., Talvitie, E. J., and Bowling, M. The effect of planning shape on dyna-style planning in high-dimensional state spaces, 2019. URL https://arxiv.org/abs/1806.01825

    2019 arXiv

  17. [25]

    A., Gilitschenski, I., Farahmand, A.-m., and Eaton, E

    Hussing, M., Voelcker, C. A., Gilitschenski, I., Farahmand, A.-m., and Eaton, E. Dissecting deep RL with high update ratios: Combatting value divergence. In Proceedings of the Reinforcement Learning Conference, 2024

    2024

  18. [26]

    Mbpo: Model-based policy optimization, 2019

    Janner, M. Mbpo: Model-based policy optimization, 2019. URL https://github.com/jannerm/mbpo. Accessed: 2024-12-11

    2019

  19. [27]

    When to trust your model: Model-based policy optimization

    Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp.\ 1122--1133, Red Hook, NY, USA, 2019. Curran Associates Inc

    2019

  20. [28]

    M., White, A., da Silva, B

    Jordan, S. M., White, A., da Silva, B. C., White, M., and Thomas, P. S. Position: Benchmarking is limited in reinforcement learning research, 2024. URL https://arxiv.org/abs/2406.16241

    2024 arXiv

  21. [29]

    and Boedecker, J

    Kalweit, G. and Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Levine, S., Vanhoucke, V., and Goldberg, K. (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp...

    2017

  22. [30]

    Bidirectional model-based policy optimization

    Lai, H., Shen, J., Zhang, W., and Yu, Y. Bidirectional model-based policy optimization. In Proceedings of the International Conference on Machine Learning, pages 5618--5627. PMLR, 2020

    2020

  23. [31]

    On effective scheduling ofmModel-based reinforcement Learning

    Lai, H., Shen, J., Zhang, W., Huang, Y., Zhang, X., Tang, R., Yu, Y., and Li, Z. On effective scheduling ofmModel-based reinforcement Learning. Advances in Neural Information Processing Systems, 34: 0 3694--3705, 2021

    2021

  24. [32]

    Reinforcement learning with augmented data

    Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33: 0 19884--19895, 2020

    2020

  25. [33]

    When to trust your data: Enhancing dyna-style model-based reinforcement learning with data filter, 2024

    Li, Y., Dong, Z., Luo, E., Wu, Y., Wu, S., and Han, S. When to trust your data: Enhancing dyna-style model-based reinforcement learning with data filter, 2024. URL https://arxiv.org/abs/2410.12160

    2024 arXiv

  26. [34]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning, 2019. URL https://arxiv.org/abs/1509.02971

    2019 arXiv

  27. [35]

    [Re] When to trust your model: model-based policy optimization

    Liu, Y., Xu, J., and Pan, Y. [Re] When to trust your model: model-based policy optimization. ReScience C, 6 0 (2), 2020. URL https://openreview.net/forum?id=rkezvT9f6r. Accepted at NeurIPS 2019 Reproducibility Challenge

    2020

  28. [36]

    A survey on model-based reinforcement learning

    Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., and Yu, Y. A survey on model-based reinforcement learning. Science China Information Sciences, 67(2):121101, 2024

    2024

  29. [37]

    Understanding and preventing capacity loss in reinforcement learning

    Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2022

    2022

  30. [38]

    Overestimation, overfitting, and plasticity in actor-critic: the Bitter lesson of Reinforcement learning

    Nauman, M., Bortkiewicz, M., Miłoś, P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the Bitter lesson of Reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning , pp.\ 37342-...

    2024

  31. [39]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 16828--16847. PMLR, 2022

    2022

  32. [40]

    Pytorch: An imperative style, high-performance deep learning library, 2019

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An impera...

    2019 arXiv

  33. [41]

    Mind the model, not the agent: the primacy bias in model-based RL

    Qiao, Z., Lyu, J., and Li, X. Mind the model, not the agent: the primacy bias in model-based RL . In Proceedings of the European Conference on Artificial Intelligence, 2023

    2023

  34. [42]

    EPOpt : learning robust neural network policies using model ensembles

    Rajeswaran, A., Ghotra, S., Levine, S., and Ravindran, B. EPOpt : learning robust neural network policies using model ensembles. In Proceedings of the International Conference on Learning Representations, 2017

    2017

  35. [43]

    High-dimensional continuous control using generalized advantage estimation, 2015

    Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation, 2015. URL https://arxiv.org/abs/1506.02438

    2015 arXiv

  36. [44]

    Learning off-policy with online planning

    Sikchi, H., Zhou, W., and Held, D. Learning off-policy with online planning. In Proceedings of the Conference on Robot Learning, pages 1622--1633. PMLR, 2022

    2022

  37. [45]

    CURL : contrastive unsupervised representations for reinforcement learning

    Laskin, M., Srinivas, A., and Abbeel, P. CURL : contrastive unsupervised representations for reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 5639--5650. PMLR, 2020

    2020

  38. [46]

    Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2 0 (4): 0 160–163, July 1991. ISSN 0163-5719. doi:10.1145/122344.122377. URL https://doi.org/10.1145/122344.122377

    1991

  39. [47]

    Model regularization for stable sample rollouts

    Talvitie, E. Model regularization for stable sample rollouts. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI'14, pp.\ 780–789, Arlington, Virginia, USA, 2014. AUAI Press. ISBN 9780974903910

    2014

  40. [48]

    dm\_control: Software and Tasks for Continuous Control

    Tassa, Y., Tunyasuvunakool, S., Muldal, A., Doron, Y., Trochim, P., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., and Heess, N. dm\_control: Software and Tasks for Continuous Control . Software Impacts, 6: 0 100022, November 2020. ISSN 26659638. doi:10.1016/j.simpa.2...

    2020

  41. [49]

    and Schwartz, A

    Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of 4th Connectionist Models Summer School. Erlbaum Associates, June 1993

    1993

  42. [50]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012. URL https://api.semanticscholar.org/CorpusID:5230692

    2012

  43. [51]

    A., Hussing, M., and Eaton, E

    Voelcker, C. A., Hussing, M., and Eaton, E. Can we hop in general? A discussion of benchmark selection and design using the Hopper environment. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024

    2024

  44. [52]

    Model-based policy optimization under approximate Bayesian inference

    Wang, C., Chen, Y., and Murphy, K. Model-based policy optimization under approximate Bayesian inference. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 238. PMLR, 2024

    2024

  45. [53]

    Benchmarking model-based reinforcement learning, 2019

    Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J. Benchmarking model-based reinforcement learning, 2019. URL https://arxiv.org/abs/1907.02057

    2019 arXiv

  46. [54]

    Live in the moment: learning dynamics model adapted to evolving policy

    Wang, X., Wongkamjan, W., Jia, R., and Huang, F. Live in the moment: learning dynamics model adapted to evolving policy. In Proceedings of the International Conference on Machine Learning, pages 36470--36493. PMLR, 2023

    2023

  47. [55]

    Accelerated policy learning with parallel differentiable simulation

    Xu, J., Makoviychuk, V., Narang, Y., Ramos, F., Matusik, W., Garg, A., and Macklin, M. Accelerated policy learning with parallel differentiable simulation. In Proceedings of the International Conference on Learning Representations, 2022

    2022

  48. [56]

    Mastering visual continuous control: improved data-augmented reinforcement learning

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: improved data-augmented reinforcement learning. In Proceedings of the Tenth International Conference on Learning Representations, 2022

    2022

  49. [57]

    Is model ensemble necessary? Model-based RL via a single model with Lipschitz regularized value function

    Zheng, R., Wang, X., Xu, H., and Huang, F. Is model ensemble necessary? Model-based RL via a single model with Lipschitz regularized value function. In Proceedings of the International Conference on Learning Representations, 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.