Pith. sign in

REVIEW 2 major objections 1 minor 6 cited by

Exploring Model-based Planning with Policy Networks

T0 review · 2 major / 1 minor · reviewed 2026-05-25 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimizing planning over policy networks inside a dynamics model yields state-of-the-art sample efficiency on MuJoCo tasks.

desk verdict POPLIN's direct optimization over policy parameters during planning is a clear technical step, but the 3x sample-efficiency claim rests on an unverified assumption about model accuracy over the horizon. read the letter →

arxiv 1906.08649 v1 pith:NYJHYOBH submitted 2019-06-20 cs.LG cs.AIcs.ROstat.ML

Tingwu Wang , Jimmy Ba This is my paper · ORCID
classification cs.LGcs.AIcs.ROstat.ML
keywords model-basedreinforcementlearningpolicynetworksonlineplanningsampleefficiencyMuJoCocontinuouscontrolmodelpredictive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes POPLIN, a model-based reinforcement learning method that formulates planning as optimization over a policy network rather than random sampling in action space. At each step the algorithm either optimizes action sequences initialized from the policy or optimizes the policy parameters directly, all inside a learned dynamics model. This produces policies that require roughly three times fewer environment samples than prior methods such as PETS, TD3 and SAC while reaching higher final performance. The authors further observe that the optimization surface is smoother when working in parameter space than in raw action space. In some environments the resulting policy network can be used at test time without continued model-predictive control.

What carries the argument

Policy network used to initialize or directly parameterize the optimization of actions inside the learned dynamics model at every time step.

What would settle it

An experiment that measures model prediction error over the planning horizon and shows that the reported performance gains disappear once that error exceeds a modest threshold while all other algorithmic choices remain fixed.

Watch

Extended reading notes

Core claim

Formulating each planning step as an optimization problem over a policy network—either by refining action sequences that the network proposes or by directly adjusting the network parameters—inside the learned dynamics model produces action sequences that transfer to the real environment more effectively than random search in action space.

Load-bearing premise

The learned dynamics model must stay accurate enough over the chosen planning horizon for the optimized actions or parameters to produce useful behavior when executed in the real environment.

Editorial extensions

If this is right

  • Planning becomes more efficient in high-dimensional continuous action spaces because the policy network supplies a structured starting point or parameterization.
  • The smoother optimization landscape in parameter space reduces the number of samples needed to reach high-performing policies.
  • For some locomotion tasks the distilled policy can be deployed directly without repeated online planning at test time.
  • The same planning procedure can be applied on top of any differentiable dynamics model that supports gradient-based optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Policy networks may act as a useful regularizer that keeps planned trajectories within regions where the model is more reliable.
  • The approach could be combined with ensemble or uncertainty-aware dynamics models to further extend the reliable planning horizon.
  • Similar parameter-space planning might improve efficiency in other sequential decision problems where an approximate model exists but exhaustive search is intractable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes POPLIN, a model-based RL method that formulates online planning as optimization over action sequences initialized from a policy network or directly over policy parameters. It reports state-of-the-art results on MuJoCo locomotion tasks, claiming approximately 3x greater sample efficiency than PETS, TD3, and SAC; attributes gains to a smoother optimization landscape in parameter space; shows that a distilled policy can sometimes be deployed without MPC at test time; and releases code.

Significance. If the empirical claims hold after verification of model fidelity, the work would usefully demonstrate that parameter-space planning can outperform pure action-space search in MBRL while retaining the sample-efficiency advantages of model-based methods. The open-source code is a clear strength that enables direct reproduction and extension.

major comments (2)
  1. [Abstract] Abstract: the central claim that POPLIN is 'about 3x more sample efficient' than PETS, TD3, and SAC is load-bearing for the contribution yet is presented without reported multi-step dynamics-model error, held-out trajectory prediction accuracy, planning-horizon length, or statistical significance tests on the performance differences.
  2. [Abstract] Abstract / experiments: the transfer assumption that optimizing inside the learned model produces actions that succeed in the real environment is invoked without any reported planned-vs-executed discrepancy or compounding-error diagnostics on the MuJoCo tasks; this directly affects whether the 3x efficiency gain can be attributed to the planning component.
minor comments (1)
  1. [Abstract] Abstract: 'Further more' should be 'Furthermore'.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We agree that additional empirical details will strengthen the presentation of our results and will revise the manuscript accordingly. Our point-by-point responses follow.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that POPLIN is 'about 3x more sample efficient' than PETS, TD3, and SAC is load-bearing for the contribution yet is presented without reported multi-step dynamics-model error, held-out trajectory prediction accuracy, planning-horizon length, or statistical significance tests on the performance differences.

    Authors: We agree these details should be reported. The planning horizon length is 10 steps for POPLIN, PETS, and the model-free baselines (Section 4.1). We will add multi-step model prediction error and held-out trajectory accuracy metrics in the revised version. For statistical significance, the learning curves already aggregate 5 seeds with standard-deviation shading; we will add explicit discussion of the performance gaps in the text and caption. revision: yes

  2. Referee: [Abstract] Abstract / experiments: the transfer assumption that optimizing inside the learned model produces actions that succeed in the real environment is invoked without any reported planned-vs-executed discrepancy or compounding-error diagnostics on the MuJoCo tasks; this directly affects whether the 3x efficiency gain can be attributed to the planning component.

    Authors: All reported returns are obtained by executing the first planned action in the true MuJoCo environment at every step (standard MPC procedure). The sample-efficiency comparison therefore already reflects real-environment performance. To address the request for explicit diagnostics, we will add planned-versus-executed trajectory discrepancy plots over the horizon in the appendix of the revision. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; empirical algorithm with benchmark results

full rationale

The paper introduces the POPLIN algorithm combining policy networks with model-based planning and reports empirical results on MuJoCo environments showing improved sample efficiency over baselines. No derivation chain, first-principles prediction, or uniqueness theorem is claimed. Performance claims rest on external benchmark comparisons rather than any quantity fitted inside the paper and then renamed as a prediction. No self-definitional steps, fitted-input predictions, or load-bearing self-citations appear in the provided text. The central result is therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The method inherits standard RL assumptions (Markov decision process, differentiable dynamics model) but introduces no new free parameters or invented entities visible in the abstract; the central empirical claim rests on the unstated premise that the learned model is accurate enough for the reported planning horizon.

assumptions (1)
  • domain assumption The environment can be modeled by a differentiable dynamics function that remains valid over the multi-step planning horizon.
    Implicit in any model-based planning claim; required for the optimization to produce transferable actions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Model-based Planning with Policy Networks." pith.science (2026). https://pith.science/paper/NYJHYOBH

@misc{pith2026190608649,
  author       = {Pith},
  title        = {Pith review of: Exploring Model-based Planning with Policy Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYJHYOBH}},
  note         = {Machine review of arXiv:1906.08649}
}
read the original abstract

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in https://github.com/WilsonWangTHU/POPLIN.

Figures

Figures reproduced from arXiv: 1906.08649 by the authors.

Figure 1
Figure 1. We transform each planned candidate action trajectory with PCA into a 2D blue scatter. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance curves of POPLIN-P, POPLIN-A and other state-of-the-art algorithms on [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The MPC control and policy control performance of the proposed POPLIN-A, and POPLIN [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: The performance of PETS, POPLIN-A, POPLIN-P using different population size of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The reward optimization surface in the solution space. The expected reward is higher from [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: The performance of POPLIN-A, POPLIN-P-BC, POPLIN-P-Avg, POPLIN-P-GAN using [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 6
Figure 6. Figure 6: The action distribu￾tion in a episode visualized in the projected 2D PCA space. 5.4 Ablation Study In this section, we study how sensitive our algorithms are with respect to some of the crucial hyper-parameters, for example, the initial variance of the CEM noise distri…
Figure 8
Figure 8. Figure 8: Full Performance of POPLIN-P, POPLIN-A and other state-of-the-art algorithms on 12 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The planning performance and the testing performance of the proposed POPLIN-A, and [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The performance of POPLIN-A, POPLIN-P-BC, POPLIN-P-Avg, POPLIN-P-GAN using [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The performance of POPLIN-A, POPLIN-P, and PETS of different random seeds. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: The performance of PETS, POPLIN-A, POPLIN-P-Avg, POPLIN-P-BC and POPLIN [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Reward surface in solution space (action space) for PETS algorithm. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Reward surface in solution space (action space) for POPLIN-A-Replan. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Reward surface in solution space (action space) for POPLIN-A-Init. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Reward surface in solution space (parameter space) for POPLIN-P with 0 hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Reward surface in solution space (parameter space) for POPLIN-P using 1 hidden layer. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: The color indicates the expected cost (negative of expected reward). We emphasis that all [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: The figures are the planned trajectories of PETS. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: The figures are the planned trajectories of POPLIN-P using 1 hidden layer MLP. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: The figures are the planned trajectories of POPLIN-P using 0 hidden layer MLP. [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mastering Atari with Discrete World Models

    cs.LG 2020-10 accept novelty 7.0 of 10

    DreamerV2 reaches human-level performance on 55 Atari games by learning behaviors inside a separately trained discrete-latent world model.

  2. MODIP: Efficient Model-Based Optimization for Diffusion Policies

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    MODIP fine-tunes diffusion policies offline-to-online by training a world model, running MPC with terminal state values inside it to create targets, and using policy-independent TD critics, yielding gains over BC on D...

  3. RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms

    cs.LG 2025-09 conditional novelty 6.0 of 10

    RT-HCP combines a physics-informed model, multi-step planning, and an actor-critic policy to learn a swing-up controller on a real Furuta pendulum under strict time and sample limits.

  4. Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning

    cs.RO 2025-02 conditional novelty 6.0 of 10

    SGFT uses a simulation-trained value function to guide real-world exploration via potential-based reward shaping and short-horizon objectives, substantially improving fine-tuning sample efficiency.

  5. First Order Model-Based RL through Decoupled Backpropagation

    cs.RO 2025-08 conditional novelty 5.0 of 10

    By computing gradients through a learned dynamics model while unrolling trajectories in the real simulator, DMO achieves SHAC-level sample efficiency with standard simulators and deploys on a real quadruped.

  6. Bounding Distributional Shifts in World Modeling through Novelty Detection

    cs.RO 2025-08 conditional novelty 4.0 of 10

    Attaching a VAE novelty detector to the DINO-WM world model and penalizing out-of-distribution predicted states in CEM planning lowers Chamfer distance on small-data robot manipulation benchmarks.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · cited by 6 Pith papers

  1. [1]

    The cross-entropy method for optimization

    Zdravko I Botev, Dirk P Kroese, Reuven Y Rubinstein, and Pierre L’Ecuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35–59. Elsevier, 2013

    work page 2013

  2. [2]

    OpenAI Gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    work page Pith review arXiv 2016

  3. [3]

    Sample- efficient reinforcement learning with stochastic ensemble value expansion

    Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pages 8224–8234, 2018

    work page 2018

  4. [4]

    Combining model-based and model-free updates for trajectory-centric reinforcement learning

    Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-based and model-free updates for trajectory-centric reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-V olume 70, pages 703–711. JMLR. org, 2017

    work page 2017

  5. [5]

    Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

    Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv:1805.12114, 2018

    work page Pith review arXiv 2018

  6. [6]

    A tutorial on the cross-entropy method

    Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19–67, 2005

    work page 2005

  7. [7]

    Pilco: A model-based and data-efficient approach to policy search

    Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465–472, 2011

    work page 2011

  8. [8]

    Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

    Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018

    work page Pith review arXiv 2018

Show all 43 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018

    2018 arXiv

  2. [10]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014

    2014

  3. [11]

    Continuous deep q-learning with model-based acceleration

    Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning , pages 2829–2838, 2016

    2016

  4. [12]

    Recurrent world models facilitate policy evolution

    David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems , pages 2450–2462, 2018

    2018

  5. [13]

    World models

    David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  6. [14]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018

    2018 arXiv

  7. [15]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018

    2018 arXiv

  8. [16]

    Emergence of locomotion behaviours in rich environments

    Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017

    2017 arXiv

  9. [17]

    Learning continuous control policies by stochastic value gradients

    Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pages 2944–2952, 2015

    2015

  10. [18]

    Model- based reinforcement learning for atari

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019

    1903

  11. [19]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 9

    2013 arXiv

  12. [20]

    Model-ensemble trust-region policy optimization

    Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018

    2018 arXiv

  13. [21]

    Learning neural network policies with guided policy search under unknown dynamics

    Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems , pages 1071–1079, 2014

    2014

  14. [22]

    End-to-end training of deep visuomotor policies

    Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334–1373, 2016

    2016

  15. [23]

    Guided policy search

    Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning, pages 1–9, 2013

    2013

  16. [24]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems , pages 6389–6399, 2018

    2018

  17. [25]

    Iterative linear quadratic regulator design for nonlinear biological movement systems

    Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (1), pages 222–229, 2004

    2004

  18. [26]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  19. [27]

    Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees

    Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. ICLR, 2019

    2019

  20. [28]

    Playing atari with deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  21. [29]

    Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning

    Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. arXiv preprint arXiv:1708.02596, 2017

    2017 arXiv

  22. [30]

    The loss surface of deep and wide neural networks

    Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning-V olume 70 , pages 2603–2612. JMLR. org, 2017

    2017

  23. [31]

    Robust constrained model predictive control

    Arthur George Richards. Robust constrained model predictive control . PhD thesis, Mas- sachusetts Institute of Technology, 2005

    2005

  24. [32]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems , pages 2234–2242, 2016

    2016

  25. [33]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1889–1897, 2015

    2015

  26. [34]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  27. [35]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

    2016

  28. [36]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 2017

    2017

  29. [37]

    Exponentially vanishing sub-optimal local minima in multilayer neural networks

    Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks. arXiv preprint arXiv:1702.05777, 2017

    2017 arXiv

  30. [38]

    Integrated architectures for learning, planning, and reacting based on approximating dynamic programming

    Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pages 216–224. Elsevier, 1990

    1990

  31. [39]

    Dyna, an integrated architecture for learning, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160–163, 1991. 10

    1991

  32. [40]

    Synthesis and stabilization of complex behaviors through online trajectory optimization

    Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. InIntelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 4906–4913. IEEE, 2012

    2012

  33. [41]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , pages 5026–5033. IEEE, 2012

    2012

  34. [42]

    A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems

    Emanuel Todorov and Weiwei Li. A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300–306. IEEE, 2005

    2005

  35. [43]

    Cheetah", and the HalfCHeetah from OpenAI Gym is named as

    Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J Johnson, and Sergey Levine. Solar: Deep structured latent representations for model-based reinforcement learning. arXiv preprint arXiv:1808.09105, 2018. 11 A Appendix A.1 Algorithm Diagrams To better illustrate...

    2018 arXiv

Pith tools

Reviewed May 25, 2026 · model on record in the stance chip above.