Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Model-based Lookahead Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Model-free and model-based reinforcement learning can be combined so that an agent reaches model-free performance while using the small amount of data typical of model-based methods.

desk verdict A useful hybrid MPC/MFRL method with strong internal ablations, but the headline claim about matching MFRL performance with MBRL data-efficiency is not supported by the chosen baselines. read the letter →

arxiv 1908.06012 v1 pith:CQ6O6IBW submitted 2019-08-15 cs.LG cs.AIstat.ML

Zhang-Wei Hong , Joni Pajarinen , Jan Peters This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords model-basedreinforcementlearningmodelpredictivecontrolmodel-freeTRPOsampleefficiencycontinuousvaluefunctionMuJoCo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that model-free and model-based reinforcement learning can be combined so that an agent reaches the final performance of model-free methods while using the small amount of data typical of model-based methods. The proposed approach, MPC-MFRL, trains a policy, a value function, and a forward dynamics model on the same interactions, then at evaluation time uses the policy to sample short simulated rollouts, the value function to score their endpoints, and a soft-greedy average to choose the action. On four MuJoCo continuous-control benchmarks the method is reported to outperform pure TRPO, random-sampling MPC, and CEM-based MPC, with the largest gains on the harder tasks. The paper's ablations identify why each component matters: policy-collected data improves the dynamics model, policy-guided sampling improves planning, the value function prevents short-horizon myopia, and averaging over the top action sequences guards against model error.

What carries the argument

The load-bearing object is the model predictive control loop with three MFRL components substituted into the classic three stages. The policy replaces the uniform action distribution in trajectory sampling, so Eq.~5 produces high-value rollouts; the value function replaces the zero terminal reward, so Eq.~6 scores a short-horizon rollout by its immediate simulated rewards plus $V_{\theta_V}$ at the endpoint; and soft-greedy action selection averages the top $E$ action sequences rather than taking the single best one, which damps the max-operator bias caused by forward-model error. The value-function terminal reward is the piece that lets the planner use a short horizon $H$ without being shortsighted, and the policy sampling is the piece that keeps those short rollouts near states the value function can judge.

What would settle it

On a continuous-control task, record the states visited during MPC-MFRL's simulated rollouts, execute the same state-action sequences in the real environment, and compare the learned value function's prediction with the actual Monte Carlo return from those states. If the terminal value is systematically over-optimistic by more than the reward gain the planner obtains, or if MPC-MFRL fails to beat the TRPO and CEM-based MPC baselines at matched sample counts on a new task, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that the long-standing trade-off between model-free final performance and model-based sample efficiency can be broken by placing a model-free policy and value function inside a model predictive control loop. During training, the exploratory policy $\pi_{\theta_\pi}$ collects the transitions used to update the policy, the value function $V_{\theta_V}$, and the forward dynamics model $f_{\theta_f}$ from the same data. During evaluation, MPC-MFRL draws actions in simulation from the policy (Eq.~5), scores each simulated trajectory by the sum of task rewards plus the learned value function as terminal reward (Eq.~6), and executes the first action of the average of the top $E$ action sequences under soft-greedy selection (Eq.~7). The paper argues that the policy keeps simulated states within the value function's training distribution, that the value function lets a short planning horizon avoid compounding model errors while still looking beyond the horizon, and that soft-greedy averaging removes the bias of greedy selection under an approximate model. Section~4.3 reports that MPC-MFRL achieves better performance than all baselines on Swimmer, Reacher, HalfCheetah, and Ant, matching the model-free final performance at model-based data usage.

Load-bearing premise

The load-bearing premise is that the value function, trained only on real states, remains accurate on the simulated states reached when the learned dynamics model is rolled out under the policy's action distribution; if those states drift out of its training distribution, the terminal reward in Eq. 6 becomes over-optimistic and planning degrades.

Editorial extensions

If this is right

  • If the claim holds, a robot or other real-world system can get model-free-level task performance while collecting only a small fraction of the environment interactions that pure model-free RL needs.
  • The result implies that model-based planners should use the policy and value function as priors, not only the learned dynamics model, and this does not require a perfect model or restricted state and action spaces.
  • Because policy-collected data trains a more accurate forward dynamics model, the exploration policy is a component of model quality, not a separate concern from model learning.
  • Soft-greedy action selection offers a simple, uncertainty-free way to make MPC robust to approximate dynamics models, complementing methods that rely on probabilistic ensembles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is replacing TRPO with other model-free algorithms: if the benefit comes from policy-guided sampling and value-based evaluation, then DDPG-style or PPO-style policies should preserve or amplify the gains when trained on the same data.
  • The overestimation risk the paper observes in Section 4.6 suggests a concrete repair: subtract an uncertainty or ensemble-disagreement penalty from the terminal value before ranking trajectories; the paper proposes no correction beyond soft-greedy averaging.
  • Since planning scores rollouts with the task reward $R$, the method still needs a known reward function at evaluation time; extending it to settings where the reward must also be learned or inferred would make it applicable beyond simulated benchmarks.
  • The framework is described for deterministic dynamics, so a testable extension is whether probabilistic dynamics models in the same loop would let the value-function terminal reward rescue planning under stochastic transitions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MPC-MFRL, a framework that combines model-free reinforcement learning with model predictive control. During training, an MFRL policy collects environment interactions, and these same data are used to train the policy, the value function, and a deterministic forward dynamics model. During evaluation, MPC is performed by sampling simulated trajectories with the MFRL policy, scoring them with the reward plus a value-function terminal term (Eq. 6), and selecting actions via a soft-greedy average over the best action sequences (Eq. 7). Experiments on Swimmer, Reacher, HalfCheetah, and Ant compare against TRPO variants and two MPC baselines. The authors claim that the approach achieves MFRL-level final performance with MBRL-level data efficiency, and they provide ablations in Sections 4.4-4.7 to support individual design choices.

Significance. If the headline claim were fully supported, the paper would make a useful contribution: it offers a simple and modular way to combine MFRL policies, value functions, and MPC, and its ablations are informative. The demonstrations that policy-collected data improve dynamics-model accuracy (Section 4.4), that policy-guided sampling helps planning (Section 4.5), and that soft-greedy selection helps under model error (Section 4.7) are valuable and internally plausible. However, the central claim as stated in the abstract and conclusion overreaches the evidence. The baseline set includes only TRPO as an MFRL method and no state-of-the-art MBRL method such as PETS, so the experiments primarily show feasibility and internal consistency rather than closing the gap between state-of-the-art MFRL and MBRL on standard continuous control benchmarks.

major comments (3)
  1. [Section 4.1, Section 4.3, Abstract] The abstract and conclusion claim that MPC-MFRL 'can achieve MFRL's level of performance while being as data-efficient as MBRL,' but the experimental design does not support this claim as stated. The MFRL baselines MF(S) and MF(D) are both TRPO, and the MBRL baselines MPC-Random and MPC-CEM use deterministic forward dynamics models with random or on-policy exploration. No comparison is made with stronger MFRL algorithms such as SAC, PPO, or DDPG, and no comparison is made with a probabilistic-ensemble MBRL method such as PETS, which was the standard high-sample-efficiency MBRL baseline on these MuJoCo tasks at the time. Consequently, the Section 4.3 statement that MPC-MFRL 'achieves better performance than all baseline methods' holds only for the narrow baseline set considered. To support the central claim, the authors should add competitive baselines or substantially soften the claim to match the evidence.
  2. [Section 3.2, Eq. (6), Section 4.6] The trajectory evaluation in Eq. (6) relies on a value function trained on real experience to score states visited during simulated rollouts under the learned dynamics model and the MFRL policy. The paper itself acknowledges in Section 4.6 that uniform action sampling can lead to states where the value function overestimates, and that longer planning horizons suffer from compounding model error, but it provides no quantitative diagnosis or mitigation beyond soft-greedy action averaging. Since the value-function terminal reward is a central component of the proposed approach, the authors should analyze the distribution shift between real and simulated states (for example, by measuring value prediction error on simulated rollouts) and either correct the overestimation or state clearly the conditions under which the benefit of the value-function term disappears.
  3. [Section 4.2, Fig. 2] The evaluation reports the best-so-far average return over 5 seeds with bootstrapped confidence intervals, but no statistical tests are performed. Statements such as the improvement being 'particularly significant' in Ant and HalfCheetah are not backed by significance testing, and with only 5 seeds, overlapping intervals in the figures make the superiority claims fragile. The authors should report per-environment tables with means, standard errors, and appropriate significance tests, or explicitly restrict their claims to descriptive comparisons.
minor comments (5)
  1. [Section 4.1] The text says implementation details are 'in supplementary material,' but no supplementary material is included in the manuscript; please provide full hyperparameters, network sizes, optimizer settings, and training schedules, and consider releasing code to enable independent verification.
  2. [Section 4.6] There is a wording error: 'the terminal reward (i.e., V(st)) in simulated trajectories of MPC-MFRL (Z=U) are similar than MPC-MFRL (Z=pi)' should read 'is similar to'; also 'overstimation' should be 'overestimation'.
  3. [Algorithm 1] In the trajectory sampling loop, the state update uses a_n_h while the sampled action is denoted \hat{a}^n_h; this notation is inconsistent and should be aligned.
  4. [Eq. (7)] The soft-greedy equation writes '\bar{a}^{1:N}_{1:H} = argsort ...' which conflates indices with sorted action sequences; please clarify the notation so that the averaging over the E best action sequences is unambiguous.
  5. [References] Reference [1] lists 'Bertsekas, D. P.' four times; please clean up the author list for this entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MPC-MFRL's claims are empirical comparisons; no fitted input is renamed as a prediction.

full rationale

The paper makes no formal derivation whose conclusion is built into its inputs. Its central claim, that combining MFRL and MPC reaches MFRL-level performance with MBRL-level data efficiency, is an empirical comparison evaluated on MuJoCo benchmarks. The value function and policy used in planning (Eqs. 5 and 6) are trained on real experience, and the dynamics model is trained on the same dataset; this is a standard training-evaluation loop rather than a definitional reduction. The paper even acknowledges the related risk of value-function overestimation on simulated states in Section 4.6. There is no fitted parameter that is subsequently renamed as a prediction, and no load-bearing self-citation chain: the cited works are external baselines and prior methods, not the authors' own uniqueness theorems or ansatze. The baseline selection is narrow (TRPO for MFRL, no PETS-style probabilistic ensemble for MBRL), but this is an evidence gap or experimental weakness, not circularity. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The method components (policy, value function, dynamics model, soft-greedy selection) are combinations of existing constructs. The primary burden is not invented entities but algorithm hyperparameters that are not reported.

free parameters (4)
  • planning horizon H = not reported (ablation uses 2, 5, 20)
    Chosen by hand; main experiments do not report the value used in Fig. 2.
  • number of simulated trajectories N = not reported
    Chosen by hand; affects planning quality and computation.
  • soft-greedy top-E action sequences = not reported
    Chosen by hand; controls the trade-off between optimism and averaging.
  • dynamics model hidden units = not reported (ablation uses varying counts)
    Chosen by hand; affects model capacity and performance.
assumptions (4)
  • domain assumption The environment is a deterministic Markov decision process with a known reward function R(s,a).
    Stated in Section 2.1; the paper restricts to deterministic cases for simplicity.
  • domain assumption The learned forward dynamics model approximates the true dynamics well enough over the planning horizon H.
    Required for planning; the paper acknowledges approximation errors but does not bound them.
  • domain assumption The value function is accurate on states visited during policy-guided simulations.
    Section 3.2 assumes the policy prevents sampling unreachable states, so the terminal reward in Eq. 6 is reliable. This is the main fragility.
  • standard math Standard policy gradient training works: TRPO optimizes the policy and the value function is learned by regression.
    Standard algorithmic assumptions for policy gradient methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model-based Lookahead Reinforcement Learning." pith.science (2026). https://pith.science/paper/CQ6O6IBW

@misc{pith2026190806012,
  author       = {Pith},
  title        = {Pith review of: Model-based Lookahead Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQ6O6IBW}},
  note         = {Machine review of arXiv:1908.06012}
}
read the original abstract

Model-based Reinforcement Learning (MBRL) allows data-efficient learning which is required in real world applications such as robotics. However, despite the impressive data-efficiency, MBRL does not achieve the final performance of state-of-the-art Model-free Reinforcement Learning (MFRL) methods. We leverage the strengths of both realms and propose an approach that obtains high performance with a small amount of data. In particular, we combine MFRL and Model Predictive Control (MPC). While MFRL's strength in exploration allows us to train a better forward dynamics model for MPC, MPC improves the performance of the MFRL policy by sampling-based planning. The experimental results in standard continuous control benchmarks show that our approach can achieve MFRL`s level of performance while being as data-efficient as MBRL.

Figures

Figures reproduced from arXiv: 1908.06012 by the authors.

Figure 1
Figure 1. Overview of MPC-MFRL at evaluation time: In state st, MPC-MFRL samples trajectories using an MFRL policy, evaluates sampled trajectories by an MFRL value function, and then chooses an action at based on Eq. 4. The environment transitions to st+1 and the process starts from the beginning. The upper row illustrates planning in simulation. The lower row depicts interaction with the real environment. policy π 0 (if usin… view at source ↗
Figure 2
Figure 2. Mean and bootstrapped confidence interval (solid lines and error bars, over 5 distinct random seeds) of "Average return" (see Section 4.2 for evaluation details and definition of "Average return") for different methods. "Num. timestep (M)" is the number of millions of interactions with the environment. Our method MPC-MFRL outperforms comparison methods. For comparison method and evaluation details see Section 4.1 [… view at source ↗
Figure 3
Figure 3. (a) We measure "Average testing error" of a forward dynamics model using a pre-collected testing dataset. Policy indicates collecting data using an MFRL policy, while Random+MPC represents uniform random exploration with on-policy data aggregation [23]; (b) The evaluation results of MPC-MFRL with different training schemes: MPC-MFRL (Policy) is the original MPC-MFRL, while MPC-MFRL (Random+MPC) trains the forward dy… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: a shows MPC-MFRL (Z = π) outperforms MPC-MFRL (Z = U) in all tasks and therefore suggests that the MFRL control policy πθ π can more readily sample trajectories of high value in the real environment than the baselines. Furthermore, MPC-MFRL (Z = π) still surpasses MPC-…
Figure 5
Figure 5. Figure 5: (a) The evaluation results of different action selection approaches: MPC-MFRL (w / SG) indicates the original MPC-MFRL, while MPC-MFRL (w/o SG) represents MPC-MFRL withoug soft-greedy action selection. The rest of legends are identical to [PITH_FULL_IMAGE:figures/full…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 42 canonical work pages

  1. [1]

    P., Bertsekas, D

    Bertsekas, D. P., Bertsekas, D. P., Bertsekas, D. P., and Bertsekas, D. P.Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA, 2005

    work page 2005

  2. [2]

    Openai gym, 2016

    Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016. 9

    work page 2016

  3. [3]

    Sample-efficient reinforcement learning with stochastic ensemble value expansion

    Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 8234–8244, 2018

    work page 2018

  4. [4]

    Burnham, K. P. and Anderson, D. R. Model selection and multimodel inference: a practical information-theoretic approach. Springer Science & Business Media, 2003

    work page 2003

  5. [5]

    Path integral guided policy search

    Chebotar, Y ., Kalakrishnan, M., Yahya, A., Li, A., Schaal, S., and Levine, S. Path integral guided policy search. In Proc. Int. Conf. Robotics and Automation (ICRA) , pp. 3381–3388, 2017

    work page 2017

  6. [6]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 4759–4770, 2018

    work page 2018

  7. [7]

    Model-based reinforcement learning via meta-policy optimization

    Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y ., Asfour, T., and Abbeel, P. Model-based reinforcement learning via meta-policy optimization. arXiv preprint arXiv:1809.05214, 2018

    arXiv 2018

  8. [8]

    and Rasmussen, C

    Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In Proc. Int. Conf. Machine Learning (ICML), pp. 465–472, 2011

    work page 2011

Show all 45 references
  1. [9]

    S., Landau, S., Leese, M., and Stahl, D

    Everitt, B. S., Landau, S., Leese, M., and Stahl, D. Miscellaneous clustering methods. Cluster Analysis, pp. 215–255, 2011

    2011

  2. [10]

    I., Gonzalez, J

    Feinberg, V ., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., and Levine, S. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101, 2018

    2018 arXiv

  3. [11]

    E., Prett, D

    Garcia, C. E., Prett, D. M., and Morari, M. Model predictive control: theory and practice—a survey. Automatica, 25(3):335–348, 1989

    1989

  4. [12]

    Continuous deep q-learning with model-based acceleration

    Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Continuous deep q-learning with model-based acceleration. In Proc. Int. Conf. Machine Learning (ICML), pp. 2829–2838, 2016

    2016

  5. [13]

    and Boedecker, J

    Kalweit, G. and Boedecker, J. Uncertainty-driven imagination for continuous deep reinforcement learning. In Proc. Conf. Robot Learning (CoRL), pp. 195–206, 2017

    2017

  6. [14]

    and Deisenroth, M

    Kamthe, S. and Deisenroth, M. Data-efficient reinforcement learning with probabilistic model predictive control. In Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS), volume 84, pp. 1701–1710, 2018

    2018

  7. [15]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Representations (ICRL), 2015

    2015

  8. [16]

    Model-ensemble trust-region policy optimization

    Kurutach, T., Clavera, I., Duan, Y ., Tamar, A., and Abbeel, P. Model-ensemble trust-region policy optimization. In Proc. Int. Conf. Learning Representations (ICRL), 2018

    2018

  9. [17]

    and Abbeel, P

    Levine, S. and Abbeel, P. Learning neural network policies with guided policy search under unknown dynamics. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 1071–1079, 2014

    2014

  10. [18]

    and Koltun, V

    Levine, S. and Koltun, V . Guided policy search. InProc. Int. Conf. Machine Learning (ICML), pp. 1–9, 2013

    2013

  11. [19]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wier- stra, D. Continuous control with deep reinforcement learning. In Proc. Int. Conf. Learning Representations (ICRL), 2016

    2016

  12. [20]

    Plan online, learn offline: Efficient learning and exploration via model-based control

    Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, I. Plan online, learn offline: Efficient learning and exploration via model-based control. In Proc. Int. Conf. Learning Representations (ICRL), 2019

    2019

  13. [21]

    Human-level control through deep reinforcement learning

    Mnih, V .et al.. Human-level control through deep reinforcement learning. Nature, vol. 518, no. 7540, pp. 529-533, February 2015. 10

    2015

  14. [22]

    Asynchronous methods for deep reinforcement learning

    Mnih, V .et al.. Asynchronous methods for deep reinforcement learning. In Proc. Int. Conf. Machine Learning (ICML), pp. 1928–1937, 2016

    1928

  15. [23]

    S., and Levine, S

    Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In Proc. Int. Conf. Robotics and Automation (ICRA), 2018

    2018

  16. [24]

    Value prediction network

    Oh, J., Singh, S., and Lee, H. Value prediction network. In Advances in Neural Information Processing Systems, pp. 6118–6128, 2017

    2017

  17. [25]

    Temporal difference models: Model-free deep RL for model-based control

    Pong*, V ., Gu*, S., Dalal, M., and Levine, S. Temporal difference models: Model-free deep RL for model-based control. In Proc. Int. Conf. Learning Representations (ICRL), 2018

    2018

  18. [26]

    Imagination-augmented agents for deep reinforcement learning

    Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Jimenez Rezende, D., Puig- domènech Badia, A., Vinyals, O., Heess, N., Li, Y ., Pascanu, R., Battaglia, P., Hassabis, D., Silver, D., and Wierstra, D. Imagination-augmented agents for deep reinforcement learning. I...

    2017

  19. [27]

    Richards, A. G. Robust constrained model predictive control . PhD thesis, Massachusetts Institute of Technology, 2005

    2005

  20. [28]

    The cross-entropy method for combinatorial and continuous optimization

    Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127–190, 1999

    1999

  21. [29]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Proc. Int. Conf. Machine Learning (ICML), pp. 1889–1897, 2015

    2015

  22. [30]

    High-dimensional continuous control using generalized advantage estimation

    Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In Proc. Int. Conf. Learning Representations (ICRL), 2016

    2016

  23. [31]

    Proximal policy optimiza- tion algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  24. [32]

    S., and Müller, M

    Silver, D., Sutton, R. S., and Müller, M. Sample-based learning and search with permanent and transient memories. In Proc. Int. Conf. Machine Learning (ICML), pp. 968–975, 2008

    2008

  25. [33]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016

    2016

  26. [34]

    Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approx- imating dynamic programming. In Machine Learning Proceedings, pp. 216–224. Elsevier, 1990

    1990

  27. [35]

    Sutton, R. S. and Barto, A. G. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998

    1998

  28. [36]

    S., McAllester, D

    Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y . Policy gradient methods for reinforcement learning with function approximation. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 1057–1063, 2000

    2000

  29. [37]

    S., Szepesvári, C., Geramifard, A., and Bowling, M

    Sutton, R. S., Szepesvári, C., Geramifard, A., and Bowling, M. Dyna-style planning with linear function approximation and prioritized sweeping. In Proc. Conf. Uncertainty in Artificial Intelligence (UAI), pp. 528–536, 2008

    2008

  30. [38]

    Value iteration networks

    Tamar, A., Wu, Y ., Thomas, G., Levine, S., and Abbeel, P. Value iteration networks. InProc. Advances in Neural Information Processing Systems (NIPS), pp. 2154–2162, 2016

    2016

  31. [39]

    Synthesis and stabilization of complex behaviors through online trajectory optimization

    Tassa, Y ., Erez, T., and Todorov, E. Synthesis and stabilization of complex behaviors through online trajectory optimization. In Proc. Int. Conf. Robots and Systems (IROS), pp. 4906–4913, 2012

    2012

  32. [40]

    Control-limited differential dynamic programming

    Tassa, Y ., Mansard, N., and Todorov, E. Control-limited differential dynamic programming. In Proc. Int. Conf. Robotics and Automation (ICRA), pp. 1168–1175, 2014. 11

    2014

  33. [41]

    and Li, W

    Todorov, E. and Li, W. A generalized iterative lqg method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proc. American Control Conf., pp. 300–306, 2005

    2005

  34. [42]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. InProc. Int. Conf. Robots and Systems (IROS), pp. 5026–5033, 2012

    2012

  35. [43]

    M., Boots, B., and Theodorou, E

    Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. Information theoretic mpc for model-based reinforcement learning. In Proc. Int. Conf. Robotics and Automation (ICRA), 2017

    2017

  36. [44]

    Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992

    1992

  37. [45]

    Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search

    Zhang, T., Kahn, G., Levine, S., and Abbeel, P. Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search. In Proc. Int. Conf. Robotics and Automation (ICRA), pp. 528–535, 2016. 12

    2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.