Pith. sign in

REVIEW 3 major objections 6 minor 66 references

By unrolling trajectories in an ordinary simulator and backpropagating through a learned model, DMO obtains policy gradients nearly as accurate as a differentiable simulator's.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

By computing gradients through a learned dynamics model while unrolling trajectories in the real simulator, DMO achieves SHAC-level sample efficiency with standard simulators and deploys on a real quadruped.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A clean, well-executed revival of the SVG(infinity) decoupling idea, with a fair ablation and a real-robot demo; the central mechanism holds up, but the strongest performance claims need a bit more care. the 3 major comments →

arxiv 2509.00215 v2 pith:7UNGNVKS submitted 2025-08-29 cs.RO cs.AIcs.LG

First Order Model-Based RL through Decoupled Backpropagation

Joseph Amigo , Rooholla Khorrambakht , Elliot Chane-Sane , Nicolas Mansard , Ludovic Righetti This is my paper
classification cs.RO cs.AIcs.LG
keywords Model-Based Reinforcement LearningFirst-Order Policy OptimizationDecoupled BackpropagationLearned Dynamics ModelsSample EfficiencyQuadruped LocomotionSim-to-Real Transfer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that reinforcement learning can get the sample efficiency of differentiable-simulator optimizers without ever needing simulator gradients. The proposed method, DMO, unrolls policy trajectories in an ordinary high-fidelity simulator, then computes policy gradients by backpropagating through a learned neural-network model of the simulator's dynamics, evaluated at the true simulator states rather than at the model's own predicted states. That separation is meant to prevent the compounding prediction errors that hurt standard model-based RL. The authors report that DMO converges with roughly tenfold fewer samples than PPO, matches the efficiency of SHAC-style differentiable-simulator training, and transfers to a real quadruped robot in both four-legged and two-legged walking.

Core claim

On its own terms, the paper's finding is that the accuracy of first-order policy gradients is set by where the dynamics-model derivatives are evaluated, not by which function generates the trajectory. By replacing only the forward unrolling with a high-fidelity simulator and leaving the backward pass inside a learned differentiable model, DMO produces gradients whose cosine similarity to exact differentiable-simulator gradients is high, and it avoids the ill behavior seen in methods that unroll fully inside the learned model. The same decoupling also lets the value critic be fit to true simulator rollouts, which the paper argues is more accurate. Three instantiations, DMO-BPTT, DMO-SHAC, and

What carries the argument

The central object is the decoupled gradient swap: during training the simulator produces the real next state, while a learned MLP dynamics model produces a predicted next state from the same state-action pair; an autodiff gradient-swapping function returns the real state for the forward pass but routes backpropagation through the predicted state, so gradients flow through the learned model's Jacobians evaluated at the accurate simulator state. This uses the learned model's Jacobian at the true state as a proxy for the simulator's Jacobian. A truncated-return objective with a learned value bootstrap for the SHAC and SAPO variants, or without one for the BPTT variant, closes the loop.

Load-bearing premise

The learned dynamics model's derivatives have to be faithful enough to the real simulator's derivatives along the states the policy actually visits; otherwise the decoupled gradient is a wrong gradient computed on a correct trajectory.

What would settle it

On a contact-rich task with an exact differentiable simulator available, compute DMO's policy gradient and the simulator's true policy gradient on identical rollouts, then deliberately corrupt the learned model's Jacobians, for example by dropping contact transitions from the replay buffer. If DMO still trains successfully despite low cosine similarity between the two gradients, the paper's accuracy-of-gradients mechanism is not what drives the gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • DMO can be dropped onto existing first-order gradient methods such as SHAC, SAPO, and BPTT with only a few lines of code, because decoupling is implemented as a gradient-swapping autodiff node.
  • It reaches asymptotic performance in under four million samples, about ten times fewer than PPO, while improving wall-clock time by up to 20% despite learning a model.
  • It works with non-differentiable simulators: the real-robot experiments were trained in a GPU simulator that provides no gradients.
  • Decoupling itself, not just model learning, drives the gains: the model-based-forward ablation nearly halves asymptotic performance.
  • The critic can be learned from true simulator rollouts, which the paper claims is more accurate, and the method remains stable even with a batch size of one on dense-reward tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling claim generalizes, the field's investment in differentiable simulators may be partly redundant: any simulator plus a locally accurate learned Jacobian could give the same first-order updates, at least where rewards are differentiable.
  • The same trick should extend to richer world models such as latent or vision-based ones, since the forward unrolling stays in the simulator; the paper's MLP limitation concerns the model class, not the decoupling idea itself.
  • A testable consequence is that DMO's policy-gradient quality should track the Jacobian accuracy of the learned model along the on-policy state distribution; measuring that correlation would directly check the mechanism.
  • The requirement of differentiable rewards is a design constraint rather than a detail: applying DMO to sparse-reward tasks would force the value bootstrap to carry more of the learning, so reward shaping becomes a first-class concern.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DMO (Decoupled forward-backward Model-based policy Optimization), a first-order model-based RL method that unrolls trajectories with a high-fidelity (possibly non-differentiable) simulator while computing policy gradients by backpropagating through a learned differentiable dynamics model. The learned model is trained on replay data with a one-step maximum-likelihood objective, and its Jacobians are evaluated at the true simulator states during the backward pass. The authors instantiate DMO on top of BPTT, SHAC, and SAPO, evaluate it on five DFlex control tasks plus an AllegroHand task, and deploy policies on a real Unitree Go2 in both quadrupedal and bipedal modes. They report large sample-efficiency gains over PPO and SAC, competitive final performance with SHAC/SAPO, and a wall-clock advantage over model-free baselines. The core decoupling idea is supported by an ablation against a model-rollout variant and by a cosine-similarity analysis of gradient directions.

Significance. If the central claim is correct, DMO offers a practical way to obtain cheap first-order policy gradients without a differentiable simulator, avoiding the compounding-error problem of standard FoG-MBRL while retaining the sample-efficiency benefits of analytical gradients. The decoupled backpropagation formula in Appendix A.1.3 is correct and is implemented with a simple PyTorch trick; the paper honestly credits prior work (SVG(∞), PILCO) for the general idea and provides a systematic modern evaluation. The internal ablation against model-based forward rollouts is well designed and shows a real performance difference, which strengthens the paper's main mechanistic claim. The real-robot deployment is a valuable addition, though it is presented as a demonstration rather than a controlled study. The main weakness is that the paper does not directly establish that the learned model's Jacobians are accurate enough along the visited state-action distribution; this is the key assumption on which the 'as accurate as a differentiable simulator' claim rests.

major comments (3)
  1. [Section 3.2 and Eq. (4)] The central claim that DMO yields policy gradients 'as accurate as those of a differentiable simulator' depends on the learned model's Jacobians ∂f̂/∂s and ∂f̂/∂a being close to the true simulator Jacobians along the states actually visited by the policy. The training objective in Eq. (4) is a one-step maximum-likelihood fit of the conditional next-state distribution; it controls the conditional mean, but gives no guarantee on the derivative of the conditional mean. In contact-rich tasks, the true transition is non-smooth while the MLP is smooth, so ∂f̂ can be substantially biased exactly where the policy gradient changes. The only direct evidence, Figure 5 (right), measures cosine similarity of full policy gradients, not Jacobian error, and the comparison against the model-forward trajectory uses different forward states, so it does not isolate Jacobian fidelity at the same (s,a). Pleas
  2. [Algorithm 1 and Section 3.2] There is an indexing error in the description of the backward pass. Algorithm 1 and the text in Section 3.2 state that DMO uses ∂f̂(s,a)/∂s evaluated at (s_{t+1}, a_{t+1}) and ∂f̂(s,a)/∂a at (s_{t+1}, a_{t+1}) to approximate the true dynamics Jacobians. However, the analytical policy gradient in Eq. (3) requires ∂f/∂s and ∂f/∂a evaluated at (s_t, a_t). Appendix A.1.3 correctly writes the decoupled formula with ∂f̂(s,a)/∂s at (s_t, a_t). As written, Algorithm 1 would backpropagate through the wrong time step. Please correct the subscripts in Algorithm 1 and in the Section 3.2 discussion to (s_t, a_t), and make the implementation consistent with Appendix A.1.3.
  3. [Figure 5 (right) and Section 4.3] The gradient-similarity experiment is not sufficient to support the claim that decoupled gradients are 'more precise' than model-rollout gradients. The three trajectories are unrolled 'in parallel' and 'under the same conditions,' but it is not stated whether the actions and initial states are identical across the three graphs. If the model-rollout trajectory diverges in state space, then the cosine similarity between DFlex and model-forward gradients conflates state-distribution mismatch with Jacobian error. Even for the DMO graph, the cosine similarity between DFlex and DMO gradients is an aggregate over states and does not show where or why the gradients differ. Please clarify the experimental protocol and report a per-transition Jacobian comparison at identical (s,a) points, or at least a state-conditioned breakdown. This is necessary to attribute the observed performance gains to th
minor comments (6)
  1. [Eq. (4)] The objective written as L_f(ϕ) = E_{(s,a,s')∼B}[ p_ϕ(s' | s,a) ] is not a maximum-likelihood objective as stated; MLE would maximize the log-density, E[ log p_ϕ(s'|s,a) ]. Please correct the equation or clarify the notation.
  2. [Section 3.1, Eq. (3)] In the third line of Eq. (3), the derivative ∂πθ(s)/∂θ is written with a subscript (θ̃, s_t), where θ̃ is not defined. This appears to be a typo; please replace with the proper evaluation notation.
  3. [Section 4.2] The claim that DMO 'achieves up to 20% improvement' in wall-clock time is not tied to a specific baseline in the main text. Figure 4 (right) compares only PPO and SAC, not MAAC or SHAC. Please state the comparison explicitly.
  4. [Appendix A.1.3] The class name 'GradientSwapingFunction' contains a typo ('Swaping' -> 'Swapping'). Also, the code comment and explanation could be clearer about the fact that the backward path through f̂ receives the gradient with respect to the true next state by using a clone node.
  5. [Section 4.3] The phrase 'generated under the exact same conditions as DMO-SHAC' is ambiguous: it should specify whether the same random seed, initial state, and action sequence are used for the learned-model rollouts. This matters for interpreting Figure 5 (right).
  6. [Appendix A.2.1, Figure 6] SHAC/SAPO results are only shown as final performance; no learning curves or wall-clock times are given for these baselines. The paper's claim of being competitive with SHAC in 'efficiency and final convergence accuracy' would be strengthened by reporting SHAC's sample/time curves on the shared benchmarks.

Circularity Check

0 steps flagged

No significant circularity: DMO's decoupled gradient is a surrogate approximation with external empirical checks, not a fitted quantity being re-predicted.

full rationale

The paper's derivation is self-contained in the relevant sense. The learned model f_hat is fitted to replay transitions via maximum likelihood (Eq. 4), and the policy gradient uses f_hat's Jacobians evaluated on simulator states (Algorithm 1, Appendix A.1.3). This is a surrogate-gradient approximation, not a definitional reduction: the fitted object is the conditional next-state distribution, whereas the claimed output is a policy-gradient vector. The cosine-similarity experiment (Figure 5, right) and the model-based-forward ablation (Figure 3, right) compare DMO against the differentiable-simulator gradient and against the full-model-rollout variant, respectively, providing external empirical checks that are not built into the training objective. The known decoupling idea is explicitly credited to prior work (SVG(∞), PILCO, [52]) rather than presented as a new uniqueness result; the paper's stated contribution is the modern empirical study and integration with FoG methods, which does not depend on a self-citation chain. The Limitations section honestly flags the MLP model's incapability with complex inputs, which is an assumption about Jacobian fidelity, not circular reasoning. There is no fitted constant later renamed as a prediction, no self-citation used to forbid alternatives, and no equation that reduces to its input by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical entity or fitted constant for a theoretical derivation; its free parameters are ordinary hyperparameters and task reward weights. The key added assumption is that a learned one-step model can supply reliable Jacobians along true trajectories.

free parameters (3)
  • Rollout horizon H = 16
    Hand-chosen truncation length for the actor loss (Equations 5-7). It trades bias against variance; the paper notes dense rewards make H=16 near optimal for Go2 (Section 4.2). The result likely depends on this choice.
  • Entropy temperature alpha (DMO-SAPO) = 1.0 initial, adapted with lr 5e-3
    Trade-off between exploration and exploitation in the SAPO variant (Equation 6). Environment-specific hyperparameter, not derived.
  • Go2 reward weights (k... values) = e.g., k_xy_vel=0.5, k_orient=5.0, k_fht=30.0
    Hand-tuned shaping weights for the real-robot tasks (Appendix A.3). The method needs differentiable rewards, so these weights define the learning signal; the real-robot results depend on them.
axioms (4)
  • domain assumption The simulator f is an accurate model of the real robot's dynamics for sim-to-real transfer.
    Assumed in the real-robot setup (Section 4.1); if the simulator-reality gap is large, the learned Jacobians will be misleading on the real robot.
  • domain assumption The learned model f_hat provides sufficiently accurate Jacobians of f along the training distribution.
    This is the load-bearing premise of the decoupled gradient: the paper's own limitation section notes the MLP is ill-suited for images/point clouds, and Section 4.3 measures the gradient approximation quality via cosine similarity.
  • domain assumption Rewards are differentiable with respect to state and action.
    Explicitly stated as a limitation: 'it requires differentiable reward functions... reward functions often need to be redesigned' (Limitations).
  • standard math The reparameterization trick produces valid unbiased gradients for stochastic Gaussian policies and dynamics models.
    Used in Appendix A.1.4 to allow gradient flow through sampling; standard result.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of First Order Model-Based RL through Decoupled Backpropagation." pith.science (2026). https://pith.science/paper/7UNGNVKS

@misc{pith2026250900215,
  author       = {Pith},
  title        = {Pith review of: First Order Model-Based RL through Decoupled Backpropagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UNGNVKS}},
  note         = {Machine review of arXiv:2509.00215}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

There is growing interest in reinforcement learning (RL) methods that leverage the simulator's derivatives to improve learning efficiency. While early gradient-based approaches have demonstrated superior performance compared to derivative-free methods, accessing simulator gradients is often impractical due to their implementation cost or unavailability. Model-based RL (MBRL) can approximate these gradients via learned dynamics models, but the solver efficiency suffers from compounding prediction errors during training rollouts, which can degrade policy performance. We propose an approach that decouples trajectory generation from gradient computation: trajectories are unrolled using a simulator, while gradients are computed via backpropagation through a learned differentiable model of the simulator. This hybrid design enables efficient and consistent first-order policy optimization, even when simulator gradients are unavailable, as well as learning a critic from simulation rollouts, which is more accurate. Our method achieves the sample efficiency and speed of specialized optimizers such as SHAC, while maintaining the generality of standard approaches like PPO and avoiding ill behaviors observed in other first-order MBRL methods. We empirically validate our algorithm on benchmark control tasks and demonstrate its effectiveness on a real Go2 quadruped robot, across both quadrupedal and bipedal locomotion tasks.

Figures

Figures reproduced from arXiv: 2509.00215 by Elliot Chane-Sane, Joseph Amigo, Ludovic Righetti, Nicolas Mansard, Rooholla Khorrambakht.

Figure 1
Figure 1. Figure 1: Go2 Walking on four and two legs using policies optimized with DMO. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualizations of Environments Trained with DMO. Each image represents a distinct simulation environment: Ant, SNU Humanoid, Cheetah, Hopper, Allegro Hand, and Humanoid. Real Robot Experiment Setup for Quadrupedal Motion We trained a velocity-commanded walking policy for the Unitree Go2 quadruped robots using DMO and successfully deployed it on the real robot. For the simulation, we used the GPU-accelerate… view at source ↗
Figure 3
Figure 3. Figure 3: Left: Sample Efficiency at 4M Samples. Results for DMO, PPO, SAC, and MAAC, all limited to 4M samples. Right: Sample Efficiency with Model-Based Ablation. Comparison of DMO to its counterpart that uses learned model forward passes, both at 4M samples. Aggregate normalized scores with mean and 95% confidence intervals over all environments and 5 seeds are shown [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: Sample Efficiency at Extended Training. Results for DMO and MAAC using 4M samples, while PPO is trained for 160M samples and SAC for 40M samples. Right: Wall-Clock Time Efficiency at Extended Training. Results for the same sample allocations as on the left. Aggregate normalized scores with mean and 95% confidence intervals across all environments and 5 seeds are shown. also compared with the main bas… view at source ↗
Figure 5
Figure 5. Figure 5: Left: Sample Efficiency of DMO-BPTT Across Batch Sizes for the Go2 Quadrupedal Task. This illustrates the sample efficiency of DMO-BPTT for various batch sizes, with mean and 95% confidence intervals calculated over 5 seeds. Notably, sample efficiency increases as batch size decreases to 1, and DMO-BPTT remains stable even in this configuration. Right: Cosine Simi￾larity of Gradient Computations. This show… view at source ↗
Figure 6
Figure 6. Figure 6: Episodic Return Performance Across Environments. This figure shows the episodic return (mean ± std) at 4M samples (8M for Go2BipedalEnv). DMO represents the best-performing DMO version for each environment. Although SHAC utilizes true model derivatives and does not rely on learned dynamics, DMO achieves competitive performance. SHAC results are unavailable for Go2Env and Go2BipedalEnv due to IsaacGym not b… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 40 canonical work pages · 4 internal anchors

  1. [1]

    Cheng, K

    X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309.14341, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Hoeller, N

    D. Hoeller, N. Rudin, D. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(88):eadi7566, 2024

    work page 2024

  3. [3]

    Zhuang, Z

    Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint arXiv:2309.05665, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learning

    E. Chane-Sane, P.-A. Leziart, T. Flayols, O. Stasse, P. Sou `eres, and N. Mansard. Cat: Constraints as terminations for legged locomotion reinforcement learning. arXiv preprint arXiv:2403.18765, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Chane-Sane, J

    E. Chane-Sane, J. Amigo, T. Flayols, L. Righetti, and N. Mansard. Soloparkour: Constrained reinforcement learning for visual locomotion from privileged experience. In Conference on Robot Learning. arXiv, 2024

    work page 2024

  6. [6]

    Radosavovic, T

    I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89):eadi9579, 2024

    work page 2024

  7. [7]

    Zhuang, S

    Z. Zhuang, S. Yao, and H. Zhao. Humanoid parkour learning. arXiv preprint arXiv:2406.10759, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Handa, A

    A. Handa, A. Allshire, V . Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5977–5984. IEEE, 2023

    work page 2023

  9. [9]

    Allshire, M

    A. Allshire, M. MittaI, V . Lodaya, V . Makoviychuk, D. Makoviichuk, F. Widmaier, M. W ¨uthrich, S. Bauer, A. Handa, and A. Garg. Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 11802–11809. IEEE, 2022. 9

    work page 2022

  10. [10]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning, 2019. URL https://arxiv.org/ abs/1509.02971

    Pith/arXiv arXiv 2019

  11. [11]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

    work page 2018

  12. [12]

    Haarnoja, A

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications, 2019. URL https: //arxiv.org/abs/1812.05905

    Pith/arXiv arXiv 2019

  13. [13]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  14. [14]

    Makoviychuk, L

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021

    work page 2021

  15. [15]

    Rudin, D

    N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. InConference on Robot Learning, pages 91–100. PMLR, 2022

    work page 2022

  16. [16]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026–

    work page 2012

  17. [17]

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax–a differen- tiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    J. Xu, V . Makoviychuk, Y . Narang, F. Ramos, W. Matusik, A. Garg, and M. Macklin. Accel- erated policy learning with parallel differentiable simulation. In International Conference on Learning Representations, 2021

    work page 2021

  19. [19]

    E. Xing, V . Luk, and J. Oh. Stabilizing reinforcement learning in differentiable multiphysics simulation. arXiv preprint arXiv:2412.12089, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020

    Pith/arXiv arXiv 2010

  21. [21]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Janner, J

    M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019

    work page 2019

  23. [23]

    Clavera, V

    I. Clavera, V . Fu, and P. Abbeel. Model-augmented actor-critic: Backpropagating through paths, 2020. URL https://arxiv.org/abs/2005.08068

    Pith/arXiv arXiv 2020

  24. [24]

    Elsayed, G

    M. Elsayed, G. Vasan, and A. R. Mahmood. Deep reinforcement learning without experi- ence replay, target networks, or batch updates. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability

    work page 2024

  25. [25]

    R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximat- ing dynamic programming. In Machine learning proceedings 1990 , pages 216–224. Elsevier, 1990. 10

    work page 1990

  26. [26]

    S. Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning , pages 2829–2838. PMLR, 2016

    work page 2016

  27. [27]

    Kurutach, I

    T. Kurutach, I. Clavera, Y . Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018

    Pith/arXiv arXiv 2018

  28. [28]

    Buckman, D

    J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion.Advances in neural information processing systems, 31, 2018

    work page 2018

  29. [29]

    Feinberg, A

    V . Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803.00101 , 2018

    Pith/arXiv arXiv 2018

  30. [30]

    T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization, 2020. URL https://arxiv.org/abs/2005.13239

    Pith/arXiv arXiv 2020

  31. [31]

    C. Li, A. Krause, and M. Hutter. Offline robotic world model: Learning robotic policies without a physics simulator, 2025. URL https://arxiv.org/abs/2504.16680

    arXiv 2025

  32. [32]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=S1lOTC4tDS

    work page 2020

  33. [33]

    P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning, pages 2226–2240. PMLR, 2023

    work page 2023

  34. [34]

    Ghugare, H

    R. Ghugare, H. Bharadhwaj, B. Eysenbach, S. Levine, and R. Salakhutdinov. Simplifying model-based rl: learning representations, latent-space models, and policies with one objective. arXiv preprint arXiv:2209.08466, 2022

    Pith/arXiv arXiv 2022

  35. [35]

    Georgiev, V

    I. Georgiev, V . Giridhar, N. Hansen, and A. Garg. Pwm: Policy learning with large world models. arXiv preprint arXiv:2407.02466, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    B. Amos, S. Stanton, D. Yarats, and A. G. Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. In Learning for Dynamics and Control , pages 6–20. PMLR, 2021

    work page 2021

  37. [37]

    Byravan, J

    A. Byravan, J. T. Springenberg, A. Abdolmaleki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. InConference on Robot Learning, pages 566–589. PMLR, 2020

    work page 2020

  38. [38]

    Lambert, K

    N. Lambert, K. Pister, and R. Calandra. Investigating compounding prediction errors in learned dynamics models, 2022. URL https://arxiv.org/abs/2203.09637

    Pith/arXiv arXiv 2022

  39. [39]

    C. Xiao, Y . Wu, C. Ma, D. Schuurmans, and M. M ¨uller. Learning to combat compounding- error in model-based reinforcement learning, 2019. URL https://arxiv.org/abs/1912. 11206

    work page 2019

  40. [40]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  41. [41]

    Y . Niu, Y . Pu, Z. Yang, X. Li, T. Zhou, J. Ren, S. Hu, H. Li, and Y . Liu. Lightzero: A unified benchmark for monte carlo tree search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024. 11

    work page 2024

  42. [42]

    Y . Pu, Y . Niu, J. Ren, Z. Yang, H. Li, and Y . Liu. Unizero: Generalized and efficient planning with scalable latent world models. arXiv preprint arXiv:2406.10667, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  43. [43]

    C. Xuan, Y . Niu, Y . Pu, S. Hu, Y . Liu, and J. Yang. Rezero: Boosting mcts-based algorithms by backward-view and entire-buffer reanalyze. arXiv preprint arXiv:2404.16364, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018

    work page 2018

  45. [45]

    Hafner, T

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning , pages 2555–2565. PMLR, 2019

    work page 2019

  46. [46]

    Wang and J

    T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint arXiv:1906.08649, 2019

    Pith/arXiv arXiv 1906

  47. [47]

    Hansen, X

    N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. In ICML, 2022

    work page 2022

  48. [48]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024

    work page 2024

  49. [49]

    Bechtle, Y

    S. Bechtle, Y . Lin, A. Rai, L. Righetti, and F. Meier. Curious ilqr: Resolving uncertainty in model-based rl. In Proceedings of the Conference on Robot Learning, volume 100 of Proceed- ings of Machine Learning Research , page 162–171, Osaka, Japan, Nov. 2019

    work page 2019

  50. [50]

    Deisenroth and C

    M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference on machine learning (ICML- 11), pages 465–472, 2011

    work page 2011

  51. [51]

    Levine and P

    S. Levine and P. Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Wein- berger, editors, Advances in Neural Information Processing Systems , volume 27. Curran As- sociates, Inc., 2014. URL https://proceedings.neurips.cc/paper_files/paper/ 2014/file/6766a...

    work page 2014

  52. [52]

    Heess, G

    N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y . Tassa. Learning continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015

    work page 2015

  53. [53]

    Y . Song, S. Kim, and D. Scaramuzza. Learning quadruped locomotion using differentiable simulation. arXiv preprint arXiv:2403.14864, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Georgiev, K

    I. Georgiev, K. Srinivasan, J. Xu, E. Heiden, and A. Garg. Adaptive horizon actor-critic for pol- icy learningin contact-rich differentiable simulation. In International Conference on Machine Learning. PMLR, 2024

    work page 2024

  55. [55]

    DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulation

    J. Bagajo, C. Schwarke, V . Klemm, I. Georgiev, J.-P. Sleiman, J. Tordesillas, A. Garg, and M. Hutter. Diffsim2real: Deploying quadrupedal locomotion policies purely trained in differ- entiable simulation. arXiv preprint arXiv:2411.02189, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  56. [56]

    L. Metz, C. D. Freeman, S. S. Schoenholz, and T. Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05803, 2021

    Pith/arXiv arXiv 2021

  57. [57]

    H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning , pages 20668–20696. PMLR, 2022. 12

    work page 2022

  58. [58]

    Q. L. Lidec, L. Montaut, C. Schmid, I. Laptev, and J. Carpentier. Augmenting differentiable physics with randomized smoothing. arXiv preprint arXiv:2206.11884, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  59. [59]

    D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022. URLhttps://arxiv. org/abs/1312.6114

    Pith/arXiv arXiv 2022

  60. [60]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Rai- son, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Py- torch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer...

    work page

  61. [61]

    Duclusaud, G

    M. Duclusaud, G. Passault, V . Padois, and O. Ly. Extended friction models for the physics simulation of servo actuators, 2025. URL https://arxiv.org/abs/2410.08650

    work page arXiv 2025

  62. [62]

    G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning, pages 22–31. PMLR, 2023

    work page 2023

  63. [63]

    Y . Li, J. Li, W. Fu, and Y . Wu. Learning agile bipedal motions on a quadrupedal robot. In2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 9735–9742. IEEE, 2024

    work page 2024

  64. [64]

    0.2 0 −1.0 # (yaw-rotated world vector) – vf = RW B

    S. Huang, Q. Gallou ´edec, F. Felten, A. Raffin, R. F. J. Dossa, Y . Zhao, R. Sullivan, V . Makoviy- chuk, D. Makoviichuk, M. H. Danesh, C. Roum ´egous, J. Weng, C. Chen, M. M. Rah- man, J. G. M. Ara ´ujo, G. Quan, D. Tan, T. Klein, R. Charakorn, M. Towers, Y . Berthelot, K. Mehta, D. Chakraborty, A. KG, V . Charraut, C. Ye, Z. Liu, L. N. Alegre, A. Nikul...

    Pith/arXiv arXiv 2024

  65. [2019]

    URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

    work page 2019

  66. [5033]

    doi:10.1109/IROS.2012.6386109

    IEEE, 2012. doi:10.1109/IROS.2012.6386109

    arXiv 2012

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.