Pith. sign in

REVIEW 3 major objections 5 minor 38 references

A Comparison of Action Spaces for Learning Manipulation Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Impedance actions beat torque in every manipulation task tested

desk verdict Useful empirical comparison of action spaces for manipulation RL, but the central ordering rests on single runs with no uncertainty estimates. read the letter →

arxiv 1908.08659 v1 pith:MGOF76JJ submitted 2019-08-23 cs.RO cs.LG

Patrick Varin , Lev Grossman , Scott Kuindersma This is my paper · ORCID
classification cs.ROcs.LG
keywords reinforcementlearningmanipulationactionspacesimpedancecontrolinversedynamicssampleefficiencyproximalpolicyoptimizationsoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the action space of a reinforcement learning policy—what the policy's output actually commands—matters as much as the learning algorithm. It argues that defining actions as reference inputs to a task-space impedance controller makes dynamic manipulation tasks dramatically more sample-efficient than outputting joint torques or joint-space PD references. Across three simulated tasks (peg insertion, hammering, and pushing) and two algorithms (PPO and SAC), the impedance action space reached success criteria fastest in all six conditions, often by a factor of several. If this holds, a large share of the sample cost of learning manipulation can be avoided by better wiring the policy output through existing model-based controllers, without changing rewards or algorithms.

What carries the argument

The central object is the action space of the Markov decision process, specifically the choice to make actions references $(x_{\mathrm{des}}, \dot{x}_{\mathrm{des}})$ for a task-space impedance controller. This is a low-level controller that drives the end effector to behave like a mass-spring-damper toward a desired pose and velocity while using an inverse-dynamics model to compensate for the arm's inertial, Coriolis, and gravitational terms; a null-space damping term handles redundant degrees of freedom. Its work in the paper is to replace the policy's burden of learning joint-level dynamics and inverse kinematics with a much simpler problem: nudging a compliant task-space target, so that exploration is spent on the task rather than on robot mechanics.

What would settle it

Re-run the same three tasks and two algorithms with, say, ten seeds per action space and report the distribution of steps to success; if the impedance-reference action space is not fastest in most cells after averaging, the claimed uniform ordering is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims a consistent ordering of learning speed across tasks and algorithms: impedance references are fastest, inverse dynamics next in most cases, then joint PD, then direct torque. The authors write that in all experiments the impedance controller action space learned fastest, often followed by inverse dynamics, PD, and then torque. They attribute the advantage to two mechanisms: the impedance controller provides inherent mechanical compliance, which matters for contact-rich tasks such as pushing a block without tipping it, and because actions are specified in task space, the policy's output aligns directly with reward-relevant quantities, so it does not need to solve inverse kinematics during exploration. The margin can be large: in hammering with SAC, impedance reached the success criterion in 0.012 million steps versus 0.145 million for torque.

Load-bearing premise

The ranking in Table I is based on one untested run per condition, so the claim assumes those single runs are representative of each action-space and algorithm combination.

Editorial extensions

If this is right

  • Practitioners can reduce the sample budget for learning manipulation tasks by choosing impedance-reference actions over torque or PD references, without changing the reward function or learning algorithm.
  • Exploiting known inertial and kinematic models of the arm helps even when contact dynamics are not modeled; the gains come from using available structure, not from modeling the environment.
  • Inverse dynamics control is a cheaper middle ground that outperforms PD control in most conditions, particularly in velocity-dominated tasks such as hammering.
  • The action-space choice changes qualitative behavior: policies trained in compliant action spaces produce slower, controlled motions, while torque policies tend to collide aggressively with the environment.
  • A fair comparison of torque and PD baselines requires careful engineering, including gravity compensation and mass scaling, so naive baselines may underestimate their sample cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The task-space alignment mechanism suggests a testable extension: if the reward is re-expressed in joint coordinates, the impedance advantage should shrink, and if the reward is purely task-space, it should grow.
  • A natural next experiment is to make the impedance gains themselves learnable or state-dependent, since the paper keeps them fixed and does not explore variable impedance.
  • If the same ordering transfers to physical hardware, the impedance action space may also reduce the sim-to-real gap, because the learned policy commands compliant task-space goals rather than raw torques while the low-level controller absorbs dynamic effects.
  • The paper's framing implies that the largest remaining sample-efficiency gains in manipulation may come from choosing the right interface between the policy and the controller rather than from further advances in RL algorithms alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares four action spaces (direct torque, joint-space PD, inverse dynamics, and task-space impedance) for reinforcement learning of three simulated manipulation tasks (peg insertion, hammering, pushing) with two algorithms (PPO and SAC). The central claim is that defining the action space as references to a task-space impedance controller substantially improves sample efficiency, with a consistent ranking across all six task/algorithm combinations. The evidence is Table I, which reports the number of steps to reach task-specific success thresholds, together with qualitative observations about learned behavior. The paper frames its contribution as preliminary evidence that incorporating model-based controllers, particularly impedance control, into the action space reduces sample complexity for dynamic manipulation.

Significance. If the claimed effect is real, the paper provides a practically useful and systematic comparison for a design choice that is often made ad hoc: how to parameterize the action space for model-free RL on manipulation tasks. Its scope is broad (four action spaces, three tasks, two RL algorithms), the code and hyperparameters are released, and the finding that impedance references outperform direct torque and joint-space PD is plausible and consistent with prior work on residual and model-based control. The main limitation is statistical: with a single run per condition and no variance estimates, the paper cannot yet support the strength of its abstract and conclusion claims. The manuscript is a reasonable pilot study, but the present evidence is too thin for the current wording.

major comments (3)
  1. [Section VI, Table I and Abstract] The central claim rests entirely on Table I, but each of the 24 conditions appears to have been run exactly once, with no repeated seeds, no error bars, and no variance estimates. In deep RL, sample-efficiency curves are strongly seed-dependent, so a strict ordering observed in single runs cannot distinguish a robust effect from stochastic variation. The abstract's 'significantly reduces' and Section VI's 'we found the impedance controller action space to have learned the fastest' are therefore not supported by the evidence as presented. Please report multiple seeds (at least 5) with medians and interquartile ranges, or explicitly downgrade the language to 'in the runs we performed' throughout the abstract and conclusions.
  2. [Section IV and Appendix] The comparison is partly a comparison of the authors' tuning effort rather than of the action spaces alone. The PD proportional gains (Kp in Eq. (3)), the impedance stiffness K and damping B in Eq. (11), and the derivative gains are hand-chosen, but their numerical values are not reported anywhere. Section VI even states that increasing PD gains by an order of magnitude would harm compliance, which shows that the ranking is sensitive to these choices. Please report the exact gain values and the tuning procedure used, or state clearly that the conclusions are conditional on the specific gains selected.
  3. [Section VI, success criteria] The success thresholds (80% average nail depth, 80% insertion success rate, 80% of normalized pushing distance) are arbitrary, and the sample counts in Table I are measured against these thresholds. Different thresholds could easily change the relative ordering, especially where the values are close (e.g., SAC insertion: impedance 0.39 vs. ID 0.47, or PPO pushing: impedance 0.21 vs. PD 0.23). Please justify the threshold choices or provide a sensitivity analysis over a range of thresholds.
minor comments (5)
  1. [Table I] The SAC pushing row reads '0.18 0.24 0.90 * 1'; there appear to be five entries for four columns, and the footnote says the torque action space never crossed the threshold. Please fix the table formatting so the PD and torque entries are unambiguous.
  2. [Appendix] The text says 'The episode lengths were capped at 2 seconds for object pushing and peg insertion and 3 seconds for hamming.' The word 'hamming' should be 'hammering'.
  3. [Section IV-D, Eq. (10)] Equation (10) applies the pseudoinverse J^+ to a sum that includes (K(x_des - x(q)) + \dot{x}_des - B J \dot{q} - \dot{J} \dot{q}); the parentheses are unbalanced and the notation mixes Cartesian and joint quantities. The simplified law in Eq. (11) then drops the \dot{J}\dot{q} term. Please clarify the derivation or add a sentence explaining that Eq. (11) is the implemented approximation.
  4. [Figure 2] Figure 2 is referenced as showing learning trends, but the figure itself contains only qualitative stills without learning curves or axes. Either replace it with actual learning curves or remove the reference in the text.
  5. [Section II] The phrase 'inject very little bias into the learning problem' is used for torque and PD control; it would be clearer to specify what kind of bias is meant, for example, the controller does not impose a particular dynamic response.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sample-efficiency comparisons are measured outcomes, not derived from fitted constants or self-citations.

full rationale

The paper's central claim is an empirical comparison of four RL action spaces measured by the number of training steps required to reach task-specific success thresholds. Table I reports measured step counts, and the text's ranking (impedance fastest, followed by ID, PD, torque) is a summary of those measurements rather than a quantity derived from the action-space definitions or from any fitted parameter. The impedance controller does embed a compliance model by construction, but the paper does not derive its faster learning from that construction; it reports observed learning curves and then offers compliance and task-space alignment as plausible explanations. No parameter is fit to a subset of the data and then renamed as a prediction, and no load-bearing claim rests on a self-citation. The authors' own GitHub repository is cited only as a code-release pointer, not as evidence for the main result. The most serious weakness in the paper is methodological rather than circular: the single-run-per-condition experiments lack seed variance and statistical error bars, so the uniform ordering across all six task-algorithm cells is not statistically established. That concern belongs to evidence quality and reproducibility, not to circular derivation. Under the hard rules requiring a quoted reduction or fitted-input prediction to flag circularity, no such step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and does not introduce new theoretical entities. The main costs are hand-tuned controller gains and arbitrary success thresholds, plus domain assumptions about simulation fidelity and algorithm implementations.

free parameters (3)
  • PD controller proportional gains (unscaled Kp) = not reported
    Chosen per task and per joint by hand; determines stiffness and tracking, affecting the relative performance of PD and impedance action spaces. Values are not given in the paper.
  • Impedance stiffness K and damping B = not reported
    Hand-selected per task; central to the impedance action space's compliant behavior and thus to the main result. Values are not reported, limiting reproducibility.
  • Success criteria thresholds = 80%
    Defined as 80% nail depth, 80% insertion success, or 80% of normalized distance, these thresholds determine the measured sample counts and could affect the ranking if changed.
assumptions (3)
  • domain assumption The inverse dynamics and impedance controllers have exact knowledge of the simulated manipulator dynamics (H, C, G, and Jacobian from the same MuJoCo model).
    The model-based controllers use the simulator's own model; any model mismatch would change the comparison, and the paper explicitly notes that simulation-to-hardware transfer is not addressed (Section VII).
  • domain assumption MuJoCo's simulated dynamics and contact models are a valid proxy for real manipulation, including the modified zero-gravity variants.
    All results are in simulation, with gravity disabled for hammering and insertion to allow the torque controller to learn; the validity of the ranking outside these conditions is assumed.
  • domain assumption The stable-baselines implementations of PPO and SAC correctly instantiate the published algorithms with the stated hyperparameters.
    The comparison relies on off-the-shelf implementations; correctness of these implementations is taken as given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comparison of Action Spaces for Learning Manipulation Tasks." pith.science (2026). https://pith.science/paper/MGOF76JJ

@misc{pith2026190808659,
  author       = {Pith},
  title        = {Pith review of: A Comparison of Action Spaces for Learning Manipulation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGOF76JJ}},
  note         = {Machine review of arXiv:1908.08659}
}
read the original abstract

Designing reinforcement learning (RL) problems that can produce delicate and precise manipulation policies requires careful choice of the reward function, state, and action spaces. Much prior work on applying RL to manipulation tasks has defined the action space in terms of direct joint torques or reference positions for a joint-space proportional derivative (PD) controller. In practice, it is often possible to add additional structure by taking advantage of model-based controllers that support both accurate positioning and control of the dynamic response of the manipulator. In this paper, we evaluate how the choice of action space for dynamic manipulation tasks affects the sample complexity as well as the final quality of learned policies. We compare learning performance across three tasks (peg insertion, hammering, and pushing), four action spaces (torque, joint PD, inverse dynamics, and impedance control), and using two modern reinforcement learning algorithms (Proximal Policy Optimization and Soft Actor-Critic). Our results lend support to the hypothesis that learning references for a task-space impedance controller significantly reduces the number of samples needed to achieve good performance across all tasks and algorithms.

Figures

Figures reproduced from arXiv: 1908.08659 by the authors.

Figure 1
Figure 1. Simulated experiments from left to right: peg insertion, hammering, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Training examples from learning hammering, pushing, and peg insertion using PPO and SAC. Hammering with PPO (left) and SAC (middle) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Frames from a learned hammer policy. The policy learns to take multiple swings at the nail in order to complete the task. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages

  1. [1]

    Learning Dexterous In-Hand Manipulation,

    OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning Dexterous In-Hand Manipulation,” arXiv:1808.00177 [cs, stat], Aug. 2018

    arXiv 2018

  2. [2]

    Learning Hand- Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection,

    S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning Hand- Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection,” arXiv:1603.02199 [cs] , Mar. 2016

    arXiv 2016

  3. [3]

    Continuous control with deep reinforce- ment learning,

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforce- ment learning,” arXiv:1509.02971 [cs, stat] , Sept. 2015

    arXiv 2015

  4. [4]

    Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images,

    M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller, “Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images,” in Proceedings of the 28th International Conference on Neural Information Processing Systems - V olume 2 , NIPS’15, (Cambridge, MA, USA), pp. 2746–2754, MIT Press, 2015

    work page 2015

  5. [5]

    Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates,

    S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates,” arXiv:1610.00633 [cs] , Oct. 2016

    arXiv 2016

  6. [6]

    End-to-End Training of Deep Visuomotor Policies,

    S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-End Training of Deep Visuomotor Policies,” arXiv:1504.00702 [cs] , Apr. 2015

    arXiv 2015

  7. [7]

    Data- efficient Deep Reinforcement Learning for Dexterous Manipulation,

    I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Ve- cerik, T. Lampe, Y . Tassa, T. Erez, and M. Riedmiller, “Data- efficient Deep Reinforcement Learning for Dexterous Manipulation,” arXiv:1704.03073 [cs] , Apr. 2017

    arXiv 2017

  8. [8]

    Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

    Y . Chebotar, K. Hausman, M. Zhang, G. Sukhatme, S. Schaal, and S. Levine, “Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning,” arXiv:1703.03078 [cs] , Mar. 2017

    work page Pith review arXiv 2017

Show all 38 references
  1. [9]

    Robotic Arm Control and Task Training through Deep Reinforcement Learn- ing,

    A. Franceschetti, E. Tosello, N. Castaman, and S. Ghidoni, “Robotic Arm Control and Task Training through Deep Reinforcement Learn- ing,” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , p. 8, 2018

    2018

  2. [10]

    Deep- Mimic: Example-Guided Deep Reinforcement Learning of Physics- Based Character Skills,

    X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne, “Deep- Mimic: Example-Guided Deep Reinforcement Learning of Physics- Based Character Skills,” arXiv:1804.02717 [cs] , Apr. 2018

    2018 arXiv

  3. [11]

    Learning Force Control Policies for Compliant Manipulation,

    M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal, “Learning Force Control Policies for Compliant Manipulation,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 6, 2011

    2011

  4. [12]

    Residual Reinforcement Learning for Robot Control,

    T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine, “Residual Reinforcement Learning for Robot Control,” arXiv:1812.03201 [cs] , Dec. 2018

    2018 arXiv

  5. [13]

    Quasi-Static Assembly of Compliantly Supported Rigid Parts,

    D. E. Whitney, “Quasi-Static Assembly of Compliantly Supported Rigid Parts,” Journal of Dynamic Systems, Measurement, and Control , vol. 104, pp. 65–77, Mar. 1982

    1982

  6. [14]

    Proximal Policy Optimization Algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv:1707.06347 [cs] , July 2017

    2017 arXiv

  7. [15]

    Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” arXiv:1801.01290 [cs, stat] , Jan. 2018

    2018 arXiv

  8. [16]

    Soft Actor- Critic Algorithms and Applications,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft Actor- Critic Algorithms and Applications,”arXiv:1812.05905 [cs, stat], Dec. 2018

    2018 arXiv

  9. [17]

    Learning reactive ad- mittance control,

    V . Gullapalli, R. A. Grupen, and A. G. Barto, “Learning reactive ad- mittance control,” in Proceedings 1992 IEEE International Conference on Robotics and Automation , pp. 1475–1480 vol.2, May 1992

    1992

  10. [18]

    Impedance Learning for Robotic Contact Tasks Using Natural Actor-Critic Algorithm,

    B. Kim, J. Park, S. Park, and S. Kang, “Impedance Learning for Robotic Contact Tasks Using Natural Actor-Critic Algorithm,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , vol. 40, pp. 433–443, Apr. 2010

    2010

  11. [19]

    Learning variable impedance control,

    J. Buchli, F. Stulp, E. Theodorou, and S. Schaal, “Learning variable impedance control,” The International Journal of Robotics Research , vol. 30, pp. 820–833, June 2011

    2011

  12. [20]

    Composable Deep Reinforcement Learning for Robotic Manipula- tion,

    T. Haarnoja, V . Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine, “Composable Deep Reinforcement Learning for Robotic Manipula- tion,” arXiv:1803.06773 [cs, stat] , Mar. 2018

    2018 arXiv

  13. [21]

    Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,

    A. Rajeswaran, V . Kumar, A. Gupta, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” arXiv:1709.10087 [cs], Sept. 2017

    2017 arXiv

  14. [22]

    Stable Proportional-Derivative Con- trollers,

    J. Tan, K. Liu, and G. Turk, “Stable Proportional-Derivative Con- trollers,” IEEE Computer Graphics and Applications , vol. 31, pp. 34– 44, July 2011

    2011

  15. [23]

    Interactive Character Animation Using Simulated Physics: A State-of-the-Art Review,

    T. Geijtenbeek and N. Pronost, “Interactive Character Animation Using Simulated Physics: A State-of-the-Art Review,” Computer Graphics F orum, vol. 31, no. 8, pp. 2492–2515, 2012

    2012

  16. [24]

    Simulation and Control of Skeleton-driven Soft Body Characters,

    L. Liu, K. Yin, B. Wang, and B. Guo, “Simulation and Control of Skeleton-driven Soft Body Characters,” ACM Trans. Graph. , vol. 32, pp. 215:1–215:8, Nov. 2013

    2013

  17. [25]

    Articulated Swimming Creatures,

    J. Tan, Y . Gu, G. Turk, and C. K. Liu, “Articulated Swimming Creatures,” in ACM SIGGRAPH 2011 Papers , SIGGRAPH ’11, (New York, NY , USA), pp. 58:1–58:12, ACM, 2011

    2011

  18. [26]

    Goal directed multi-finger manipulation: Control policies and analysis,

    S. Andrews and P. G. Kry, “Goal directed multi-finger manipulation: Control policies and analysis,” Computers & Graphics , vol. 37, pp. 830–839, Nov. 2013

    2013

  19. [27]

    Guided Learning of Control Graphs for Physics-Based Characters,

    L. Liu, M. V . D. Panne, and K. Yin, “Guided Learning of Control Graphs for Physics-Based Characters,” ACM Trans. Graph. , vol. 35, pp. 29:1–29:14, May 2016

    2016

  20. [28]

    DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning,

    X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne, “DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning,” ACM Trans. Graph. , vol. 36, pp. 41:1–41:13, July 2017

    2017

  21. [29]

    Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning,

    L. Liu and J. Hodgins, “Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning,” ACM Trans. Graph., vol. 36, June 2017

    2017

  22. [30]

    Residual Policy Learning,

    T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, “Residual Policy Learning,” arXiv:1812.06298 [cs] , Dec. 2018

    2018 arXiv

  23. [31]

    Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?,

    X. B. Peng and M. van de Panne, “Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?,” in Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation, SCA ’17, (New York, NY , USA), pp. 12:1–12:13, ACM, 2017

    2017

  24. [32]

    Stable baselines

    A. Hill, A. Raffin, M. Ernestus, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y . Wu, “Stable baselines.”https://github.com/hill-a/ stable-baselines, 2018

    2018

  25. [33]

    Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation,

    R. S. Sutton, D. M. A. llester, S. Singh, and Y . Mansour, “Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation,” in Advances in Neural Information Processing Systems (NIPS), pp. 1057–1063, 2000

    2000

  26. [34]

    Trust Region Policy Optimization,

    J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “Trust Region Policy Optimization,” arXiv:1502.05477 [cs] , Feb. 2015

    2015 arXiv

  27. [35]

    Reinforcement Learning with Deep Energy-Based Policies,

    T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, “Reinforcement Learning with Deep Energy-Based Policies,” arXiv:1702.08165 [cs] , Feb. 2017

    2017 arXiv

  28. [36]

    Addressing Function Approximation Error in Actor-Critic Methods,

    S. Fujimoto, H. van Hoof, and D. Meger, “Addressing Function Approximation Error in Actor-Critic Methods,” arXiv:1802.09477 [cs, stat], Feb. 2018

    2018 arXiv

  29. [37]

    MuJoCo: A physics engine for model-based control,

    E. Todorov, T. Erez, Y . Tassa, and tassa, “MuJoCo: A physics engine for model-based control,” in Proceedings of the IEEERAS International Conference on Intelligent Robots , 2012

    2012

  30. [38]

    Gym-kuka-mujoco

    P. Varin and L. Grossman, “Gym-kuka-mujoco.” HarvardAgileR- oboticsLab, Feb. 2019

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.