Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces MTBench, a GPU-accelerated benchmark that runs 70 multi-task robotics environments in parallel, and presents evidence that in this regime on-policy methods beat off-policy methods, wall-clock time matters more than…

desk verdict Useful benchmark with real engineering substance, but the Meta-World conclusions rest on a training-time success metric the authors never validate against the standard evaluator. read the letter →

arxiv 2507.23172 v2 pith:NYRTQ224 submitted 2025-07-31 cs.RO

Viraj Joshi , Zifan Xu , Bo Liu , Peter Stone , Amy Zhang This is my paper
classification cs.RO
keywords multi-taskreinforcementlearningmassivelyparalleltrainingGPU-acceleratedsimulationroboticsbenchmarkon-policyvsoff-policycurriculumMeta-WorldParkour
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-task reinforcement learning (MTRL) research has mostly used off-policy algorithms like SAC on CPU-based benchmarks, where experiments take days or weeks. This paper asks what happens when MTRL is moved to massively parallelized GPU-accelerated simulation, and argues that the field's off-policy assumption no longer holds. It presents MTBench, an open benchmark with 50 manipulation and 20 locomotion tasks that runs thousands of environments in parallel on one GPU, plus implementations of four base algorithms and seven MTRL schemes. The central claim is that MTBench makes MTRL evaluation dramatically faster while changing which methods win: on-policy PPO outperforms SAC, multi-task conflict mainly hurts the critic rather than the policy, and curriculum learning is essential for sparse-reward locomotion tasks.

What carries the argument

The central object is MTBench, a benchmark built on NVIDIA IsaacGym's Tensor API that allocates fixed blocks of GPU-simulated environments to each task, so heterogeneous tasks are simulated simultaneously in a single process and all data stays on the GPU. This is what makes on-policy algorithms practical: methods like MT-PPO can collect fresh batches from the current policy across thousands of environments without CPU-GPU transfer. The second piece of machinery is a critic-free variant (MT-GRPO with Monte Carlo returns) used to isolate the value-learning bottleneck, and the third is a simple difficulty-based curriculum for the Parkour domain.

What would settle it

Run a standard Meta-World evaluation (separate rollout with fixed initial states, success measured at the end or at any point) on MTBench-trained policies and compare success rates and rankings to the training-environment metric; large discrepancies would invalidate the benchmark's comparability claim. Alternatively, if an off-policy method with a well-tuned update-to-data ratio matches MT-PPO's wall-clock success rate on MT50-rand, observation O1 would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that in the massively parallel regime, the standard assumptions of MTRL flip: choosing an on-policy algorithm matters more than which MTRL scheme is applied, and wall-clock efficiency, not sample efficiency, is the right comparison. Concretely, MT-PPO and MT-GRPO reach substantially higher success rates than MT-SAC in Meta-World MT10-rand and MT50-rand while taking roughly 22 minutes versus 12 hours at 200M frames, and the gap grows with task count. The paper identifies value learning as the key bottleneck: gradient conflicts appear in critic gradients, and removing the critic entirely (MT-GRPO with Monte Carlo returns) nearly matches or beats most MTRL schemes. In sparse-reward Parkour tasks, no MTRL method fixes exploration by itself, and a simple curriculum that advances terrain difficulty yields about 10% progress gain. These observations come from a unified benchmark, MTBench, that assigns blocks of parallel environments to tasks in IsaacGym.

Load-bearing premise

The reported success rate is the proportion of training environments that terminate successfully during training, averaged over the last five epochs, rather than a separate evaluation rollout over fixed initial states; if this does not match standard Meta-World success evaluation, the method rankings are not comparable to prior work.

Editorial extensions

If this is right

  • MTRL research should shift to on-policy methods in massively parallelized settings; off-policy methods like SAC need substantial reworking or adaptive update-to-data ratios to be competitive.
  • Wall-clock time becomes the primary comparison metric for MTRL, since experience collection scales easily with more GPUs.
  • Future MTRL method development should target the critic: reducing gradient conflicts in value learning, or removing the critic via Monte Carlo returns, improves performance.
  • Sparse-reward multi-task training in this regime requires curriculum learning; MTRL schemes alone do not provide exploration.
  • MTBench provides a common testbed that can cut evaluation time from days to hours, enabling broader hyperparameter and seed sweeps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success-rate metric is computed from training environments that terminate during the last 5 epochs rather than from separate evaluation rollouts (Appendix B.1); if that measure diverges from standard Meta-World success evaluation, the reported numbers and rankings may not be directly comparable to prior Meta-World results.
  • The benchmark's speed advantage suggests MTRL could adopt the 'pretrain then finetune' paradigm: use MTBench's environments to rapidly collect offline data and finetune online, a direction the authors mention as future work.
  • If value learning is the bottleneck, then off-policy methods that reduce critic bias through n-step returns, such as the adapted PQN, may become more competitive with on-policy methods once tuned further.
  • A testable extension is to apply MTBench's curriculum idea to dense-reward Meta-World tasks with task difficulty ratings; the claim that curriculum matters mainly for sparse rewards predicts little gain, whereas a large gain would weaken observation O4.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MTBench, a GPU-accelerated benchmark for massively parallelized multi-task reinforcement learning, comprising 50 Meta-World manipulation tasks and 20 Parkour locomotion tasks. It integrates four base RL algorithms (MT-PPO, MT-GRPO, MT-SAC, MT-PQN) with seven MTRL schemes (PCGrad, CAGrad, FAMO, Soft-Modularization, CARE, PaCo, MOORE) and reports experiments across MT10/MT50 and Parkour-easy/hard settings. Based on the results, the paper proposes four observations: on-policy methods outperform off-policy methods in this regime, wall-clock time should be prioritized over sample efficiency, value learning is the main bottleneck in MTRL, and curriculum learning is essential for sparse-reward tasks. The central claim is that MTBench provides a fast and unified evaluation framework for MTRL and uncovers unique challenges from combining massive parallelism with MTRL.

Significance. If the observations are robust, the paper makes a useful contribution by providing a high-throughput testbed for MTRL research and by documenting that on-policy methods are competitive or superior in the massively parallel regime. The paper's strengths include extensive experiments with 10 seeds and bootstrap confidence intervals, a broad coverage of MTRL algorithms and architectures, and the promise of open-source code. However, the validity of the Meta-World conclusions rests on a nonstandard training-time success metric that has not been calibrated against standard Meta-World evaluation, and several comparison protocols are not fully controlled (different robots, re-tuned rewards, differing environment counts across baselines). These issues must be addressed before the benchmark claims and the observations can be taken at face value.

major comments (3)
  1. [Appendix B.1] The success rate definition is the proportion of training environments that succeed during training, averaged over the last 5 epochs, rather than a held-out evaluation over fixed initial states with deterministic rollouts. This deviates from the standard Meta-World success metric and can overcount episodes where the agent reaches the goal by exploration noise rather than by policy competence. Because Figures 4, 5, 7, and 8 and Table 2 all use this metric, the method rankings and the observations O1-O3 are conditional on this measure. The paper acknowledges the deviation but provides no calibration against the standard Meta-World evaluation protocol. A comparability check, even on a subset of methods, is needed to establish that the reported rankings are not artifacts of the training-time metric.
  2. [Section 3.1 and Appendix D] The Meta-World re-implementation changes the robot from Sawyer to Franka Emika Panda and re-tunes reward functions 'to ensure that the tasks are individually solvable.' This means the results are not directly comparable to prior Meta-World numbers, and the paper does not quantify the impact of these changes. In addition, Appendix D shows that environment counts differ across methods: MT-PPO uses 24,576 environments for MT10/MT50, while PCGrad uses 24,576/8,192, CAGrad uses 24,576/6,144, MT-GRPO uses 4,096/24,576, MT-SAC uses 4,096, and MT-PQN uses 8,192. Since environment count is a key factor in massively parallel training, performance differences among methods may be confounded with this variable. The paper should either hold environment count fixed or provide a sensitivity analysis demonstrating that the observed rankings are robust to this choice.
  3. [Section 4.4 and Appendix D] The claim that MT-GRPO is a simple baseline using 'the same hyperparameters as MT-PPO' is contradicted by the hyperparameter tables. Table 4 reports horizon length 150 versus 32 for MT-PPO, minibatch size 16,384/76,800 versus 16,384/32,768, and number of environments 4,096/24,576 versus 24,576/24,576. Since the critic-elimination comparison is the primary evidence for observation O3, these differences could confound the result. The authors should either match all hyperparameters except the critic or explicitly analyze how the differing horizon, minibatch size, and environment count affect the comparison.
minor comments (5)
  1. [Section 3.2] The sentence 'The observation is compromised by proprioceptive observation in R48' should use 'composed of' instead of 'compromised by.'
  2. [Section 2.2] The sentence 'Each task τ is sampled the task distribution p(T )' is missing the word 'from' after 'sampled.'
  3. [Table 1] The table header says 'average success rate' but the columns report progress P in percent; the label should be corrected to 'progress' for clarity.
  4. [Figure 6] The figure shows cosine similarity ranges for actor and critic gradients, but it is unclear whether the shadow area is the min-max range over tasks, over seeds, or over time steps. Please clarify the computation and state how many runs are included.
  5. [Abstract] The phrase 'superior speed of evaluating MTRL approaches using MTBench' could be more precise; the experiments measure training and evaluation speed jointly, and the abstract should distinguish benchmarking throughput from the evaluation protocol itself.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's observations are empirical measurements, and its one self-referential evaluation choice (training-env success rate) affects comparability, not logical derivation.

full rationale

This is an empirical benchmark paper, not a derivation paper. The central claims (O1-O4) are supported by measured learning curves, bootstrap CIs, and ablations rather than by equations that reduce to their inputs. The one self-referential point is Appendix B.1: success rate is computed on the training environments (proportion of episodes ending in success during training, averaged over the last 5 epochs) rather than on held-out evaluation rollouts. This is a real threat to comparability with standard Meta-World numbers, but it is not circular in the derivation sense: no quantity is defined in terms of a conclusion, and the rankings could in principle have come out differently under this metric. The self-citations (FAMO, CAGrad, CARE) are baselines that are evaluated, not load-bearing evidence for the observations; the cited prior work on parallel RL (D'Oro, Li, Gallici) and curriculum (Liang) supplies context, not the proof of the benchmark's measurements. MT-GRPO is defined as critic-free MT-PPO, so Figure 8 is an ablation, not a result forced by definition. Therefore no circular step is exhibited; score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims are empirical benchmark findings, not derived from first principles. The benchmark relies on domain assumptions about simulation fidelity and evaluation validity, but no data-fitted free parameters are introduced to manufacture a prediction.

assumptions (3)
  • domain assumption IsaacGym Tensor API faithfully simulates the Meta-World and Parkour task dynamics with the re-implementations and re-tuned rewards.
    Used throughout Section 3; if the re-tuned rewards alter task difficulty, the benchmark results may not transfer to the original Meta-World or Eurekaverse.
  • domain assumption The success-rate metric measured during training episodes (proportion of environments that terminate with success, averaged over last 5 epochs) is a valid measure of task performance.
    Appendix B.1; the main experiments use this metric for all Meta-World conclusions.
  • domain assumption Appending a task embedding to a universal state space is sufficient for one policy to distinguish and solve all tasks.
    Section 2.2 problem statement; underlies all multi-task methods in the benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks." pith.science (2026). https://pith.science/paper/NYRTQ224

@misc{pith2026250723172,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYRTQ224}},
  note         = {Machine review of arXiv:2507.23172}
}
abstract

Multi-task Reinforcement Learning (MTRL) has emerged as a critical training paradigm for applying reinforcement learning (RL) to a set of complex real-world robotic tasks, which demands a generalizable and robust policy. At the same time, \emph{massively parallelized training} has gained popularity, not only for significantly accelerating data collection through GPU-accelerated simulation but also for enabling diverse data collection across multiple tasks by simulating heterogeneous scenes in parallel. However, existing MTRL research has largely been limited to off-policy methods like SAC in the low-parallelization regime. MTRL could capitalize on the higher asymptotic performance of on-policy algorithms, whose batches require data from the current policy, and as a result, take advantage of massive parallelization offered by GPU-accelerated simulation. To bridge this gap, we introduce a massively parallelized $\textbf{M}$ulti-$\textbf{T}$ask $\textbf{Bench}$mark for robotics (MTBench), an open-sourced benchmark featuring a broad distribution of 50 manipulation tasks and 20 locomotion tasks, implemented using the GPU-accelerated simulator IsaacGym. MTBench also includes four base RL algorithms combined with seven state-of-the-art MTRL algorithms and architectures, providing a unified framework for evaluating their performance. Our extensive experiments highlight the superior speed of evaluating MTRL approaches using MTBench, while also uncovering unique challenges that arise from combining massive parallelism with MTRL. Code is available at https://github.com/Viraj-Joshi/MTBench

Figures

Figures reproduced from arXiv: 2507.23172 by the authors.

Figure 1
Figure 1. MTBench is a benchmark that leverages massive parallelism for MTRL in two robotics [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustrations of non-parametric tasks variation, parametric tasks variation of Faucet Open, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustrations of non-parametric tasks variation, parametric tasks variation of Jump On and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Vanilla MTRL performance in Meta-World. We report the pointwise 95% percentile bootstrap CIs of the average success rates using 10 seeds for each RL algorithm in the MT10-rand and MT50-rand evaluation settings. On-policy methods (MT-PPO, MT-GRPO) continue to improve wi…
Figure 5
Figure 5. Figure 5: We compare the 95% bootstrapped confidence intervals of the average success rate of all [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The average gradient cosine similarity across all task pairs in MT10-rand for both actor and critic networks. The shadow areas represent the ranges between minimum and maximum co￾sine similarities. Multihead PaCO MH-MOORE MH-CARE Soft-Modularization 5 0 5 10 15 MT10 MT…
Figure 8
Figure 8. Figure 8: Eliminating the difficulty of critic estimation consistently improves performance over most MTRL approaches using MT-PPO when comparing the 95% bootstrapped CI of av￾erage success rates in Meta-World. Each ap￾proach uses 1B frames per run over 10 seeds. Value learning …
Figure 9
Figure 9. Figure 9: 95% bootstrapped CIs of the average success rate of all MT-PPO MTRL approaches using [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    MR.Q combines predictive auxiliary tasks with high-capacity value functions in a model-free architecture to achieve strong multitask RL performance without planning.

  2. TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    TOPPO reformulates PPO with critic balancing to address gradient ill-conditioning in multi-task RL and reports stronger mean and tail performance than SAC baselines on Meta-World+ using fewer parameters and steps.

  3. Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Simplicial embeddings — group-wise softmax feature layers — improve sample efficiency and final performance of FastTD3, FastSAC, and PPO across continuous- and discrete-control benchmarks at no meaningful runtime cost.

Reference graph

Works this paper leans on

75 extracted references · 29 canonical work pages · cited by 3 Pith papers

  1. [1]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34: 0 29304--29320, 2021

    2021

  2. [2]

    Locomujoco: A comprehensive imitation learning benchmark for locomotion

    Firas Al-Hafez, Guoping Zhao, Jan Peters, and Davide Tateo. Locomujoco: A comprehensive imitation learning benchmark for locomotion. In 6th Robot Learning Workshop, NeurIPS, 2023

    work page 2023

  3. [3]

    Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger

    Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel W \"u thrich, Stefan Bauer, Ankur Handa, and Animesh Garg. Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger. arXiv preprint arXiv:2108.09779, 2021

    work page Pith review arXiv 2021

  4. [4]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  5. [5]

    Reinforcement learning through asynchronous advantage actor-critic on a gpu

    Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on a gpu. arXiv preprint arXiv:1611.06256, 2016

    arXiv 2016

  6. [6]

    Jumanji: a diverse suite of scalable reinforcement learning environments in jax

    Cl \'e ment Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I Midgley, Elshadai Tegegn, Tristan Kalloniatis, et al. Jumanji: a diverse suite of scalable reinforcement learning environments in jax. arXiv preprint arXiv:2306.09884, 2023

    arXiv 2023

  7. [7]

    Daxbench: Benchmarking deformable object manipulation with differentiable physics

    Siwei Chen, Yiqing Xu, Cunjun Yu, Linfeng Li, Xiao Ma, Zhongwen Xu, and David Hsu. Daxbench: Benchmarking deformable object manipulation with differentiable physics. arXiv preprint arXiv:2210.13066, 2022

    arXiv 2022

  8. [8]

    Extreme parkour with legged robots

    Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 11443--11450. IEEE, 2024

    work page 2024

Show all 75 references
  1. [9]

    Leveraging procedural generation to benchmark reinforcement learning

    Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp.\ 2048--2056. PMLR, 2020

    2020

  2. [10]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022

    2022

  3. [11]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning,...

    2018

  4. [12]

    Franka emika panda robot, 2017

    Franka Robotics . Franka emika panda robot, 2017. URL https://www.franka.de. Accessed: 2025-02-17

    2017

  5. [13]

    Brax--a differentiable physics engine for large scale rigid body simulation

    C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax--a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281, 2021

    2021 arXiv

  6. [14]

    Deep whole-body control: learning a unified policy for manipulation and locomotion

    Zipeng Fu, Xuxin Cheng, and Deepak Pathak. Deep whole-body control: learning a unified policy for manipulation and locomotion. In Conference on Robot Learning, pp.\ 138--149. PMLR, 2023

    2023

  7. [15]

    Simplifying deep temporal difference learning, 2024

    Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning, 2024. URL https://arxiv.org/abs/2407.04811

    2024 arXiv

  8. [16]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv.org/abs/1801.01290

    2018 arXiv

  9. [17]

    Multi-task reinforcement learning with mixture of orthogonal experts, 2024

    Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts, 2024. URL https://arxiv.org/abs/2311.11385

    2024 arXiv

  10. [18]

    Multi-task deep reinforcement learning with popart

    Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado Van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.\ 3796--3803, 2019

    2019

  11. [19]

    Distributed prioritized experience replay, 2018

    Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay, 2018. URL https://arxiv.org/abs/1803.00933

    2018 arXiv

  12. [20]

    Learning agile and dynamic motor skills for legged robots

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4 0 (26): 0 eaau5872, 2019

    2019

  13. [21]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. URL https://arxiv.org/abs/1502.03167

    2015 arXiv

  14. [22]

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020

    2020

  15. [23]

    A survey of zero-shot generalisation in deep reinforcement learning

    Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rockt \"a schel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76: 0 201--264, 2023

    2023

  16. [24]

    Pgx: Hardware-accelerated parallel game simulators for reinforcement learning

    Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 45716--45743, 2023

    2023

  17. [25]

    gymnax : A JAX -based reinforcement learning environment library, 2022

    Robert Tjarko Lange. gymnax : A JAX -based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax

    2022

  18. [26]

    Learning quadrupedal locomotion over challenging terrain

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5 0 (47): 0 eabc5986, 2020

    2020

  19. [27]

    Parallel q -learning: Scaling off-policy reinforcement learning under massively parallel simulation

    Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel q -learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning. PMLR, 2023

    2023

  20. [28]

    Rllib: Abstractions for distributed reinforcement learning

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pp.\ 3053--3062. PMLR, 2018 a

    2018

  21. [29]

    Gpu-accelerated robotic simulation for distributed reinforcement learning

    Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, and Dieter Fox. Gpu-accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, pp.\ 270--282. PMLR, 2018 b

    2018

  22. [30]

    Eurekaverse: Environment curriculum generation via large language models, 2024

    William Liang, Sam Wang, Hung-Ju Wang, Osbert Bastani, Dinesh Jayaraman, and Yecheng Jason Ma. Eurekaverse: Environment curriculum generation via large language models, 2024. URL https://arxiv.org/abs/2411.01775

    2024 arXiv

  23. [31]

    Famo: Fast adaptive multitask optimization, 2023 a

    Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization, 2023 a . URL https://arxiv.org/abs/2306.03792

    2023 arXiv

  24. [32]

    Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023 b

    Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023 b . URL https://arxiv.org/abs/2306.03310

    2023 arXiv

  25. [33]

    Conflict-averse gradient descent for multi-task learning, 2024

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning, 2024. URL https://arxiv.org/abs/2110.14048

    2024 arXiv

  26. [34]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 10895--10904, 2023

    2023

  27. [35]

    rl-games: A high-performance framework for reinforcement learning

    Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for reinforcement learning. https://github.com/Denys88/rl_games, May 2021

    2021

  28. [36]

    Isaac gym: High performance gpu-based physics simulation for robot learning, 2021

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021. URL https://arxiv.org/a...

    2021 arXiv

  29. [37]

    Rapid locomotion via reinforcement learning

    Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43 0 (4): 0 572--587, 2024

    2024

  30. [38]

    Craftax: A lightning-fast benchmark for open-ended reinforcement learning, 2024

    Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning, 2024. URL https://arxiv.org/abs/2402.16801

    2024 arXiv

  31. [39]

    Orbit: A unified simulation framework for interactive robot learning environments

    Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learni...

    2023

  32. [40]

    Playing atari with deep reinforcement learning, 2013

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https://arxiv.org/abs/1312.5602

    2013 arXiv

  33. [41]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wi...

    2015

  34. [42]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PmLR, 2016

    1928

  35. [43]

    Popgym: Benchmarking partially observable reinforcement learning

    Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. Popgym: Benchmarking partially observable reinforcement learning. arXiv preprint arXiv:2303.01859, 2023

    2023 arXiv

  36. [44]

    Massively parallel methods for deep reinforcement learning

    Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296, 2015

    2015 arXiv

  37. [45]

    Learning dexterous in-hand manipulation, 2019

    OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in...

    2019 arXiv

  38. [46]

    Ogbench: Benchmarking offline goal-conditioned rl

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092, 2024

    2024 arXiv

  39. [47]

    Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning

    Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning, pp.\ 7652--7662. PMLR, 2020

    2020

  40. [48]

    Learning to push by grasping: Using multiple tasks for effective learning, 2016

    Lerrel Pinto and Abhinav Gupta. Learning to push by grasping: Using multiple tasks for effective learning, 2016. URL https://arxiv.org/abs/1609.09025

    2016 arXiv

  41. [49]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp.\ 91--100. PMLR, 2022

    2022

  42. [50]

    Jaxmarl: Multi-agent rl environments and algorithms in jax, 2024

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whit...

    2024 arXiv

  43. [51]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  44. [52]

    Solving continuous control via q-learning, 2023

    Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via q-learning, 2023. URL https://arxiv.org/abs/2210.12566

    2023 arXiv

  45. [53]

    Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation

    Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  46. [54]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  47. [55]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 0 (7587): 0 484-...

    2016

  48. [56]

    Sapg: Split and aggregate policy gradients

    Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: Split and aggregate policy gradients. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), Proceedings of Machine Learning Research, Vienna, Austria, July 2024. PMLR

    2024

  49. [57]

    Mtrl - multi task rl algorithms

    Shagun Sodhani and Amy Zhang. Mtrl - multi task rl algorithms. Github, 2021. URL https://github.com/facebookresearch/mtrl

    2021

  50. [58]

    Multi-task reinforcement learning with context-based representations

    Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:231879645

    2021

  51. [59]

    Paco: Parameter-compositional multi-task reinforcement learning

    Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-compositional multi-task reinforcement learning. ArXiv, abs/2210.11653, 2022. URL https://api.semanticscholar.org/CorpusID:253080666

    2022 arXiv

  52. [60]

    Leibo, Karl Tuyls, and Thore Graepel

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning, 2017. URL https://arxiv.org...

    2017 arXiv

  53. [61]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  54. [62]

    Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai

    Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized...

    arXiv

  55. [63]

    Deepmind control suite, 2018

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018. URL https://arxiv.org/abs/1801.00690

    2018 arXiv

  56. [64]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012. doi:10.1109/IROS.2012.6386109

    2012

  57. [65]

    Go1 User Manual

    Unitree Robotics . Go1 User Manual. Unitree Robotics, 2021. Available at https://www.unitree.com/go1

    2021

  58. [66]

    Dueling network architectures for deep reinforcement learning, 2016

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning, 2016. URL https://arxiv.org/abs/1511.06581

    2016 arXiv

  59. [67]

    Outracing champion gran turismo drivers with deep reinforcement learning

    Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602 0 (7896): 0 22...

    2022

  60. [68]

    Stabilizing reinforcement learning in differentiable multiphysics simulation

    Eliot Xing, Vernon Luk, and Jean Oh. Stabilizing reinforcement learning in differentiable multiphysics simulation. arXiv preprint arXiv:2412.12089, 2024

    2024 arXiv

  61. [69]

    Multi-task reinforcement learning with soft modularization, 2020

    Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization, 2020. URL https://arxiv.org/abs/2003.13661

    2020 arXiv

  62. [70]

    Gradient surgery for multi-task learning, 2020

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020. URL https://arxiv.org/abs/2001.06782

    2020 arXiv

  63. [71]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URL https://arxiv.org/abs/1910.10897

    2021 arXiv

  64. [72]

    Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel

    Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel. Mujoco playground: An open-source framework for gpu-accelerated robot le...

    2025

  65. [73]

    robosuite: A modular simulation framework and benchmark for robot learning

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart \' n-Mart \' n, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293, 2020

    2009 arXiv

  66. [74]

    Robot parkour learning

    Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. arXiv preprint arXiv:2309.05665, 2023

    2023 arXiv

  67. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.