Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

PPO plateaus when its outer-loop step size outgrows update noise, and scaling to 1 million parallel environments prevents it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:40 UTC pith:WD22VE6U

load-bearing objection A useful scaling recipe and a real 1T-transition result, but the 'not capacity/exploration' claim is under-tested and should be softened. the 4 major comments →

arxiv 2603.06009 v2 pith:WD22VE6U submitted 2026-03-06 cs.LG

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments

Michael Beukman , Khimya Khetarpal , Zeyu Zheng , Will Dabney , Jakob Foerster , Michael Dennis , Clare Lyle This is my paper
classification cs.LG
keywords PPOlearning stagnationplateausparallel environmentsstochastic optimizationouter loopData to Divergence RatioKinetix
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that PPO's characteristic performance plateaus in dense-reward domains are not caused by exploration, network capacity, or optimization failures, but by a step-size-versus-noise mismatch in PPO's outer loop. The authors model the outer loop—sampling fresh rollouts and updating the policy—as stochastic optimization, where the 'step size' is the allowed policy deviation (regulated by clipping or the EWMA reference policy) and the 'noise' is how well a sampled batch approximates the true objective. When the outer step is too large relative to the noise, the policy thrashes around a suboptimal solution, much like SGD at high learning rates. The paper shows that increasing the number of parallel environments reduces both step size and noise simultaneously, and demonstrates that scaling PPO to over one million environments sustains monotonic improvement up to one trillion transitions in an open-ended task space.

Core claim

On the paper's terms: PPO's inner-loop minibatch SGD is not the bottleneck; the outer loop is. By abstracting the inner loop away, the authors show that a plateau arises when the regularized policy update is too large compared with the information in the collected rollouts. This predicts—and the experiments confirm—that plateaus can be induced or removed by changing only the outer step size (e.g., the center of mass of the proximal policy, or PPO's clip epsilon), and that larger rollout batches make PPO less susceptible to weak regularization. The central practical discovery is the Data to Divergence Ratio (DDR) heuristic: as the training budget grows, the required DDR grows, and increasing

What carries the argument

The outer-loop-as-stochastic-optimization model. The 'outer step size' is the allowable policy divergence per update (set by regularization strength—epsilon in PPO or the EWMA reference-policy age/COM), and the 'update noise' is the gap between the empirical loss on a sampled rollout batch and the true return objective. They define the Data to Divergence Ratio (DDR)—samples per unit KL divergence from the behavior policy—as the quantity that must grow with training budget; parallelization raises DDR by adding data per update and by making the behavior policy effectively older. The proposed scaling recipe is to keep minibatch size and learning rate fixed and only increase the number of miniba

Load-bearing premise

The headline Kinetix result assumes that the 1M-environment run's deviations from the recommended recipe (1024 minibatches instead of 16384, 16x larger minibatches, 4x learning rate) do not confound the comparison, and that the DDR-vs-budget trend measured on 512 Jax2D locomotion tasks transfers to Kinetix SFL at one trillion transitions.

What would settle it

Run the same SFL Kinetix setup with 1M parallel environments but the recipe's full 16384 minibatches (fixed minibatch size, fixed LR) and compare final solve rate at matching samples; if the 1024-minibatch config matches or beats it, the claim that parallelization alone (rather than the co-scaled learning rate) drives the result is contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Increasing the number of parallel environments is a reliable remedy for PPO plateaus, because it both reduces update noise and lowers the effective outer step size.
  • When scaling parallelization, keeping the inner optimization hyperparameters fixed (minibatch size, learning rate) is more reliable than scaling minibatch size or learning rate; mis-scaling can cause severe degradation.
  • PPO-EWMA and clip-epsilon affect the same underlying mechanism, so changes in one can be counteracted by the other.
  • In the open-ended Kinetix domain with SFL curriculum, prior baselines plateau below 10B interactions, while 1M parallel environments yield monotonic improvement to 1T interactions.
  • Learning-rate reduction alone can avoid plateaus but is too slow in wall-clock; parallelization achieves the same effect faster.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the outer-loop model is right, adaptive outer step-size schemes (e.g., annealing regularization strength based on observed gradient noise) should be able to reproduce the benefits of massive parallelization at much lower hardware cost—a testable engineering target.
  • The DDR-budget relationship suggests a scaling law for on-policy RL: required samples per update should grow roughly with total budget, implying a predictable path for compute allocation.
  • The results imply that standard PPO baselines in open-ended domains may be under-reporting achievable performance; plateaus that look like capacity limits could be step-size artifacts.
  • Applying the same recipe to sparse-reward tasks with hard exploration may not transfer, since the paper's analysis presumes dense-reward smooth landscapes; that boundary is untested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that PPO plateaus in dense-reward, non-exploration-hard settings are often caused by an overly large outer-loop step size relative to the noise in the sampled objective, rather than by exploration failure, capacity loss, or inner-loop optimization challenges. It abstracts PPO's outer loop as stochastic optimization, identifies regularization strength (COM/epsilon) as the step-size control and the number of samples per update as the noise control, and proposes increasing the number of parallel environments as a simple way to reduce both. A scaling recipe is proposed (keep minibatch size and learning rate fixed, increase the number of minibatches), demonstrated in IsaacGym robotics tasks, and then applied to SFL on Kinetix at up to 1M parallel environments, reporting monotonic improvement up to ~1e12 transitions.

Significance. If the causal story holds, the paper identifies a concrete, actionable failure mode of PPO and provides a scalable remedy, which would be valuable for the RL community. The work is also notable for its scale: 1M parallel environments and 1T transitions is a substantial empirical contribution. Strengths include the clean interventions modulating COM/epsilon and minibatch size, the checkpoint-switch experiment showing that plateaus can be entered and exited by changing the outer step size, the clear recipe and its validation in a separate robotics benchmark, and the ablation appendix (learning rate, PPO-EWMA, filtering) that rules out several trivial confounds. The paper also ships many controlled comparisons across seeds. However, the central mechanistic claim is underdetermined by the experiments, and the headline Kinetix result mixes the recommended recipe with a hybrid scaling rule, so the paper's strongest conclusions need additional support or more careful qualification.

major comments (4)
  1. [Abstract and §3, Figures 2–3] The paper repeatedly claims plateaus arise 'not because of known exploration, capacity, or optimisation challenges,' but no experiment directly manipulates exploration or capacity in the plateau regime. The checkpoint-switch experiment in Figure 3 shows that changing COM can rescue a plateau, but it does not compare against, e.g., raising the entropy coefficient or widening the network at the same COM=8 plateau. Without such positive controls, the claim that the mechanism is specifically outer-loop noise/step size rather than a capacity or exploration effect is underdetermined. Please either add these positive controls or soften the claim to 'not solely explained by' those factors.
  2. [§6 and Appendix D, Figure 11] The headline 1M-environment Kinetix result does not actually run the recipe recommended in §5. As stated in Appendix D, the 1M run uses 1024 minibatches, each 16× larger than default, with a 4× larger learning rate. This is precisely the 'Scale LR | Scale Minibatch Size' configuration that Figure 9 shows can be unstable. The pure recipe would call for 16384 minibatches with fixed minibatch size and LR, which is not reported. The improvement in Figure 11 therefore cannot be attributed solely to increased parallelization; it is the result of a hybrid co-scaling rule. The paper should either run the pure recipe at 1M (even for one environment size) or explicitly frame the Kinetix result as arising from a hybrid scaling rule and justify that the instability seen in Figure 9 does not apply here.
  3. [§4.4 and §6, Figure 8] The statement 'As predicted by the shifting optima in Figure 8' is circular. Figure 8 is a scatter plot of DDR versus solve rate computed from the same COM/epoch/batch-size sweeps whose outcomes it is then used to explain. DDR is a descriptive aggregate of those runs, not an independent predictive model. The paper should either fit the DDR–budget relationship on training runs and test it on held-out configurations, or explicitly rephrase this as a post-hoc diagnostic rather than a prediction.
  4. [§7 and §E.3] In Kinetix, increasing parallelization also increases the diversity of tasks seen per update, since each environment is unique. The paper acknowledges Bharthulwar et al.'s diversity result but does not isolate diversity from the proposed noise/step-size mechanism. Section E.3 shows that additional filtering alone is insufficient at fixed parallelization, but that ablation does not manipulate diversity. The 1M result could be largely due to the diversity benefit of 1M unique environments rather than the lower outer step size. An experiment that controls diversity (e.g., reusing a fixed set of environments across all workers, or comparing against a 1M run with duplicated tasks) would be needed to separate these explanations. If this is infeasible, the causal interpretation of the Kinetix results should be explicitly flagged as confounded by diversity.
minor comments (5)
  1. [Abstract and §6, Figure 11] The word 'monotonic' is stronger than what Figure 11 shows; the curves are noisy and truncated. Unless there is a formal monotonicity guarantee, consider 'sustained' or 'near-monotonic improvement.'
  2. [Table 2] The SFL column leaves 'PPO # parallel environments' empty. Please fill in the values used for the baseline and for the 8k/65k/1M runs.
  3. [§4.4] DDR is introduced verbally but never defined by an equation. Please give an explicit formula, including how KL divergence is measured (e.g., behavior policy vs. proximal policy) and how 'unit KL' is computed.
  4. [Appendix D] The choice of 1024 minibatches is justified by wall-clock time, but the text says 'we cannot have more minibatches than we have parallel environments' and then uses 1024 when 8192 are possible. Please clarify why 1024 rather than another number, and note the trade-off more precisely.
  5. [§4.2] The explanation that Adam momentum causes the benefit of many epochs at epsilon=0 is plausible but speculative. A small ablation with Adam disabled or with a different optimizer would make the claim more solid; otherwise, it could be softened.

Circularity Check

0 steps flagged

No significant circularity: central claims rest on direct interventions and out-of-sample Kinetix runs; the DDR trend is correlational, not a fitted prediction.

full rationale

The paper's central claims are empirically grounded rather than definitionally circular. The 'outer loop as stochastic optimization' framing is an analogy, and its components—regularization strength (COM, epsilon), number of samples, epochs, batch size—are treated as manipulable variables. The predictions are tested by direct interventions in Jax2D locomotion tasks (e.g., changing COM during training, varying epsilon, batch size, and epochs), and the Kinetix result is a directly measured out-of-sample outcome, not a value read off from a fitted curve. The DDR trend in Figure 8 is computed from the same Jax2D runs used to display it, so it is correlational; however, the paper does not fit a model to DDR and then derive Kinetix performance from it. The phrase 'As predicted by the shifting optima in Figure 8' is retrospective rhetoric, but the Kinetix runs are independent empirical evidence and could have failed, so no reduction to inputs occurs. Self-citations (Matthews et al. 2025; Rutherford et al. 2024) supply the environment and the SFL baseline, but the paper re-runs comparisons and the load-bearing scaling result is produced in this paper. The acknowledged deviations in Section D (1024 minibatches, 16x larger minibatches, 4x LR) are a potential confound for the recipe, not a circularity. The under-tested exclusion of exploration/capacity explanations is a correctness/completeness concern, not a circular derivation. No step in the claimed derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper is an empirical study: it introduces the DDR metric and the outer/inner-loop vocabulary but no new physical, architectural, or formal entities. The central claim rests on an analogy plus several hand-chosen configurations; no constants are fitted to a closed-form theory.

free parameters (3)
  • Kinetix SFL 1M hybrid minibatch/LR configuration = 1024 minibatches; minibatch 16x default (≈262144); LR 4x default (2e-4)
    Hand-chosen trade-off between performance and wall-clock; explicitly deviates from the recommended fix-minibatch-size recipe.
  • SFL filtering/buffer parameters for 1M run = N=4M, K=8192, T=2
    Adjusted by hand to use additional GPUs; ablation in Section E shows these alone are insufficient, but the 1M result depends on them.
  • Analysis sweeps over COM/epsilon/epochs/minibatch = COM 8–8192; epsilon 0.01–5.0; epochs 1–32; minibatch 4096–98304
    Used to establish the outer-step-size mechanism; selected by grid search rather than derived from a theory.
axioms (5)
  • domain assumption PPO's outer loop can be modeled as stochastic optimization with step size set by regularization and noise set by samples collected between updates.
    Invoked throughout Section 3; no formal derivation from the PPO loss is given.
  • ad hoc to paper Plateaus studied are not caused by exploration, capacity, or inner-loop optimization challenges.
    Stated in the abstract and Section 1, but capacity and exploration are not directly ablated.
  • domain assumption Increasing the number of parallel environments monotonically increases behavior-policy age, reducing outer step size.
    Taken from Hilton et al. (2022) and used to connect parallelization to regularization.
  • domain assumption The DDR-vs-budget relationship from 512 Jax2D locomotion tasks transfers to Kinetix SFL at trillions of transitions.
    Needed for the main scaling recommendation; Figure 8 only reaches 10B interactions.
  • standard math SGD with too-large step size thrashes near a local optimum, and this behavior is analogous to PPO plateaus.
    Classic stochastic-approximation intuition; used as the explanatory backbone of Section 3.

pith-pipeline@v1.3.0-alltime-deepseek · 20015 in / 13521 out tokens · 126909 ms · 2026-08-02T18:40:20.372898+00:00 · methodology

0 comments
read the original abstract

An agent's performance stagnating at a suboptimal level is a common problem in deep on-policy RL. Focusing on PPO, we show that plateaus in certain regimes arise not because of known exploration, capacity, or optimisation challenges, but because sample-based estimates of the loss eventually become poor proxies for the true objective over the course of training. Looking deeper, PPO alternates between sampling rollouts from several parallel environments online using the current policy (which we call the "outer loop") and performing repeated minibatch SGD steps against this offline dataset (the "inner loop"). In our work, we abstract away the inner loop, and conceptually model the outer loop as standard stochastic optimisation. The step size is then controlled by the regularisation strength towards the previous policy and the gradient noise by the number of samples collected between policy update steps. This framing predicts that, much like in SGD, if the outer step size is too large relative to the noise, updates become uninformative and lead to the policy thrashing around a local optimum instead of converging. Recasting PPO in this light makes it clear that there are two ways to address this particular type of learning stagnation: either reduce the step size or increase the number of samples collected between updates. We validate the predictions of our model and conclude that increasing the number of parallel environments is a simple way to avoid these plateaus by simultaneously altering both these factors. Applying our analysis and scaling PPO to more than 1M parallel environments enables monotonic performance improvement up to one trillion transitions and leads to vastly superior performance compared to prior baselines in a complex open-ended domain.

Figures

Figures reproduced from arXiv: 2603.06009 by Clare Lyle, Jakob Foerster, Khimya Khetarpal, Michael Beukman, Michael Dennis, Will Dabney, Zeyu Zheng.

Figure 1
Figure 1. Figure 1: Comparing the behavior in (a) PPO and (b) a simple convex optimization problem with [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) In stochastic optimization, the update magnitude is consistently large when the step [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Loading checkpoints and retraining with a different COM recovers the performance of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Tuning the learning rate cannot counteract a poor outer step size. Here we sweep over [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Weak regularization, corresponding to either (a) too low of a COM or (b) too large of a [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: How changing number of epochs influences (a) PPO-EWMA and (b) normal PPO. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Showing the effect of larger minibatches when changing the (a) COM of [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: DDR vs. the maximum solve rate achieved in that run for various compute budgets. Each [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Comparing different approaches for varying the number of parallel environments. Keeping [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: We take the code from Singla et al. (2024), and make one change—setting the minibatch size to 16k (which is the default in Isaacgym) instead of using 96k like Singla et al. (2024) do. When using our recommendations, vanilla PPO performs much better across the board, and the gap between it and SAPG is reduced. Furthermore, SAPG also benefits from the same change. 6 Batch Size Scaling Enables Open-Ended Lea… view at source ↗
Figure 11
Figure 11. Figure 11: SFL on Kinetix, showing that increasing the number of parallel environments main￾tains performance improvement for much longer. The dashed red line is an approximation of optimal performance, since not all sampled environments are solvable, while the grey line indicates a ran￾dom policy’s performance. We plot mean and 95% CI over 3 seeds. The curves are truncated at different x-values since using fewer pa… view at source ↗
Figure 12
Figure 12. Figure 12: Comparing the effect of COM vs ϵ. (a) Showing the performance of the best ϵ for various COMs, showing that we can find an ϵ to (mostly) counteract the effect of changing the COM in PPO-EWMA. (b) A heatmap of final performance for a 2D grid search over the PPO-EWMA COM and ϵ. Overall, most reasonable values of the COM have a corresponding ϵ that performs well; however, extreme values of ϵ are too unstable … view at source ↗
Figure 13
Figure 13. Figure 13: Plotting the steps per second for the training, which includes the environment step and the neural network optimization. This cor￾responds to the same results as in [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 15
Figure 15. Figure 15: Comparing performance when using 1M parallel environments, but a different number [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Comparing runtime when using different minibatch sizes and 1M parallel environments. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: SFL results when keeping the number of environments fixed at 8192, but changing the [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: SFL results when keeping the number of environments fixed at 8192, but using PPO [PITH_FULL_IMAGE:figures/full_fig_p021_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: SFL results using 8192 environments, but either sampling more levels to filter through, [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: SFL Results on hand-designed environments. While the performance is much more [PITH_FULL_IMAGE:figures/full_fig_p022_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

    cs.LG 2026-04 unverdicted novelty 6.0

    Entrocraft uses rejection sampling to enforce custom entropy curves in LLM RL, sustaining longer training, better generalization, and higher output diversity than prior regularization approaches.

  2. Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

    cs.LG 2026-04 unverdicted novelty 6.0

    Entrocraft uses rejection sampling to enforce precise entropy schedules in LLM RL by biasing advantages, enabling longer training, better generalization, and higher performance than baselines.

Reference graph

Works this paper leans on

69 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    What matters in on-policy reinforcement learning? a large-scale empirical study

    Marcin Andrychowicz, Anton Raichuk, Piotr Sta \'n czyk, Manu Orsini, Sertan Girgin, Raphael Marinier, L \'e onard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990, 2020

    Pith/arXiv arXiv 2006

  3. [3]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    2016

  4. [4]

    Nonlinear programming

    Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48 0 (3): 0 334--334, 1997

    1997

  5. [5]

    Staggered environment resets improve massively parallel on-policy reinforcement learning

    Sid Bharthulwar, Stone Tao, and Hao Su. Staggered environment resets improve massively parallel on-policy reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=hesM5BWtOJ

    2025

  6. [6]

    Towards deeper deep reinforcement learning with spectral normalization

    Nils Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 34: 0 8242--8255, 2021

    2021

  7. [7]

    Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P

    Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos-B...

    Pith/arXiv arXiv 2024

  8. [8]

    Mixtures of experts unlock parameter scaling for deep RL

    Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL . In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=X9VMhfFxwn

    2024

  9. [9]

    Two-timescale networks for nonlinear value function approximation

    Wesley Chung, Somjit Nath, Ajin Joseph, and Martha White. Two-timescale networks for nonlinear value function approximation. In International conference on learning representations, 2018

    2018

  10. [10]

    Bayen, Stuart Russell, Andrew Critch, and Sergey Levine

    Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 985e9a46e10005356bbaf194249f6856-Abstract.html

    2020

  11. [11]

    Revisiting lars for large batch training generalization of neural networks

    Khoi Do, Minh-Duong Nguyen, Nguyen Tien Hoa, Long Tran-Thanh, Nguyen H Tran, and Quoc-Viet Pham. Revisiting lars for large batch training generalization of neural networks. IEEE Transactions on Artificial Intelligence, 6 0 (5): 0 1321--1333, 2024

    2024

  12. [12]

    First return, then explore

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590 0 (7847): 0 580--586, 2021

    2021

  13. [13]

    Implementation matters in deep policy gradients: A case study on ppo and trpo

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020

    Pith/arXiv arXiv 2005

  14. [14]

    Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem

    C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/brax

    2021

  15. [15]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Doll \'a r, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    Pith/arXiv arXiv 2017

  17. [17]

    Learning rates as a function of batch size: A random matrix theory approach to neural network training

    Diego Granziol, Stefan Zohren, and Stephen Roberts. Learning rates as a function of batch size: A random matrix theory approach to neural network training. Journal of Machine Learning Research, 23 0 (173): 0 1--65, 2022

    2022

  18. [18]

    Batch size-invariance for policy optimization

    Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. Advances in Neural Information Processing Systems, 35: 0 17086--17098, 2022

    2022

  19. [19]

    Scaling laws for single-agent reinforcement learning

    Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Position: Open-endedness is essential for artificial superhuman intelligence

    Edward Hughes, Michael D Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rockt \"a schel. Position: Open-endedness is essential for artificial superhuman intelligence. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=Bc4vZ2CX7E

    2024

  21. [21]

    Are deep policy gradient algorithms truly policy gradient algorithms

    Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Are deep policy gradient algorithms truly policy gradient algorithms. arXiv preprint arXiv:1811.02553, 2018

    Pith/arXiv arXiv 2018

  22. [22]

    Prioritized level replay

    Minqi Jiang, Edward Grefenstette, and Tim Rockt \"a schel. Prioritized level replay. In International Conference on Machine Learning, pp.\ 4940--4950. PMLR, 2021

    2021

  23. [23]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  24. [24]

    One weird trick for parallelizing convolutional neural networks

    Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014

    Pith/arXiv arXiv 2014

  25. [25]

    u ttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \

    Heinrich K \"u ttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rockt \"a schel. The nethack learning environment. Advances in Neural Information Processing Systems, 33: 0 7671--7684, 2020

    2020

  26. [26]

    gymnax : A JAX -based reinforcement learning environment library, 2022

    Robert Tjarko Lange. gymnax : A JAX -based reinforcement learning environment library, 2022. URL http://github.com/RobertTLange/gymnax

    2022

  27. [27]

    Wurman, Jaegul Choo, Peter Stone, and Takuma Seno

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=jXLiDKsuDo

    2025

  28. [28]

    Hyperspherical normalization for scalable deep reinforcement learning

    Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502.15280, 2025 b

    Pith/arXiv arXiv 2025

  29. [29]

    Linear and nonlinear programming, volume 2

    David G Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984

    1984

  30. [30]

    Understanding and preventing capacity loss in reinforcement learning

    Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ZkC8wKoLbQ7

    2022

  31. [31]

    Disentangling the causes of plasticity loss in neural networks

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. 274: 0 750--783, 29 Jul--01 Aug 2025. URL https://proceedings.mlr.press/v274/lyle25a.html

    2025

  32. [32]

    Isaac gym: High performance GPU based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance GPU based physics simulation for robot learning. In Joaquin Vanschoren and Sai - Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Tr...

    2021

  33. [33]

    On the sdes and scaling rules for adaptive gradient algorithms

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 0 7697--7711, 2022

    2022

  34. [34]

    Craftax: A lightning-fast benchmark for open-ended reinforcement learning

    Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning. In ICML, 2024

    2024

  35. [35]

    Kinetix: Investigating the training of general agents through open-ended physics-based control tasks

    Michael Matthews, Michael Beukman, Chris Lu, and Jakob Foerster. Kinetix: Investigating the training of general agents through open-ended physics-based control tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2410.23208

    Pith/arXiv arXiv 2025

  36. [36]

    An empirical model of large-batch training

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018

    Pith/arXiv arXiv 2018

  37. [37]

    Multi-task reinforcement learning enables parameter scaling

    Reginald McLean, Evangelos Chatzaroulas, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Multi-task reinforcement learning enables parameter scaling. In Reinforcement Learning Conference, 2025. URL https://openreview.net/forum?id=eBWwBIFV7T

    2025

  38. [38]

    Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mi o \'s , and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. Advances in neural information processing systems, 37: 0 113038--113071, 2024

    2024

  39. [39]

    The primacy bias in deep reinforcement learning

    Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

    2022

  40. [40]

    XL and-minigrid: Scalable meta-reinforcement learning environments in JAX

    Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Viacheslav Sinii, Artem Agarkov, and Sergey Kolesnikov. XL and-minigrid: Scalable meta-reinforcement learning environments in JAX . In Intrinsically-Motivated and Open-Ended Learning Workshop, NeurIPS2023, 2023. URL https://openreview.net/forum?id=xALDC4aHGz

    2023

  41. [41]

    XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning

    Alexander Nikulin, Ilya Zisman, Alexey Zemtsov, Viacheslav Sinii, Vladislav Kurenkov, and Sergey Kolesnikov. Xland-100b: A large-scale multi-task dataset for in-context reinforcement learning. CoRR, abs/2406.08973, 2024. doi:10.48550/ARXIV.2406.08973. URL https://doi.org/10.48550/arXiv.2406.08973

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.08973 2024

  42. [42]

    Numerical optimization

    Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 2006

    2006

  43. [43]

    Training larger networks for deep reinforcement learning

    Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.07920, 2021

    Pith/arXiv arXiv 2021

  44. [44]

    Evolving curricula with regret-based environment design

    Jack Parker - Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rockt \" a schel. Evolving curricula with regret-based environment design. In Proceedings of the International Conference on Machine Learning, pp.\ 17473--17498. PMLR , 2022. URL https://proceedings.mlr.press/v162/parker-holder22a.html

    2022

  45. [45]

    Some methods of speeding up the convergence of iteration methods

    Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4 0 (5): 0 1--17, 1964

    1964

  46. [46]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  47. [47]

    Jaxmarl: Multi-agent rl environments in jax

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint arXiv:2311.10090, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    No regrets: Investigating and improving regret approximations for curriculum discovery

    Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. Advances in Neural Information Processing Systems, 37: 0 16071--16101, 2024

    2024

  49. [49]

    Value-based deep RL scales predictably

    Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Victor Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep RL scales predictably. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=FLPFPYJeVU

    2025

  50. [50]

    Jordan, and Pieter Abbeel

    John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, 2016. URL http://arxiv.org/abs/1506.02438

    Pith/arXiv arXiv 2016

  51. [51]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  52. [52]

    Bigger, better, faster: Human-level atari with human-level efficiency

    Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.\ 30365--30380. PMLR, 2023

    2023

  53. [53]

    Sapg: split and aggregate policy gradients

    Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: split and aggregate policy gradients. arXiv preprint arXiv:2407.20230, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Smith, Pieter-Jan Kindermans, and Quoc V

    Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1Yy1BxCZ

    2018

  55. [55]

    Why open-endedness matters

    Kenneth O Stanley. Why open-endedness matters. Artificial life, 25 0 (3): 0 232--235, 2019

    2019

  56. [56]

    Characterization and mitigation of training instabilities in microscaling formats

    Huangyuan Su, Mujin Kwun, Stephanie Gil, Sham Kakade, and Nikhil Anand. Characterization and mitigation of training instabilities in microscaling formats. arXiv preprint arXiv:2506.20752, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    On bonus-based exploration methods in the arcade learning environment

    Adrien Ali Taiga, William Fedus, Marlos C Machado, Aaron Courville, and Marc G Bellemare. On bonus-based exploration methods in the arcade learning environment. arXiv preprint arXiv:2109.11052, 2021

    Pith/arXiv arXiv 2021

  58. [58]

    Beyond the boundaries of proximal policy optimization

    Charlie B Tan, Edan Toledo, Benjamin Ellis, Jakob N Foerster, and Ferenc Husz \'a r. Beyond the boundaries of proximal policy optimization. arXiv preprint arXiv:2411.00666, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal M. P. Behbahani, Avishkar Bhoopchand, Nathalie Bradley - Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks - Thompson, Hannah Openshaw, Jack Parker - Holder, Shreya Pathak, Nicolas Perez Niev...

    Pith/arXiv arXiv doi:10.48550/arxiv.2301.07608 2023

  60. [60]

    Open-ended learning leads to generally capable agents

    Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Micha \" e l Mathieu, Nat McAleese, Nathalie Bradley - Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes - Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to ...

    Pith/arXiv arXiv 2021

  61. [61]

    Efficient exploration in reinforcement learning

    Sebastian B Thrun. Efficient exploration in reinforcement learning. Carnegie Mellon University, 1992

    1992

  62. [62]

    Stoix: Distributed Single-Agent Reinforcement Learning End-to-End in JAX , April 2024

    Edan Toledo. Stoix: Distributed Single-Agent Reinforcement Learning End-to-End in JAX , April 2024. URL https://github.com/EdanToledo/Stoix

    2024

  63. [63]

    1000 layer networks for self-supervised RL : Scaling depth can enable new goal-reaching capabilities

    Kevin Wang, Ishaan Javali, Micha Bortkiewicz, Tomasz Trzcinski, and Benjamin Eysenbach. 1000 layer networks for self-supervised RL : Scaling depth can enable new goal-reaching capabilities. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=s0JVsx3bx1

    2025

  64. [64]

    Truly proximal policy optimization

    Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp.\ 113--122. PMLR, 22--25 Jul 2020. URL https://proceedings.mlr.press/v115/wang20b.html

    2020

  65. [65]

    A finite-time analysis of two time-scale actor-critic methods

    Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33: 0 17617--17628, 2020

    2020

  66. [66]

    Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel

    Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel. Mujoco playground: An open-source framework for gpu-accelerated robot learning and sim-to-real transfer., 2025. URL https://github.com/google-deepmind/m...

    2025

  67. [67]

    Fast two-time-scale stochastic gradient method with applications in reinforcement learning

    Sihan Zeng and Thinh Doan. Fast two-time-scale stochastic gradient method with applications in reinforcement learning. In The Thirty Seventh Annual Conference on Learning Theory, pp.\ 5166--5212. PMLR, 2024

    2024

  68. [68]

    A two-time-scale stochastic optimization framework with applications in control and reinforcement learning

    Sihan Zeng, Thinh T Doan, and Justin Romberg. A two-time-scale stochastic optimization framework with applications in control and reinforcement learning. SIAM Journal on Optimization, 34 0 (1): 0 946--976, 2024

    2024

  69. [69]

    Stabilizing reinforcement learning with llms: Formulation and practices

    Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, et al. Stabilizing reinforcement learning with llms: Formulation and practices. arXiv preprint arXiv:2512.01374, 2025

    arXiv 2025