Pith. sign in

REVIEW 4 major objections 6 minor 59 references

Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deep reinforcement learning scales predictably when data, network capacity, and training budget are grown together.

desk verdict Useful organizing survey with a load-bearing but unsourced scaling-law equation and fixable citation errors; worth reviewing after revision. read the letter →

arxiv 2508.03194 v1 pith:JQKV6JG2 submitted 2025-08-05 cs.LG

Yi Ma , Hongyao Tang , Chenjun Xiao , Yaodong Yang , Wei Wei , Jianye Hao , Jiye Liang This is my paper
classification cs.LG
keywords scalinglawsdeepreinforcementlearningdatanetworktrainingbudgetreplayratiosampleefficiencysurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that the scaling laws driving progress in supervised deep learning can be brought to deep reinforcement learning, and that doing so requires thinking about three levers at once: more and better data, larger and more expressive networks, and smarter allocation of the training budget. It organizes the field into data scaling (parallel collection and synthetic generation), network scaling (width and depth growth, ensembles, mixtures of experts, and agent populations), and training budget scaling (distributed training, replay ratios, batch sizes, and auxiliary tasks). The paper's working hypothesis is that scaling these three dimensions in a coordinated way can relieve DRL's long-standing bottlenecks of sample inefficiency, reward sparsity, and unstable training. A sympathetic reader takes from this survey a roadmap: DRL scaling is not a single knob but a set of interacting levers whose joint design determines whether larger models and datasets actually help.

What carries the argument

The organizing device is the three-axis taxonomy of scaling (data, network, training budget), together with the additive power-law model $P(N,D,C) = \alpha N^{-\beta} + \gamma D^{-\delta} + \epsilon C^{-\zeta} + L_0$ borrowed from supervised learning. The survey maps each phase of the RL training pipeline onto one of the three axes: environment interaction feeds data scaling, policy and value network initialization and architecture feed network scaling, and the update loop feeds training budget scaling through replay ratios (the number of gradient updates per collected experience), batch sizes, distributed gradient computation, and auxiliary losses. This mapping is what lets the survey treat scale as a design variable in RL rather than a fixed hardware constraint.

What would settle it

A factorial study on a fixed benchmark set that varies data volume, network width, and replay ratio independently and fits the additive power-law model would settle the matter: if two-way interactions are large enough that the additive form systematically mispredicts performance, or if a fourth resource such as test-time inference compute produces gains that cannot be absorbed into any of the three axes, the survey's central organizing assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that DRL is entering a regime in which performance improves predictably with scale, just as in vision and language, but the mapping is more complex because data, network capacity, and compute interact through the agent's own behavior. The proposed taxonomy has three axes, and the survey maps representative algorithms onto it: data scaling via parallel sampling and generative augmentation, network scaling via monolithic expansion, ensembles and mixtures of experts, and agent-number scaling, and training budget scaling via distributed training, replay ratio, batch size, and auxiliary objectives. The evidence is read as showing that width scaling with normalization is the most reliable architectural lever, that high replay ratios work when paired with bias suppression and plasticity preservation, and that synthetic data can substitute for expensive real interactions when generation is filtered by value or dynamics consistency. The paper does not claim to have measured a single universal scaling law for RL; it claims that the components of such a law are now visible across the literature and that their interdependencies are the main open problem.

Load-bearing premise

The survey's three-way split of scaling into data, network, and training budget is assumed to be the right and complete way to organize the field, even though the paper admits these axes interact in ways that are poorly understood.

Editorial extensions

If this is right

  • If the survey's synthesis is correct, the default recipe for pushing DRL performance is not just bigger models but the joint scaling of data throughput and update-to-data ratio, with architectural normalization to keep training stable.
  • High replay ratios cease to be a trick and become a first-class scaling lever, but only when coupled with ensembles, normalization, or periodic resets that control primacy bias and plasticity loss.
  • Synthetic data becomes a legitimate substitute for real environment interactions in domains like robotics and offline RL, provided generation is filtered by value or dynamics consistency.
  • The same scaling logic already operates in LLM post-training, where model size, data quality, rollout length, batch size, and test-time compute are being scaled in coordination.
  • Scaling choices cannot be made axis by axis; the paper predicts that ignoring cross-axis interactions, such as critic width versus replay ratio, will lead to overfitting or wasted compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would read the survey as evidence that the field needs a compute-efficient frontier for RL analogous to loss-versus-FLOP curves in language modeling, where the x-axis is total compute and the y-axis is policy return; the paper stops short of proposing such a metric, but its synthesis makes it the natural next step.
  • A testable extension suggested by the paper's open problems is a factorial study varying data volume, network width, and replay ratio together on a fixed benchmark suite to measure interaction terms; if those interactions are large, the additive power-law model needs a coupled or multiplicative term.
  • The survey's discussion of LLM post-training hints that test-time scaling, including longer rollouts and multi-agent verification, may be a fourth dimension that the three-axis taxonomy does not fully absorb; if test-time compute keeps improving results independently of training-time scaling, the taxonomy may need revision.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This survey proposes a three-axis taxonomy for scaling deep reinforcement learning (data, network, and training budget), surveys representative methods within each axis, and connects the discussion to scaling laws in supervised learning and to recent RL-based LLM post-training. The central formal claim is Equation (5), an additive separable multi-resource power law P(N,D,C) = αN^{-β} + γD^{-δ} + εC^{-ζ} + L0, and the conclusion asserts that the systematic application of scaling laws is a transformative force in DRL. The paper also identifies open problems, particularly the poorly understood interdependencies among the three scaling dimensions.

Significance. If the formalization in Section 2.3 were sound, the paper would provide a principled framework for organizing the DRL scaling literature and for guiding resource-aware algorithm design. Even without that formalization, the survey is useful as a broad, well-structured map of methods in distributed data collection, synthetic experience replay, network width and depth scaling, ensembles, evolutionary RL, replay ratios, batch sizes, and auxiliary objectives. The inclusion of LLM post-training as a scaling application is a valuable contemporary extension, and the explicit enumeration of open problems in Section 7.3 is a strength. The paper is not an empirical contribution and does not derive new scaling laws; its value is bibliographic synthesis and hypothesis generation.

major comments (4)
  1. [Section 2.3, Eq. (5)] Equation (5) is presented as the formal basis for the survey's three-axis taxonomy, but it is neither derived nor cited. The cited scaling-law sources do not support an independent additive compute term: Kaplan et al. (2020) fit L(N,D) approximately as (N_c/N)^{α_N} + (D_c/D)^{α_D} + L0, and Hoffmann et al. (2022) fit a coupled form L(N,D) = E + A/N^α + B/D^β under compute-optimal training with C ≈ 6ND. In both cases, compute is not an independent resource whose effect enters additively. The εC^{-ζ} term is therefore not identifiable from the standard data-generation model and is inconsistent with the coupled scaling behavior in the cited literature. Since the conclusion in Section 8 that scaling laws offer 'principled pathways' relies on this equation, the paper must either provide a citation and derivation for Eq. (5) or explicitly demote it to a heuristic illustration, with the central claims adjusted accordingly.
  2. [Section 7.3 vs. Section 2.3] The paper contains a load-bearing internal inconsistency. Section 2.3 states that the separable formulation of Eq. (5) implies that each resource contributes independently to performance improvement, while Section 7.3 states that 'Most existing works treat data, network, and training budget scaling as independent axes, but their interdependencies are poorly understood.' These two statements cannot both be asserted without qualification. If the three axes are independent by assumption, the survey's taxonomy is a useful organizing device but not a demonstrated scientific claim; if they are interdependent, Eq. (5) is contradicted. The authors should state explicitly that the separability is an assumption to be tested and temper the formal language in Sections 2.3 and 8 accordingly.
  3. [Table 3 and Section 6.2] There is a concrete citation error in the replay-ratio comparison table. Table 3 lists 'Simba[Nauman et al., 2024]', but the Nauman et al. (2024) reference is BRO, not SimBa; the text in Section 6.2 correctly attributes SimBa to Lee et al. (2024), and Table 1 also lists 'Simba[Lee et al., 2024]'. Since Table 3 is the evidence table for the replay-ratio scaling section, this misattribution must be corrected and the table regenerated to ensure all entries correspond to the correct references.
  4. [Sections 1 and 2.3, References] The paper cites Marafioti et al. (2025) as one of the four foundational scaling-law references in the opening of Sections 1 and 2.3. The cited work, SmolVLM, is a small multimodal model recipe, not a scaling-law study. Including it among Kaplan et al. (2020), Henighan et al. (2020), and Hoffmann et al. (2022) is misleading about the established basis of scaling laws and weakens the survey's foundational premise. The citation should be replaced with an actual scaling-law reference or removed.
minor comments (6)
  1. [Section 3, Algorithm 1] The text refers to 'The modified RL algorithm in Algorithm 3', but the displayed pseudocode is labeled Algorithm 1; the cross-reference should be corrected.
  2. [References] The reference 'Sutton and Barto, 1988' is dated incorrectly: the cited book 'Reinforcement Learning: An Introduction' was published in 1998, not 1988.
  3. [Table 2] The use of '/' in the Critic Ensemble and Policy Ensemble columns is ambiguous; for example, a reader cannot tell whether '/' means 'not applicable' or 'no entry'. Use explicit check marks or written entries such as 'No' and 'Yes' for clarity.
  4. [Figure 2] The caption 'APE-X PQN SAPG' lists the methods, but the figure itself does not visually distinguish the three techniques; adding labeled panels or direct annotations would improve readability.
  5. [Section 1] The phrase 'a novel taxonomy for the first time' is an overclaim; the paper does not establish that no prior scaling taxonomy exists, and Section 7.3 partially undercuts the taxonomy's independence. Suggest rewording to 'a three-axis taxonomy'.
  6. [Section 7.2] The discussion of LLM RL post-training relies heavily on non-archival industry technical reports and blog posts (OpenAI, Kimi, DeepSeek, Qwen). The paper should state access dates or version identifiers, and ideally separate peer-reviewed results from technical reports.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey is descriptive, its taxonomy is an organizing choice, self-citations are illustrative, and Eq. (5) is an unsupported formalization rather than a circular derivation.

full rationale

This is a survey and organizes existing results; it fits no parameters and makes no quantitative predictions that could reduce to its own inputs. The three-axis taxonomy (data, network, training budget) is an organizing scheme, and the cited works are used descriptively in tables and literature summaries. The self-citations that appear (ERL-Re2, EvoRainbow, CHAIN, HIPODE, ReMix, MARR, DEMAR, and related works by the authors) are used as examples of the surveyed categories; none is load-bearing for the survey's central claim that scaling laws are a promising direction in DRL, which rests on external empirical results (Kaplan et al., Hoffmann et al., Hilton et al., Rybkin et al.). The formalization in Eq. (5), P(N,D,C) = αN^-β + γD^-δ + εC^-ζ + L0, is presented without citation and is not the form fitted by the cited scaling-law papers, which use coupled L(N,D) expressions rather than an additive independent compute term. However, an unsupported or inaccurate formalization is a correctness risk, not a circularity: the equation is not fitted to a subset of data and then used to predict that same data, nor is it justified by a self-citation chain. Section 7.3 itself concedes that interdependencies between the three scaling axes are poorly understood, which weakens the strength of the survey's framing but does not make the argument circular. No step in the paper reduces to its own inputs by construction, so no circularity is flagged.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey's central content rests on the fidelity of its representation of cited works and on its own organizing taxonomy. It does not introduce fitted parameters or new entities, but it does assume that the three chosen scaling dimensions are the right and sufficiently complete categories, and it presents an unproven separable scaling formula in Eq. (5).

assumptions (3)
  • ad hoc to paper The three-way taxonomy (data, network, training budget) is the right organizing principle for DRL scaling literature.
    The survey imposes this division in Section 1 and Figure 1; it is a conceptual choice, not proven. Section 7.3 even acknowledges the dimensions are not independent.
  • domain assumption Empirical results reported in cited works are accurately represented.
    The survey's conclusions rest on the fidelity of its summarization of dozens of papers. Citation errors, such as SimBa attributed to Nauman et al. 2024, undermine this assumption.
  • ad hoc to paper Equation (5) is a valid formalization of scaling behavior.
    Section 2.3 presents the additive separable scaling law without derivation or direct citation to a specific source; it appears to be the authors' construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies." pith.science (2026). https://pith.science/paper/JQKV6JG2

@misc{pith2026250803194,
  author       = {Pith},
  title        = {Pith review of: Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQKV6JG2}},
  note         = {Machine review of arXiv:2508.03194}
}
read the original abstract

In recent years, the expansion of neural network models and training data has driven remarkable progress in deep learning, particularly in computer vision and natural language processing. This advancement is underpinned by the concept of Scaling Laws, which demonstrates that scaling model parameters and training data enhances learning performance. While these fields have witnessed breakthroughs, such as the development of large language models like GPT-4 and advanced vision models like Midjourney, the application of scaling laws in deep reinforcement learning (DRL) remains relatively unexplored. Despite its potential to improve performance, the integration of scaling laws into DRL for decision making has not been fully realized. This review addresses this gap by systematically analyzing scaling strategies in three dimensions: data, network, and training budget. In data scaling, we explore methods to optimize data efficiency through parallel sampling and data generation, examining the relationship between data volume and learning outcomes. For network scaling, we investigate architectural enhancements, including monolithic expansions, ensemble and MoE methods, and agent number scaling techniques, which collectively enhance model expressivity while posing unique computational challenges. Lastly, in training budget scaling, we evaluate the impact of distributed training, high replay ratios, large batch sizes, and auxiliary training on training efficiency and convergence. By synthesizing these strategies, this review not only highlights their synergistic roles in advancing DRL for decision making but also provides a roadmap for future research. We emphasize the importance of balancing scalability with computational efficiency and outline promising directions for leveraging scaling to unlock the full potential of DRL in various tasks such as robot control, autonomous driving and LLM training.

Figures

Figures reproduced from arXiv: 2508.03194 by the authors.

Figure 1
Figure 1. Illustration of our taxonomy of the current literature on Scaling DRL. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of different parallel data collecting techniques. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Illustration of different network scaling techniques. The dotted lines indicate that the components are [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of different network ensemble techniques.In ensemble-based methods, various strategies exist [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Illustration of Evolutionary Reinforcement Learning (ERL) framework. The framework consists of: an [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 14 canonical work pages

  1. [1]

    Reducing overestimation bias in multi-agent domains using double centralized critics

    Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint arXiv:1910.01465,

    arXiv 1910

  2. [8]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437,

    arXiv

  3. [9]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    arXiv 2004

  4. [11]

    Girshick

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9726–9735. Computer Vision Foundation / IEEE,

    work page 2020

  5. [12]

    Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701,

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Pra- fulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701,

    arXiv 2010

  6. [15]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    arXiv 2001

  7. [16]

    Kimi k2: Open agentic intelligence

    Kimi-Team. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025a. Kimi-Team. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025b. Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR,

    arXiv

  8. [18]

    Simba: Simplicity bias for scaling up parameters in deep reinforcement learning

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754,

    arXiv

Show all 59 references
  1. [19]

    Hyperspherical normalization for scalable deep reinforcement learning

    Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. arXiv preprint arXiv:2502.15280,

    arXiv

  2. [21]

    Hipode: Enhancing offline reinforcement learning with high-quality synthetic data from a policy-decoupled approach

    Shixi Lian, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Hipode: Enhancing offline reinforcement learning with high-quality synthetic data from a policy-decoupled approach. arXiv preprint arXiv:2306.06329,

    arXiv

  3. [22]

    Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model

    Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprint arXiv:2507.06892,

    arXiv

  4. [23]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...

    2024

  5. [24]

    doi: 10.18653/v1/2024.emnlp-main.992

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.992. URL https://aclanthology.org/2024.emnlp-main.992/. Shalev Lifshitz, Sheila A. McIlraith, and Yilun Du. Multi-agent verification: Scaling test-time compute with multiple verifiers. arXiv preprint a...

    2024 arXiv

  6. [25]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971,

    arXiv

  7. [26]

    Scaling up rl: Unlocking diverse reasoning in llms via prolonged training

    Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, Gerald Shen, David Mosallanezhad, Di Zhang, Jonas Yang, June Yang, Oleksii Kuchaiev, Guilin Liu, Zhiding Yu, Pavlo Molchanov, Yejin Choi...

    arXiv

  8. [27]

    Y. Ma, T. Liu, B. Wei, Y. Liu, K. Xu, and W. Li. Evolutionary action selection for gradient-based policy learning. arXiv preprint arXiv:2201.04286,

    arXiv

  9. [28]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470,

    arXiv

  10. [29]

    Smolvlm: Redefining small and efficient multimodal models

    Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvlm: Redefi...

    arXiv

  11. [30]

    The impact of on-policy parallelized data collection on deep reinforcement learning networks

    Walter Mayor, Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro. The impact of on-policy parallelized data collection on deep reinforcement learning networks. arXiv preprint arXiv:2506.03404,

    arXiv

  12. [31]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,

    arXiv

  13. [33]

    Candès, and Tatsunori Hashimoto

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

    arXiv

  14. [34]

    Bigger, regularized, categorical: High-capacity value functions are efficient multi-task learners

    Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regularized, categorical: High-capacity value functions are efficient multi-task learners. arXiv preprint arXiv:2505.23150,

    arXiv

  15. [35]

    Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size

    30 Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, and Sergey Kolesnikov. Q-ensemble for offline rl: Don’t scale the ensemble, scale the batch size. arXiv preprint arXiv:2211.11092,

    arXiv

  16. [36]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  17. [37]

    Scaling off-policy reinforcement learning with batch and weight normalization

    Daniel Palenicek, Florian Vogt, and Jan Peters. Scaling off-policy reinforcement learning with batch and weight normalization. arXiv preprint arXiv:2502.07523,

    arXiv

  18. [38]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442,

    arXiv

  19. [39]

    Horizon reduction makes rl scalable

    Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. arXiv preprint arXiv:2506.04168,

  20. [40]

    Qwen3 technical report

    Qwen-Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388,

    arXiv

  21. [41]

    Value-based deep rl scales predictably

    Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep rl scales predictably. arXiv preprint arXiv:2502.04327,

    arXiv

  22. [42]

    Proximal policy optimization algorithms

    31 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  23. [43]

    Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control

    Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505.22642,

    arXiv

  24. [44]

    Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation

    Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506,

    arXiv

  25. [45]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  26. [46]

    Dns: Determinantal point process based neural network sampler for ensemble reinforcement learning

    Hassam Sheikh, Kizza Frisbee, and Mariano Phielipp. Dns: Determinantal point process based neural network sampler for ensemble reinforcement learning. In ICML, pages 19731–19746. PMLR, 2022a. Hassam Sheikh, Mariano Phielipp, and Ladislau Boloni. Maximizing ensemble diversity i...

    1903 arXiv

  27. [47]

    Accelerated methods for deep reinforcement learning.arXiv preprint arXiv:1803.02811,

    Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning.arXiv preprint arXiv:1803.02811,

    arXiv

  28. [48]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018a. 32 Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe...

    2012 arXiv

  29. [50]

    Mad-td: Model- augmented data stabilizes high update ratio rl

    Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir-massoud Farahmand, and Igor Gilitschenski. Mad-td: Model- augmented data stabilizes high update ratio rl. arXiv preprint arXiv:2410.08896,

    arXiv

  30. [51]

    1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities

    Kevin Wang, Ishaan Javali, MichaĹ Bortkiewicz, Benjamin Eysenbach, et al. 1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities. arXiv preprint arXiv:2503.14858,

  31. [52]

    Prioritized generative replay

    Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. arXiv preprint arXiv:2410.18082, 2024a. Yuhui Wang, Qingyuan Wu, Weida Li, Dylan R Ashley, Francesco Faccio, Chao Huang, and Jürgen Schmidhuber. Scaling value iteration n...

    arXiv

  32. [53]

    Aggressive q-learning with ensembles: achieving both high sample efficiency and high asymptotic performance

    Yanqiu Wu, Xinyue Chen, Che Wang, Yiming Zhang, and Keith W Ross. Aggressive q-learning with ensembles: achieving both high sample efficiency and high asymptotic performance. arXiv preprint arXiv:2111.09159,

    arXiv

  33. [54]

    Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning

    Linjie Xu, Zichuan Liu, Alexander Dockhorn, Diego Perez-Liebana, Jinyu Wang, Lei Song, and Jiang Bian. Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning. In 2024 IEEE Conference on Games (CoG), pages 1–8, Milan, Italy, August

    2024

  34. [55]

    ISBN 979-8-3503-5067-8

    IEEE. ISBN 979-8-3503-5067-8. doi: 10.1109/CoG60054.2024.10645658. URL https://ieeexplore.ieee.org/document/10645658/. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. arXiv preprint arXiv:25...

    2024

  35. [56]

    ISBN 979-8-4007-1426-9

    International Foundation for Autonomous Agents and Multiagent Systems. ISBN 979-8-4007-1426-9. event-place: Detroit, MI, USA. 33 Zhengyu Yang, Kan Ren, Xufang Luo, Minghuan Liu, Weiqing Liu, Jiang Bian, Weinan Zhang, and Dongsheng Li. Towards applicable reinforcement learning:...

    arXiv

  36. [57]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021a. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-fr...

    2005 arXiv

  37. [58]

    doi: 10.3233/FAIA230609

    ISBN 978-1-64368-436-9 978-1-64368-437-6. doi: 10.3233/FAIA230609. URL https://ebooks.iospress.nl/ doi/10.3233/FAIA230609. Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without ...

    doi

  38. [59]

    Towards A unified policy abstraction theory and representation learning approach in markov decision processes

    Min Zhang, Hongyao Tang, Jianye Hao, and Yan Zheng. Towards A unified policy abstraction theory and representation learning approach in markov decision processes. arXiv preprint arXiv:2209.07696,

    arXiv

  39. [1989]

    Reinforcement learning with augmented data

    Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In NeurIPS, volume 33, 2020a. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised representations for reinforc...

    arXiv

  40. [2012]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

    arXiv

  41. [2015]

    Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

    Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, volume 48, pages 1928–1937,

    1928

  42. [2016]

    Myosuite–a contact-rich simulation suite for musculoskeletal motor control

    Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite–a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600,

    arXiv

  43. [2017]

    Scaling laws for single-agent reinforcement learning

    Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442,

    arXiv

  44. [2018]

    Simplifying deep temporal difference learning

    Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811,

    arXiv

  45. [2019]

    Ucb exploration via q-ensembles

    Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502,

    arXiv

  46. [2020]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540,

    arXiv

  47. [2021]

    Phasic policy gradient

    Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, pages 2020–2027. PMLR,

    2020

  48. [2022]

    Keep various trajectories: promoting exploration of ensemble policies in continuous control

    Chao Li, Chen Gong, Qiang He, and Xinwen Hou. Keep various trajectories: promoting exploration of ensemble policies in continuous control. NeurIPS, 36:5223–5235, 2023a. Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospi...

    arXiv

  49. [2023]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680,

    1912 arXiv

  50. [2024]

    Reproducibility of benchmarked deep reinforcement learning tasks for continuous control

    Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133,

    arXiv

  51. [2025]

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song

    URL https://hkunlp.github.io/blog/2025/Polaris. Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. NeurIPS, 34:7436–7447,

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.