Pith. sign in

REVIEW 4 major objections 5 minor 65 references

Is Exploration or Optimization the Problem for Deep Reinforcement Learning?

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deep RL policies leave a 2-3x performance gap on the table.

desk verdict A genuinely useful diagnostic question undermined by an uncalibrated order-statistic baseline; the exploitation-gap claim is not yet established but the paper is worth a serious referee. read the letter →

arxiv 2508.01329 v1 pith:NWYDPC4N submitted 2025-08-02 cs.LG cs.AI

Glen Berseth This is my paper
classification cs.LGcs.AI
keywords deepreinforcementlearningexplorationvsexploitationoptimizationlimitsexperienceoptimalpolicypracticalsub-optimalitynon-iidtrainingperformancediagnosisAtaribenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main bottleneck in deep reinforcement learning is not exploration but exploitation: the policy's failure to reproduce the good behavior it has already generated. It introduces a practical estimator of sub-optimality that compares the return of the learned policy against the best returns in its own experience buffer, called the experience optimal policy. Across Atari, MinAtar, and continuous-control tasks, the best experienced returns are 2-3 times higher than what the trained policy achieves, implying the algorithms capture only about half of the value of the experience they collect. The paper also reports that adding exploration bonuses or scaling up networks widens this gap, which it reads as evidence that optimization limits, not data limits, are what hold deep RL back.

What carries the argument

The machinery is a new estimator of practical sub-optimality anchored on the experience optimal policy $\hat{\pi}^*$, defined as the best policy reachable from the data the agent has already collected. In deterministic settings it is the best single trajectory in the buffer; in stochastic settings it is approximated by the top 5% of returns, computed both over all past experience and over the current replay buffer. The estimator's value is the gap between $\hat{\pi}^*$ and the learned policy $\pi_\theta$, framed through the performance difference lemma. This gap is meant to separate the data problem (does the agent generate high-value experience?) from the model problem (can the optimizer turn that experience into a better policy?).

What would settle it

In a stochastic environment where a fixed optimal policy has high return variance, compute the proposed gap for an agent that already achieves the optimal expected return: if the top-5% minus mean gap remains large, the metric mistakes environmental noise for an optimization failure. Alternatively, take the top 5% of trajectories from a trained agent and behavior-clone the policy on them: if the cloned policy matches those returns, the gap is indeed an optimization problem, and if it does not, the top trajectories carry information the estimator's comparison ignores.

Watch

Extended reading notes

Core claim

The paper's central claim is that deep RL methods are limited mainly by optimization for exploitation rather than by exploration. To support this, it defines the experience optimal policy $\hat{\pi}^*$, whose value is estimated as the average return of the top 5% of trajectories in the agent's replay buffer, either over all past experience or over recent experience. The gap $V^{\hat{\pi}^*}(s_0) - V^{\pi_\theta}(s_0)$ is proposed as a measure of practical sub-optimality, and experiments show this gap is large and persistent across DQN and PPO on difficult environments such as Montezuma's Revenge, while it is small on solved tasks like HalfCheetah. Because the gap survives additional training and grows when RND exploration or larger networks are added, the author concludes that the community should focus more on optimization methods that can reliably turn collected experience into policy improvements.

Load-bearing premise

The estimate assumes that the top 5% of returns is a fair and stable target that a well-optimized policy could actually reach, rather than a set of lucky outliers produced by environment stochasticity.

Editorial extensions

If this is right

  • Researchers evaluating new RL algorithms should report the experience-optimal gap alongside average return, since average return alone cannot distinguish 'found good experience' from 'exploited good experience.'
  • Improvements that look like exploration gains, such as RND, may partly be exploitation losses: the agent finds better trajectories but converts a smaller fraction of them into policy behavior.
  • Scaling network size in deep RL should be expected to worsen optimization-limited behavior, not to fix exploration, so larger networks need accompanying optimization or regularization changes.
  • For practitioners, a large practical sub-optimality on a task is a signal to invest in optimization and stability fixes before investing in more exploration.
  • Aggregate comparison shows that DQN can beat PPO on average return while still leaving more exploitable experience unused, so algorithm rankings change when measured relative to the agent's own best experience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gap is confirmed across a wider set of algorithms and seeds, it suggests that loss of plasticity and primacy bias, rather than insufficient state coverage, are the dominant failure modes in deep RL.
  • A natural testable extension is to use the top-5% trajectories as an automatic curriculum or as targets for imitation, and measure whether the gap closes when the optimizer is given a more learnable objective.
  • The metric could be adapted to provide early stopping or checkpoint selection: an agent whose recent top-5% gap is growing may be better served by resetting parts of the network than by collecting more data.
  • Beyond the paper, the top-5% estimator may mix genuine optimization failure with intrinsic return variance in stochastic domains, so the 2-3x figure should be re-estimated on deterministic or near-deterministic tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a 'practical sub-optimality' metric for deep RL, defined as the difference between the average return of the top 5% of episodes the agent has experienced (called the experience optimal policy \hat{\pi}^*) and the learned policy's average return. The authors argue that a large gap indicates that deep RL agents are limited more by optimization/exploitation than by exploration. They present experiments with DQN and PPO across MinAtar, Atari, and MuJoCo environments, including settings with RND exploration bonuses and different network scales, and an aggregate version of the metric across the Atari-5 environments. The central claim is that RL algorithms only exploit about half of their own best experience.

Significance. If the proposed estimator were valid, it would offer a practical diagnostic to separate exploration and exploitation limitations in deep RL, which is a genuinely useful question for the community. The paper also includes deterministic-environment replays that partially address whether the best trajectories are reproducible. However, the validity of the estimator is not established, and the main empirical conclusion is not supported by the evidence as presented.

major comments (4)
  1. [Section 4.1, Eq. (5)] The metric V̂π* defined as the average return of the top 5% of collected episodes is an order statistic of the return distribution, not the value of any policy. For any stochastic policy with positive return variance—including an optimal policy that perfectly exploits its experience—the average of the top 5% of sampled returns strictly exceeds the policy's expected return. Therefore a positive gap V̂π* − V(πθ) is expected by construction and does not, by itself, demonstrate an optimization failure. The paper provides no calibration experiment, such as evaluating a frozen, fully trained policy with the same top-5% procedure, to show that a well-exploiting agent would yield a ratio near 1.
  2. [Section 5.1, Figures 1b and 3c] The 'completely deterministic' MinAtar experiments reduce one concern—that the best trajectory cannot be replayed—but they do not eliminate the order-statistic bias: DQN still collects experience with epsilon-greedy noise, so the top-5% episodes are drawn from the upper tail of the return distribution under the exploration policy. The claim that V̂π* > V̂π*D∞ > V(πθ) indicates an exploitation failure requires a counterfactual showing what the top-5% statistic would be for a policy that already reproduces its best behavior; without this, the observed gap could be an artifact of stochastic exploration rather than a deficiency of the learned policy.
  3. [Section 5.4, Eq. (6)] The aggregate normalized metric is not controlled across algorithms. Because V̂π* is an order statistic, its expected value increases with the number of episodes collected; PPO (on-policy) and DQN (off-policy) differ in how many returns are available and how they are sampled. The denominator (V̂π* − V^π0) can also be small or negative for tasks where the best experience is close to initial performance, making the normalized ratio unstable. Without controlling for the number of collected episodes and the variance of returns, the cross-algorithm comparison in Figure 6 is not reliable.
  4. [Sections 5.2 and 5.3] The experiments with RND bonuses and larger networks show changes in the raw gap between the top-5% return and the average policy return, but these changes can be explained by the same order-statistic bias: exploration bonuses increase the variance of returns and the number of high-return outliers, while larger networks may change the return distribution as well. The paper does not separate these effects from genuine changes in the policy's ability to exploit its own experience. Furthermore, all results use only 4 seeds and no explicit error bars in most figures, which is insufficient to support the quantitative claims of a 2–3× gap.
minor comments (5)
  1. [Abstract] The statement 'only exploit half of the good experience they generate' is not directly derived from the reported 2–3× gap; the paper should explain the mapping from the measured ratio to the fraction of experience exploited.
  2. [Section 4, Eq. (4)] The notation in Eq. (4) is unclear: the arg max is over <a0,...,at> ∈ D∞ while t appears both as the horizon and as the index; also D∞ is not defined precisely.
  3. [Section 4.1] There is a typo 'stocastic' in 'the best stocastic policy'.
  4. [Section 5.1] The caption of Figure 1b should specify the exact hyperparameters and evaluation protocol for the 'deterministic policy' used in the replay.
  5. [General] The paper would benefit from a discussion of how the proposed metric relates to existing off-policy or offline RL evaluation methods, and from a direct statement of the code and data availability.

Circularity Check

2 steps flagged · score 6.0 of 10

The claimed 2–3× 'exploitation gap' is an order-statistic artifact: V̂π* is defined as the top 5% of the agent's own returns, so V̂π* − V̂πθ is positive by construction for any stochastic policy and does not by itself measure optimization failure.

  1. self definitional [Section 4.1, Eq. (5); Abstract]
    "For stochastic environments, the first version the best stocastic policy from the collected experience as top 5% of experience generated by the agent V ˆπ∗D∞ (s0), where D∞ is all the experience collected by the agent. ... V ˆπ∗ (s0) = 1 k X τ ∈D0:k X at,st∈τ R(at, st) (5) Where k is equal to 1 20 × |D| and D is sorted with the highest value trajectory starting at index 0."

    By construction, V̂π* in Eq. (5) is the mean of the top 5% of the agent's own trajectory returns, while V̂πθ is an estimate of the mean of that same return distribution. For any policy with nonzero return variance—including the optimal stochastic policy, a fully trained PPO policy, or DQN during epsilon-greedy data collection—the top-5% sample average strictly exceeds the expectation, so the positive gap is an order-statistic identity, not evidence of failed exploitation. The paper's own deterministic-policy observation that the gap approaches V̂π* in Breakout confirms that the gap is generated by action-selection stochasticity; no calibration shows a perfectly exploiting policy would yield a ratio near 1.

  2. self definitional [Section 4, paragraph after Eq. (4)]
    "This policy can also be understood as deterministically replaying the highest value sequence of actions < a0, . . . , at > in the experience memory. This policy can be used to compute a new difference as the exploitation sub-optimality of the form V ˆπ∗ (s0) − V ˆπθ (s0)."

    The 'experience optimal policy' is defined as a replayed trajectory or the top-5% subset of returns, not as a stationary policy that can be recovered by optimizing on D. Calling V̂π*(s0) − V̂πθ(s0) the 'exploitation sub-optimality' assumes the learned network should be able to reproduce the best stored trajectory; that is precisely the claim the paper uses the gap to prove. The naming converts a selection-bias artifact into a diagnosis of optimization failure.

full rationale

The central quantitative claim reduces to the definition of the estimator. In Eq. (5), V̂π* is the average of the top 5% of the agent's experienced episode returns, so the inequality V̂π* ≥ V̂πθ is guaranteed by probability theory for any policy with positive return variance, regardless of whether the policy is optimally exploiting its data. The paper does not calibrate the metric: it never trains a reference policy known to exploit all available experience and shows that its top-5% gap is near 1. Its deterministic-environment experiments only replay the single best discovered trajectory; they do not remove the fact that these trajectories are sampled with epsilon-greedy or stochastic policies, and they do not establish that a gradient-trained policy can match those trajectories. The deterministic-policy curves in Section 5.1 actually show the gap shrinking when action-selection randomness is removed, which is consistent with the order-statistic interpretation. Self-citations (e.g., Tang and Berseth 2024) are not load-bearing for the main result. The paper is original in proposing an order-statistic diagnostic, but the inference from the diagnostic to 'deep RL only exploits half of its good experience' is built into the construction rather than empirically demonstrated. Hence partial circularity, score 6.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The paper's central claim depends on the unproven assumption that the top-5% return is a fair and meaningful baseline. This introduces a free parameter (the 5% threshold) and a new conceptual entity without independent validation. The metric is self-referential to the agent's own data.

free parameters (1)
  • top 5% threshold (k = |D|/20) = 5%
    The choice of the top 5% of trajectories to define the experience optimal policy V(π̂*) is arbitrary and not derived from theory. Changing this percentile would change the computed gap and potentially the conclusions.
assumptions (4)
  • domain assumption The gap between top-5% returns and the learned policy's mean return is an indicator of optimization/exploitation limitation.
    This is the central interpretive assumption, stated in Section 4. It is not proven that the gap is not simply due to stochasticity or the exploration-exploitation tradeoff.
  • domain assumption The top 5% of returns is a valid surrogate for the 'experience optimal policy'.
    The paper makes this choice in Section 4.1 without justification beyond convenience. Other quantiles would give different values.
  • ad hoc to paper The environment dynamics are such that replaying the best trajectory yields a meaningful benchmark.
    For stochastic environments the direct replay is impossible, and the soft estimators are used instead. The validity of this adaptation is not assessed.
  • standard math Standard RL definitions and the performance difference lemma (Equation 3) are correct.
    Used as background for the policy gradient and sub-optimality concepts.
invented entities (1)
  • experience optimal policy π̂*
    purpose: To represent the best policy achievable from the collected experience, used as a baseline to measure practical sub-optimality.
    This is a new conceptual construct introduced in Section 4. It is not measured against external optimality or any ground truth, and its validity as a benchmark depends entirely on the paper's assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Exploration or Optimization the Problem for Deep Reinforcement Learning?." pith.science (2026). https://pith.science/paper/NWYDPC4N

@misc{pith2026250801329,
  author       = {Pith},
  title        = {Pith review of: Is Exploration or Optimization the Problem for Deep Reinforcement Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NWYDPC4N}},
  note         = {Machine review of arXiv:2508.01329}
}
abstract

In the era of deep reinforcement learning, making progress is more complex, as the collected experience must be compressed into a deep model for future exploitation and sampling. Many papers have shown that training a deep learning policy under the changing state and action distribution leads to sub-optimal performance, or even collapse. This naturally leads to the concern that even if the community creates improved exploration algorithms or reward objectives, will those improvements fall on the \textit{deaf ears} of optimization difficulties. This work proposes a new \textit{practical} sub-optimality estimator to determine optimization limitations of deep reinforcement learning algorithms. Through experiments across environments and RL algorithms, it is shown that the difference between the best experience generated is 2-3$\times$ better than the policies' learned performance. This large difference indicates that deep RL methods only exploit half of the good experience they generate.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 51 canonical work pages

  1. [1]

    Achiam, E

    J. Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint, arXiv:1903.08894, 2019

    arXiv 1903

  2. [2]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal, Max Schwarzer, P S Castro, Aaron C Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Neural Inf Process Syst, 34: 0 29304--29320, August 2021

    work page 2021

  3. [3]

    Atari-5: Distilling the arcade learning environment down to five games

    Matthew Aitchison, Penny Sweetser, and Marcus Hautter. Atari-5: Distilling the arcade learning environment down to five games. ICML, abs/2210.02019: 0 421--438, October 2022

    arXiv 2022

  4. [4]

    Never give up: Learning directed exploration strategies, 2020

    Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies, 2020

    work page 2020

  5. [5]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016

    work page 2016

  6. [6]

    Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling

    Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pages 4148--4152. AAAI Press, 2015. ISBN 978-1-57735-738-4. URL http://dl.acm.org/citation.cfm?id=2832747.2832830

    work page arXiv 2015

  7. [7]

    Bellemare, Will Dabney, and R \' e mi Munos

    Marc G. Bellemare, Will Dabney, and R \' e mi Munos. A distributional perspective on reinforcement learning. In ICML, 2017

    work page 2017

  8. [8]

    The theory of dynamic programming

    Richard Bellman. The theory of dynamic programming. Bulletin of the American Mathematical Society, 60 0 (6): 0 503--515, 1954

    work page 1954

Show all 65 references
  1. [9]

    Interference and generalization in temporal difference learning

    Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In ICML, 2020

    2020

  2. [10]

    Exploration by random network distillation, 2018 a

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation, 2018 a

    2018

  3. [11]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. ICLR, 2018 b

    2018

  4. [12]

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In ICLR, 2021

    2021

  5. [13]

    Target network and truncation overcome the deadly triad in q-learning

    Zaiwei Chen, John - Paul Clarke, and Siva Theja Maguluri. Target network and truncation overcome the deadly triad in q-learning. arXiv preprint, arXiv:2203.02628, 2022

    2022 arXiv

  6. [14]

    Phasic policy gradient

    Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, 2021

    2021

  7. [15]

    Loss of plasticity in deep continual learning

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, August 2024

    2024

  8. [16]

    Stop regressing: Training value functions via classification for scalable deep RL

    Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL . arX...

    2024

  9. [17]

    Fujimoto, H

    S. Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018

    2018

  10. [18]

    Non-stationary learning of neural networks with automatic soft parameter reset

    Alexandre Galashov, Michalis K Titsias, Andr'as Gyorgy, Clare Lyle, Razvan Pascanu, Y W Teh, and M Sahani. Non-stationary learning of neural networks with automatic soft parameter reset. Neural Inf Process Syst, abs/2411.04034: 0 83197--83234, November 2024

    2024 arXiv

  11. [19]

    Improving performance in reinforcement learning by breaking generalization in neural networks

    Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. In AAMAS, 2020

    2020

  12. [20]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018

    2018

  13. [21]

    Temporal difference learning for model predictive control

    Nicklas Hansen, Xiaolong Wang, and H Su. Temporal difference learning for model predictive control. ICML, 162: 0 8387--8406, March 2022

    2022

  14. [22]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018

    2018

  15. [23]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pages 267--274, 2002

    2002

  16. [24]

    Discor: Corrective feedback in reinforcement learning via distribution correction

    Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In NeurIPS, 2020

    2020

  17. [25]

    Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In ICML, 2020

    2020

  18. [26]

    Maxmin q-learning: Controlling the estimation bias of q-learning

    Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020

    2020

  19. [27]

    Optidice: Offline policy optimization via stationary distribution correction estimation

    Jongmin Lee, Wonseok Jeon, Byung - Jun Lee, Joelle Pineau, and Kee - Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In ICML, 2021

    2021

  20. [28]

    Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation

    Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. In ICLR, 2022

    2022

  21. [29]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2015

    2015

  22. [30]

    Understanding plasticity in neural networks

    Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo \' A vila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In ICML, 2023

    2023

  23. [31]

    Normalization and effective learning rates in reinforcement learning

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, H V Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. Neural Inf Process Syst, abs/2407.01800: 0 106440--106473, July 2024

    2024 arXiv

  24. [32]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  25. [33]

    Algaedice: Policy gradient from arbitrary experience

    Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint, arXiv:1912.02074, 2019

    1912 arXiv

  26. [34]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mi o \'s , and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405.16158, 2024

    2024 arXiv

  27. [35]

    Nikishin, M

    E. Nikishin, M. Schwarzer, P. D'Oro, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In ICML, Proceedings of Machine Learning Research, 2022

    2022

  28. [36]

    Mixtures of experts unlock parameter scaling for deep RL

    Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob N Foerster, G Dziugaite, D Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL . ICML, abs/2402.08609, February 2024

    2024 arXiv

  29. [37]

    Chatgpt: Optimizing language models for dialogue, 2022

    OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https://openai.com/blog/chatgpt/

    2022

  30. [38]

    Dota 2 with large scale deep reinforcement learning

    OpenAI , Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto...

    2019

  31. [39]

    Bellemare, Aaron van den Oord, and Remi Munos

    Georg Ostrovski, Marc G. Bellemare, Aaron van den Oord, and Remi Munos. Count-based exploration with neural density models, 2017

    2017

  32. [40]

    The difficulty of passive learning in deep reinforcement learning

    Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In NeurIPS, 2021

    2021

  33. [41]

    Jha, Toshisada Mariyama, and Daniel Nikovski

    Kei Ota, Tomoaki Oiki, Devesh K. Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? In ICML, 2020

    2020

  34. [42]

    Fuzzy tiling activations: A simple approach to learning sparse representations online

    Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. In ICLR, 2021

    2021

  35. [43]

    Efros, and Trevor Darrell

    Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction, 2017

    2017

  36. [44]

    Bridging the gap between target networks and functional regularization

    Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan. Bridging the gap between target networks and functional regularization. Transactions on Machine Learning Research, 2022

    2022

  37. [45]

    Decoupling value and policy for generalization in reinforcement learning

    Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In ICML, 2021

    2021

  38. [46]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016

    2016

  39. [47]

    Schulman, S

    J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015

    2015

  40. [48]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, arXiv:1707.06347, 2017

    2017 arXiv

  41. [49]

    Courville, Marc G

    Max Schwarzer, Johan Samir Obando - Ceron, Aaron C. Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In ICML, 2023

    2023

  42. [50]

    H. Tang, Z. Meng, J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In AAAI, 2022

    2022

  43. [51]

    \#exploration: A study of count-based exploration for deep reinforcement learning, 2017

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. \#exploration: A study of count-based exploration for deep reinforcement learning, 2017

    2017

  44. [52]

    Improving deep reinforcement learning by reducing the chain effect of value and policy churn

    Hongyao Tang and Glen Berseth. Improving deep reinforcement learning by reducing the chain effect of value and policy churn. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=cQoAgPBARc

    2024

  45. [53]

    Temporal difference learning and td-gammon

    Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38 0 (3): 0 58--68, 1995

    1995

  46. [54]

    Deep reinforcement learning with double q-learning

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI, 2016

    2016

  47. [55]

    Deep reinforcement learning and the deadly triad

    Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint, arXiv:1812.02648, 2018

    2018 arXiv

  48. [56]

    Overcoming the spectral bias of neural value approximation

    Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approximation. In ICLR, 2022

    2022

  49. [57]

    MinAtar : An atari-inspired testbed for thorough and reproducible reinforcement learning experiments

    Kenny Young and Tian Tian. MinAtar : An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv [cs.LG], March 2019

    2019

  50. [58]

    Learning invariant representations for reinforcement learning without reconstruction

    Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021 a

    2021

  51. [59]

    Gendice: Generalized offline estimation of stationary values

    Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In ICML, 2020 a

    2020

  52. [60]

    Breaking the deadly triad with a target network

    Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with a target network. In ICML, 2021 b

    2021

  53. [61]

    Bebold: Exploration beyond the boundary of explored regions

    Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Bebold: Exploration beyond the boundary of explored regions. arXiv preprint arXiv:2012.08621, 2020 b

    2012 arXiv

  54. [62]

    Noveld: A simple yet effective exploration criterion

    Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34, 2021 c

    2021

  55. [63]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  56. [64]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  57. [65]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.