Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Optimizing the expected maximum return over multiple retries induces stochastic exploration in policy gradient methods without explicit bonuses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ReMax objective induces emergent stochastic exploration in policy gradient RL by optimizing expected max return over retries, implemented via RePPO and shown on MinAtar and Craftax.

T0 review reviewed 2026-06-28 challenge →

load-bearing objection ReMax and RePPO give a new max-return objective with a derived policy gradient, but the core claim that this produces exploration specifically because of retries rests on a shaky premise. the 2 major comments →

arxiv 2606.00151 v1 pith:AR4TMWBE submitted 2026-05-29 cs.LG cs.AI

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

Soichiro Nishimori , Paavo Parmas , Sotetsu Koyamada , Tadashi Kozuno , Toshinori Kitamura , Shin Ishii , Yutaka Matsuo This is my paper
classification cs.LG cs.AI
keywords reinforcement learningpolicy gradientexplorationReMaxRePPOPPOstochastic policiesreturn uncertainty
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper formalizes the idea that exploration is useful in RL because agents revisit similar states, allowing retries to yield better outcomes or reduce uncertainty. It proposes the ReMax objective, which evaluates a policy based on the expected maximum return from M independent executions while accounting for uncertainty. Optimizing this objective naturally leads to stochastic policies that explore, as higher potential returns from varied actions increase the max. To make this practical, a new policy gradient is derived and implemented in RePPO, which generalizes the retry count to a continuous parameter for tunable exploration. This is validated empirically on MinAtar and Craftax benchmarks where exploration improves without added bonus terms.

Core claim

ReMax is an objective that evaluates a policy by the expected maximum return over M samples, accounting for return uncertainty. Optimizing ReMax induces stochastic exploration as an emergent property without explicit bonus terms. A new policy-gradient formulation is derived for ReMax, and ReMax PPO (RePPO) is introduced as a PPO variant that optimizes ReMax while generalizing the discrete retry count M to a continuous parameter m > 0 for fine-grained control of exploration.

What carries the argument

The ReMax objective, which computes the expected maximum return over M samples of the policy while accounting for return uncertainty, and its policy gradient optimization via RePPO.

Load-bearing premise

Agents benefit from exploration only because they repeatedly encounter similar states, so that trying different actions on retries can improve performance or reduce uncertainty.

What would settle it

Training RePPO on an environment where states are never repeated and checking whether the learned policy remains stochastic or collapses to deterministic greedy behavior would test whether the emergence of exploration depends on state repetition.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Stochastic exploration emerges naturally from optimizing ReMax without needing explicit bonuses.
  • RePPO provides fine-grained control of exploration level via the continuous retry parameter m.
  • The derived policy gradient enables efficient optimization of the ReMax objective within the PPO framework.
  • Exploration improves on MinAtar and Craftax benchmarks compared to standard methods without bonus terms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ReMax mechanism may clarify why certain policy gradient methods exhibit exploration in repeated-state settings even without added incentives.
  • Generalizing ReMax beyond PPO to other policy optimization algorithms could yield similar emergent exploration effects.
  • The continuous m parameter offers a way to analyze the exploration-exploitation trade-off as a smooth function rather than discrete choices.
  • In environments lacking state repetition the advantage of ReMax over standard objectives may be reduced or absent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes ReMax, an objective that evaluates policies by the expected maximum return over M independent samples (generalized to continuous m), and claims that optimizing it via a derived policy gradient induces stochastic exploration as an emergent property without explicit bonuses. It introduces RePPO as a PPO variant implementing this objective and reports empirical improvements on MinAtar and Craftax benchmarks.

Significance. If the derivation establishes that the max-over-retries structure specifically drives the exploration (distinct from variance reduction or entropy regularization), the result would offer a new mechanism for exploration in policy gradients. The empirical evaluation on two standard benchmarks provides initial support for practical utility, though the absence of detailed error analysis or ablation on the retry-induced term limits the strength of the claim.

major comments (2)
  1. [Abstract] Abstract and introduction: the foundational premise that 'without such retries, a greedy policy is optimal' is stated without qualification, yet this does not hold in finite-horizon MDPs with unknown dynamics even in acyclic environments (e.g., tree-structured state spaces). The policy-gradient derivation must explicitly isolate the contribution of the max operator from standard exploration effects to substantiate the emergence claim.
  2. The new policy-gradient formulation for ReMax (and its continuous-m generalization in RePPO) is presented as the core technical contribution, but no equation or section is referenced in the provided text showing separation of the retry-induced stochasticity term from ordinary policy-gradient variance. Without this, attribution of exploration to the retry structure remains unverified.
minor comments (2)
  1. [Abstract] The abstract reports results on two benchmarks but supplies no details on number of seeds, statistical significance, or baseline comparisons; these should be added for reproducibility.
  2. Notation for the continuous parameter m and its relation to discrete M should be clarified early, including how the objective reduces in the limit m→1.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed feedback. We address the major comments point by point below and indicate where revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract and introduction: the foundational premise that 'without such retries, a greedy policy is optimal' is stated without qualification, yet this does not hold in finite-horizon MDPs with unknown dynamics even in acyclic environments (e.g., tree-structured state spaces). The policy-gradient derivation must explicitly isolate the contribution of the max operator from standard exploration effects to substantiate the emergence claim.

    Authors: We agree that the unqualified statement requires clarification. The premise is intended as an intuition for the value of retries when an agent can re-encounter similar states, but it does not universally hold in finite-horizon MDPs with unknown dynamics. We will revise the abstract and introduction to qualify the claim, noting that it applies under the assumption of repeated interactions with comparable states or when dynamics are known and deterministic. We will also ensure the policy-gradient derivation in Section 3 explicitly isolates the max-operator contribution. revision: yes

  2. Referee: [—] The new policy-gradient formulation for ReMax (and its continuous-m generalization in RePPO) is presented as the core technical contribution, but no equation or section is referenced in the provided text showing separation of the retry-induced stochasticity term from ordinary policy-gradient variance. Without this, attribution of exploration to the retry structure remains unverified.

    Authors: Section 3 derives the ReMax policy gradient (Equation 4) and its continuous-m extension. The resulting expression contains an additional term induced by the max over retries that encourages action stochasticity beyond standard REINFORCE variance. To make the separation explicit and address the lack of clear referencing, we will add a dedicated paragraph and equation decomposition in Section 3.2 that isolates the retry-induced stochasticity term from baseline variance effects, with explicit cross-references to the relevant equations. revision: yes

Circularity Check

0 steps flagged

No circularity: ReMax objective and gradient derivation are self-contained from stated intuition.

full rationale

The paper defines ReMax as E[max_{k=1..M} G(τ_k)] from an external intuition about retries, then derives a policy gradient and RePPO variant. No step reduces a claimed prediction to a fitted parameter by construction, no self-citation chain bears the central claim, and the emergence property is presented as a direct consequence of the defined objective rather than renamed or smuggled via prior work. The derivation chain remains independent of its own outputs.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that repeated encounters with similar states make retries useful, plus the mathematical step of deriving a policy gradient for the max-return objective; M/m is the only explicit tunable parameter.

free parameters (1)
  • M (or continuous m)
    Retry count that controls the degree of exploration; generalized from discrete positive integer to continuous m > 0.
axioms (1)
  • domain assumption Agents benefit from exploration only because they repeatedly encounter similar states
    Explicitly stated as the intuition being formalized.

reviewed 2026-06-28 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying." pith.science (2026). https://pith.science/paper/AR4TMWBE

@misc{pith2026260600151,
  author       = {Pith},
  title        = {Pith review of: Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AR4TMWBE}},
  note         = {Machine review of arXiv:2606.00151}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over $M$ samples, where $M$ is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count $M$ to a continuous parameter $m > 0$, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks.

Figures

Figures reproduced from arXiv: 2606.00151 by Paavo Parmas, Shin Ishii, Soichiro Nishimori, Sotetsu Koyamada, Tadashi Kozuno, Toshinori Kitamura, Yutaka Matsuo.

Figure 1
Figure 1. Figure 1: Bandit problems. (Left) ReMax objective for a binary bandit. The standard reinforcement learning objective (M = 1) has a deterministic optimal policy (p ∗ = 1); in contrast, increasing the retry count M ≥ 2 shifts the optimal policy toward stochastic exploration to hedge against reward uncertainty. (Center) A plot of how the optimal policy changes as the reward variance changes. While Softmax remains fixed… view at source ↗
Figure 2
Figure 2. Figure 2: Average cumulative regret and standard error over 256 runs. ReMax with M = 2 and M = 3, optimized by gradient ascent. MDPs, M ∼ P(M), capturing (epistemic) uncertainty over unexplored regions of the environment (Ghosh et al., 2021). Retrying in RL would naïvely require multiple roll￾outs from the same state until termination, which is infea￾sible without a resettable simulator (Ecoffet et al., 2021) and of… view at source ↗
Figure 3
Figure 3. Figure 3: MinAtar results. Left: normalized scores aggregated with median, IQM, and mean across four games; boxes denote RLiable summaries over 10 seeds. RePPO, without entropy bonus, outperforms PPO-V, PPO-Q with entropy, and PPO-V + RND. Right: policy entropy during Breakout training. We observe that RePPO keeps high entropy without entropy bonus, indicating the promoted exploration. 0.30 0.45 0.60 0.75 m=0.9 m=1 … view at source ↗
Figure 4
Figure 4. Figure 4: MinAtar results. Effect of retry parameter m. Left: median, IQM, and mean of normalized evaluation return across all games. The best performance occurred around m ∈ [1.2, 1.4]. Right: policy entropy on Breakout. Larger m slowed entropy decay and encouraged exploration, while smaller m leads to faster entropy decay. computation is minimal (see App. E.2). We provide the advantage computation code in App. D. … view at source ↗
Figure 5
Figure 5. Figure 5: MinAtar results. Median (all games). The results show that either removing the action-independent baseline or the Q￾replacement strategy substantially degrades performance. the learning rate) to match the number of gradient updates. Additional comparisons with the unmodified PQN hyper￾parameters (yielding ∼5× more updates) are provided in App. E.2. Full hyperparameters are listed in App. E.1. Training and … view at source ↗
Figure 6
Figure 6. Figure 6: Per-game learning curves, ordered as Breakout, Asterix, Freeway, and Space Invaders. Mean ± s.e. over 10 seeds. E. MinAtar Experiment In this appendix, we provide additional details on the MinAtar experiment. E.1. Experimental setup Network architecture. We use the same network architecture as the public implementations of PPO and PQN (official implementation). PPO (Actor–Critic). A shared CNN (Conv 2×2 + … view at source ↗
Figure 7
Figure 7. Figure 7: Aggregate metrics (Median, IQM, Mean) across all games. 0.0 0.2 0.4 0.6 0.8 1.0 Env Step (M) £10 7 0.2 0.4 0.6 0.8 1.0 Entropy (a) Breakout 0.0 0.2 0.4 0.6 0.8 1.0 Env Step (M) £10 7 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Entropy (b) Asterix 0.0 0.2 0.4 0.6 0.8 1.0 Env Step (M) £10 7 0.0 0.2 0.4 0.6 0.8 1.0 Entropy (c) Freeway 0.0 0.2 0.4 0.6 0.8 1.0 Env Step (M) £10 7 0.4 0.6 0.8 1.0 1.2 1.4 Entropy m=0.9 m=1 m=… view at source ↗
Figure 8
Figure 8. Figure 8: Policy entropy for all environments over m ∈ {0.9, 1.0, 1.2, 1.4, 1.6, 2, 3}, ordered as Breakout, Asterix, Freeway, and Space Invaders. Per-game results [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Standard deviation of the EI-based advantage on Breakout. We observe that retry parameters that performed well in our experiments, such as m = 1.2 and m = 1.4, tend to yield smaller advantage standard deviations than a less exploratory setting such as m = 0.9. This suggests that the same range of m that promotes exploration can also produce a more stable EI-based advantage signal on Breakout. We emphasize… view at source ↗
Figure 11
Figure 11. Figure 11: Normalized scores aggregated with median, IQM, and mean across 10 games; boxes denote RLiable summaries over 5 seeds. For RePPO, we see the performance peak around m = 0.9 to 1.0, and PPO without entropy performs better than that with entropy. This indicates that those environments indeed require less exploration. F. Atari Experiment We use the 10 Atari environments identified as dense-reward, hard-explor… view at source ↗
Figure 12
Figure 12. Figure 12: Plot of the return curves for all games. Mean ± s.e. over 5 seeds. Overall, PPO without entropy and RePPO with m = 0.9 and 1.0 perform better than that with entropy and RePPO with m = 1.2 and 1.4. This indicates that those environments indeed require less exploration. 0.0 0.2 0.4 0.6 0.8 1.0 Env Step (M)£10 7 0.5 1.0 1.5 2.0 2.5 3.0 Entropy Alien m=0.8 m=0.9 m=1.0 m=1.2 m=1.4 PPO-V + Ent PPO-V 0.0 0.2 0.4… view at source ↗
Figure 13
Figure 13. Figure 13: Policy entropy during training on all games. Mean ± s.e. over 5 seeds. RePPO with m = 1.2 and 1.4 maintains higher entropy, while that with m = 0.9 and 1.0 exhibits faster entropy decay, demonstrating the RePPO’s ability to control the trade-off between exploration and exploitation. G. Craftax Experiment Hyperparameters. For PPO-V and PPO-V + RND (RND), we used the same hyperparameters as in the original … view at source ↗
Figure 14
Figure 14. Figure 14: Entropy (Craftax) [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Average training time of RePPO and baseline methods. regularization or an additional intrinsic-reward model. H. Speed Benchmark MinAtar. We compare the training speed of RePPO, PPO-V, and PPO-Q in MinAtar. Fig. 15a reports the average wall-clock time on Breakout for 10M timesteps (no evaluation), averaged over 5 seeds. Hyperparameters match Sec. 5. Since our implementation uses the JAX framework, we exclu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation

    cs.LG 2026-07 conditional novelty 5.0

    A per-category best-of-K RL reward, multi-axis max@K, shifts SD3.5-M perceived-appearance distributions toward uniform coverage (Fairness Score +0.23 to +0.36) without quality loss.

Reference graph

Works this paper leans on

90 extracted references · 1 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    and Barto, Andrew G

    Sutton, Richard S. and Barto, Andrew G. , year = 2018, publisher =

    2018

  2. [2]

    Why generalization in

    Ghosh, Dibya and Rahme, Jad and Kumar, Aviral and Zhang, Amy and Adams, Ryan P and Levine, Sergey , year = 2021, journal =. Why generalization in

    2021

  3. [3]

    Mohammad Gheshlaghi Azar and Ian Osband and R

  4. [4]

    Jin, Chi and Allen-Zhu, Zeyuan and Bubeck, Sebastien and Jordan, Michael I , year = 2018, booktitle =

    2018

  5. [5]

    Strehl and Michael L

    Alexander L. Strehl and Michael L. Littman , year = 2008, journal =

    2008

  6. [6]

    and Li, Lihong and Wiewiora, Eric and Langford, John and Littman, Michael L

    Strehl, Alexander L. and Li, Lihong and Wiewiora, Eric and Langford, John and Littman, Michael L. , year = 2006, booktitle =

    2006

  7. [7]

    Jaksch, Thomas and Ortner, Ronald and Auer, Peter , year = 2010, journal =

    2010

  8. [8]

    and Littman, Michael L

    Strehl, Alexander L. and Littman, Michael L. , year = 2005, booktitle =

    2005

  9. [9]

    Dann, Christoph and Brunskill, Emma , year = 2015, booktitle =

    2015

  10. [10]

    Bellemare, Marc and Srinivasan, Sriram and Ostrovski, Georg and Schaul, Tom and Saxton, David and Munos, Remi , year = 2016, booktitle =

    2016

  11. [11]

    Stadie and Sergey Levine and Pieter Abbeel , year = 2015, journal =

    Bradly C. Stadie and Sergey Levine and Pieter Abbeel , year = 2015, journal =

    2015

  12. [12]

    and Darrell, Trevor , year = 2017, booktitle =

    Pathak, Deepak and Agrawal, Pulkit and Efros, Alexei A. and Darrell, Trevor , year = 2017, booktitle =

    2017

  13. [13]

    Thrun and Knut Möller , year = 1991, institution =

    Sebastian B. Thrun and Knut Möller , year = 1991, institution =

    1991

  14. [14]

    Gomez and J

    Yi Sun and Faustino J. Gomez and J

  15. [15]

    IEEE Transactions on Autonomous Mental Development , volume = 2, number = 3, pages =

    Schmidhuber, J. IEEE Transactions on Autonomous Mental Development , volume = 2, number = 3, pages =

  16. [16]

    Houthooft, Rein and Chen, Xi and Chen, Xi and Duan, Yan and Schulman, John and De Turck, Filip and Abbeel, Pieter , year = 2016, booktitle =

    2016

  17. [17]

    Nikolay Savinov and Anton Raichuk and Damien Vincent and Raphael Marinier and Marc Pollefeys and Timothy Lillicrap and Sylvain Gelly , year = 2019, booktitle =

    2019

  18. [18]

    Fu, Justin and Co-Reyes, John and Levine, Sergey , year = 2017, booktitle =

    2017

  19. [19]

    Exploration by random network distillation , author =

  20. [20]

    Lior Fox and Leshem Choshen and Yonatan Loewenstein , year = 2018, booktitle =

    2018

  21. [21]

    Tang, Haoran and Houthooft, Rein and Foote, Davis and Stooke, Adam and Xi Chen, OpenAI and Duan, Yan and Schulman, John and DeTurck, Filip and Abbeel, Pieter , year = 2017, booktitle =

    2017

  22. [22]

    Adrià Puigdomènech Badia and Pablo Sprechmann and Alex Vitvitskyi and Daniel Guo and Bilal Piot and Steven Kapturowski and Olivier Tieleman and Martin Arjovsky and Alexander Pritzel and Andrew Bolt and Charles Blundell , year = 2020, booktitle =

    2020

  23. [23]

    Nature , volume=

    First return, then explore , author=. Nature , volume=. 2021 , publisher=

    2021

  24. [24]

    Williams and Jing Peng , year = 1991, journal =

    Ronald J. Williams and Jing Peng , year = 1991, journal =

    1991

  25. [25]

    Machine Learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine Learning , volume=. 1992 , publisher=

    1992

  26. [26]

    Mnih, Volodymyr and Badia, Adria Puigdomenech and Mirza, Mehdi and Graves, Alex and Lillicrap, Timothy and Harley, Tim and Silver, David and Kavukcuoglu, Koray , year = 2016, booktitle =

    2016

  27. [27]

    Lasse Espeholt and Hubert Soyer and Rémi Munos and Karen Simonyan and Volodymyr Mnih and Tom Ward and Yotam Doron and Vlad Firoiu and Tim Harley and Iain Dunning and Shane Legg and Koray Kavukcuoglu , year = 2018, booktitle =

    2018

  28. [28]

    Haarnoja, Tuomas and Zhou, Aurick and Abbeel, Pieter and Levine, Sergey , year = 2018, booktitle =

    2018

  29. [29]

    Mutti, Mirco and Pratissoli, Lorenzo and Restelli, Marcello , year = 2021, journal =

    2021

  30. [30]

    Pitis, Silviu and Chan, Harris and Zhao, Stephen and Stadie, Bradly and Ba, Jimmy , year = 2020, booktitle =

    2020

  31. [31]

    Baram, Nir and Tennenholtz, Guy and Mannor, Shie , year = 2021, booktitle =

    2021

  32. [32]

    and Maas, Andrew and Bagnell, J

    Ziebart, Brian D. and Maas, Andrew and Bagnell, J. Andrew and Dey, Anind K. , year = 2008, booktitle =

    2008

  33. [33]

    Proximal policy optimization algorithms , author =

  34. [34]

    Walder, Christian and Karkhanis, Deep Tejas , journal=

  35. [35]

    International Conference on Machine Learning , pages=

    Optimizing language models for inference time objectives using reinforcement learning , author=. International Conference on Machine Learning , pages=. 2025 , organization=

    2025

  36. [36]

    Deep exploration via bootstrapped

    Osband, Ian and Blundell, Charles and Pritzel, Alexander and Van Roy, Benjamin , year = 2016, booktitle =. Deep exploration via bootstrapped

    2016

  37. [37]

    Randomized prior functions for deep reinforcement learning , author =

  38. [38]

    Journal of Machine Learning Research , volume = 20, number = 124, pages =

    Deep exploration via randomized value functions , author =. Journal of Machine Learning Research , volume = 20, number = 124, pages =

  39. [39]

    Efficient exploration through bayesian deep

    Azizzadenesheli, Kamyar and Brunskill, Emma and Anandkumar, Animashree , year = 2018, booktitle =. Efficient exploration through bayesian deep

    2018

  40. [40]

    International Conference on Learning Representations , volume=

    Efficient model-based reinforcement learning through optimistic thompson sampling , author=. International Conference on Learning Representations , volume=

  41. [41]

    Uncertainty in Artificial Intelligence , pages =

    Approximate thompson sampling via epistemic neural networks , author =. Uncertainty in Artificial Intelligence , pages =

  42. [42]

    International Conference on Machine Learning , pages =

    Posterior sampling for deep reinforcement learning , author =. International Conference on Machine Learning , pages =

  43. [43]

    International Conference on Learning Representations , volume=

    Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning , author=. International Conference on Learning Representations , volume=

  44. [44]

    International Conference on Machine Learning , pages =

    Count-based exploration with neural density models , author =. International Conference on Machine Learning , pages =

  45. [45]

    International Conference on Machine Learning , pages =

    Flipping coins to estimate pseudocounts for exploration in reinforcement learning , author =. International Conference on Machine Learning , pages =

  46. [46]

    International Conference on Learning Representations , year=

    On bonus-based exploration methods in the arcade learning environment , author=. International Conference on Learning Representations , year=

  47. [47]

    Chen, Zhipeng and Qin, Xiaobo and Wu, Youbin and Ling, Yue and Ye, Qinghao and Zhao, Wayne Xin and Shi, Guang , year = 2025, journal =

    2025

  48. [48]

    Advances in Neural Information Processing Systems , volume = 36, pages =

    Pgx: Hardware-accelerated parallel game simulators for reinforcement learning , author =. Advances in Neural Information Processing Systems , volume = 36, pages =

  49. [49]

    International Conference on Learning Representations , volume=

    Simplifying deep temporal difference learning , author=. International Conference on Learning Representations , volume=

  50. [50]

    Young, Kenny and Tian, Tian , year = 2019, journal =

    2019

  51. [51]

    Journal of Artificial Intelligence Research , volume = 47, pages =

    The arcade learning environment: An evaluation platform for general agents , author =. Journal of Artificial Intelligence Research , volume = 47, pages =

  52. [52]

    Advances in Neural Information Processing Systems , volume = 35, pages =

    Discovered policy optimisation , author =. Advances in Neural Information Processing Systems , volume = 35, pages =

  53. [53]

    Advances in Neural Information Processing Systems , volume = 34, pages =

    Deep reinforcement learning at the edge of the statistical precipice , author =. Advances in Neural Information Processing Systems , volume = 34, pages =

  54. [54]

    Journal of the ACM (JACM) , publisher =

    Near-optimal regret bounds for thompson sampling , author =. Journal of the ACM (JACM) , publisher =

  55. [55]

    Artificial Intelligence and Statistics , pages =

    Optimality of Thompson sampling for Gaussian bandits depends on priors , author =. Artificial Intelligence and Statistics , pages =

  56. [56]

    Biometrika , publisher =

    On the likelihood that one unknown probability exceeds another in view of the evidence of two samples , author =. Biometrika , publisher =

  57. [57]

    Machine Learning , publisher =

    Finite-time analysis of the multiarmed bandit problem , author =. Machine Learning , publisher =

  58. [58]

    Journal of Global Optimization , publisher =

    Efficient global optimization of expensive black-box functions , author =. Journal of Global Optimization , publisher =

  59. [59]

    Proceedings of the 19th international conference on autonomous agents and multiagent systems , pages=

    Neural replicator dynamics: Multiagent learning via hedging policy gradients , author=. Proceedings of the 19th international conference on autonomous agents and multiagent systems , pages=

  60. [60]

    Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin , year = 2013, journal =. Playing

    2013

  61. [61]

    Evaluating large language models trained on code , author =

  62. [62]

    Sukhija, Bhavya and Coros, Stelian and Krause, Andreas and Abbeel, Pieter and Sferrazza, Carmelo , booktitle=

  63. [63]

    Uncertainty-based offline reinforcement learning with diversified

    An, Gaon and Moon, Seungyong and Kim, Jang-Hyun and Song, Hyun Oh , year = 2021, journal =. Uncertainty-based offline reinforcement learning with diversified

    2021

  64. [64]

    Maximum entropy

    Eysenbach, Benjamin and Levine, Sergey , booktitle=. Maximum entropy

  65. [65]

    Advances in Neural Information Processing Systems , volume = 33, pages =

    Munchausen reinforcement learning , author =. Advances in Neural Information Processing Systems , volume = 33, pages =

  66. [66]

    Markov decision processes: discrete stochastic dynamic programming , author =

  67. [67]

    International Conference on Learning Representations , year=

    Dream to control: Learning behaviors by latent imagination , author=. International Conference on Learning Representations , year=

  68. [68]

    Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

    Reinforcement learning and control as probabilistic inference: Tutorial and review , author=. arXiv preprint arXiv:1805.00909 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  69. [69]

    Advances in Neural Information Processing Systems , volume=

    Improving deep reinforcement learning by reducing the chain effect of value and policy churn , author=. Advances in Neural Information Processing Systems , volume=

  70. [70]

    No representation, no trust: connecting representation, collapse, and trust issues in

    Moalla, Skander and Miele, Andrea and Pyatko, Daniil and Pascanu, Razvan and Gulcehre, Caglar , journal=. No representation, no trust: connecting representation, collapse, and trust issues in

  71. [71]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Craftax: a lightning-fast benchmark for open-ended reinforcement learning , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  72. [72]

    International Conference on Learning Representations , year=

    Benchmarking the spectrum of agent capabilities , author=. International Conference on Learning Representations , year=

  73. [73]

    K. The. Advances in Neural Information Processing Systems , volume=

  74. [74]

    Araújo , title =

    Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo , title =. Journal of Machine Learning Research , year =

  75. [75]

    Weng, Jiayi and Lin, Min and Huang, Shengyi and Liu, Bo and Makoviichuk, Denys and Makoviychuk, Viktor and Liu, Zichen and Song, Yufan and Luo, Ting and Jiang, Yukun and others , journal=

  76. [76]

    International Conference on Learning Representations , volume=

    Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo , author=. International Conference on Learning Representations , volume=

  77. [77]

    International Conference on Machine Learning , pages=

    Randomized exploration in reinforcement learning with general value function approximation , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  78. [78]

    2022 , url=

    Emergence of exploration in policy gradient reinforcement learning via resetting , author=. 2022 , url=

    2022

  79. [79]

    Reinforcement Learning Conference , year=

    Recursive reward aggregation , author=. Reinforcement Learning Conference , year=

  80. [80]

    International Conference on Learning Representations , year=

    Polychromic objectives for reinforcement learning , author=. International Conference on Learning Representations , year=

Showing first 80 references.

This paper was first reviewed by grok-4.3 on June 28, 2026.