Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that recent exploration bonuses add little value over simple epsilon-greedy exploration in the Arcade Learning Environment once the base agent and training budget are fixed.

desk verdict A genuinely useful controlled benchmark whose headline claim overreaches: the bonus methods do help on Montezuma's Revenge; what they don't do is transfer, and the absence of error bars or significance tests makes the 'no significant improvement' wording unsupported. read the letter →

arxiv 1908.02388 v3 pith:WFFV5RGV submitted 2019-08-06 cs.LG stat.ML

Adrien Ali Taïga , William Fedus , Marlos C. Machado , Aaron Courville , Marc G. Bellemare This is my paper · ORCID
classification cs.LGstat.ML
keywords reinforcementlearningexplorationbonusArcadeEnvironmentMontezuma'sRevengepseudo-countsrandomnetworkdistillationintrinsiccuriosityepsilon-greedybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the exploration bonuses added to recent reinforcement-learning agents earn their keep. Fixing one strong value-based agent, a 200-million-frame training budget, and tuning every bonus on Montezuma's Revenge, the authors compare pseudo-counts, curiosity-driven intrinsic rewards, and random network distillation against plain epsilon-greedy exploration and parameter-noise exploration. Their finding is that, in this setting, none of the bonuses delivers a significant improvement on Montezuma's Revenge or on the broader set of hard-exploration Atari games, and epsilon-greedy remains competitive throughout. On easier Atari games the bonuses often hurt performance, suggesting the extra reward signal can distort learning where exploration is not the bottleneck. If correct, the results imply that a strong base algorithm plus simple exploration can match or beat methods whose published gains were measured with different base agents, longer training, or extra tricks.

What carries the argument

The load-bearing object is the reward-bonus decomposition $r_t = e_t + \beta i_t$, where the agent is trained on the sum of the environment's extrinsic reward and an intrinsic exploration bonus scaled by $\beta$. Each method under test is the choice of $i_t$: an inverse-density pseudo-count for CTS and PixelCNN, a forward-model prediction error for ICM, and the prediction error of a fixed random target network for RND. What carries the argument is the experimental harness that keeps the base agent identical across methods, fixes the frame budget at 200M, and tunes only $\beta$ (and a few per-method constants) on Montezuma's Revenge, so any observed difference is attributable to the bonus rather than to a different learner. NoisyNets, which adds parameter noise rather than a bonus, is included as a comparison that optimizes the true reward.

What would settle it

Inspect the per-seed training curves in the paper's Appendix Figure 5: if the spread across five runs is wide enough that every method's best and worst runs overlap with epsilon-greedy's, then the mean curves cannot support a significant-difference claim; a rerun with paired bootstrap confidence intervals on 30 seeds would decide it.

Watch

Extended reading notes

Core claim

Under a fixed evaluation protocol, the paper's central discovery is that recent bonus-based exploration methods do not deliver the performance gains they are usually credited with. With the Rainbow agent and 200 million frames of training, CTS pseudo-counts reach about 5000 points on Montezuma's Revenge but do not significantly outpace other bonuses; on five of six hard-exploration games, every bonus method performs on par with epsilon-greedy, and on Gravitar epsilon-greedy even wins by a clear margin. On the easier Atari training-set games, pseudo-count methods end up worse than epsilon-greedy on almost every game, while RND and ICM match but do not exceed it. The paper concludes that previous progress on these games was likely driven by stronger reinforcement-learning algorithms and better credit assignment, not by the exploration bonuses themselves.

Load-bearing premise

The conclusion rests on five random seeds per method, with hyperparameters tuned on Montezuma's Revenge and no significance tests on the reported curves; if those seeds give an underpowered comparison, real bonus improvements could be missed.

Editorial extensions

If this is right

  • With a fixed Rainbow agent and a 200M-frame budget, epsilon-greedy exploration is a competitive baseline that bonus methods fail to beat on the hard-exploration game set.
  • Tuning a bonus on Montezuma's Revenge does not transfer: the relative ranking of methods on that game does not predict rankings on other hard-exploration games.
  • Adding novelty bonuses to easy-exploration games can reduce performance relative to epsilon-greedy, so the bonus is not a free improvement.
  • NoisyNets, which explores in parameter space and optimizes the true environment reward, is the most consistent method across games, which points to exploration mechanism rather than reward modification as the more reliable direction.
  • Published state-of-the-art scores on Montezuma's Revenge are likely confounded by the base agent, longer training, or additional techniques, not by the exploration bonus alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An obvious extension is to ablate the base agent: replace Rainbow with a plain DQN and rerun the same bonus comparison; if bonuses show a larger gap over epsilon-greedy there, the paper's negative result is specific to strong value-based agents rather than to the bonuses themselves.
  • The negative impact on easy games is measured with bonuses tuned for Montezuma's Revenge; tuning each bonus on each easy game could shrink the gap, so the harm may be a tuning-transfer artifact rather than an intrinsic property of bonus methods.
  • Because the protocol uses a fixed 200M-frame budget, the results do not speak to regimes of much longer training or to bonus methods combined with population-based or off-policy replay; a longer-budget replication could change the ranking.
  • For practitioners, the actionable reading is to benchmark plain epsilon-greedy before deploying a curiosity or novelty bonus, since the bonus imposes extra compute and can degrade easy-game performance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper benchmarks several bonus-based exploration methods (pseudo-counts with CTS, PixelCNN, ICM, RND) against NoisyNets and epsilon-greedy, all built on the same Rainbow agent from the Dopamine framework, trained for 200 million frames on the Arcade Learning Environment. The evaluation covers Montezuma's Revenge, the hard exploration games of Bellemare et al. (2016), and a set of easier Atari games. The central claims are that, under this controlled protocol, the evaluated bonuses do not provide significantly improved performance on Montezuma's Revenge or other hard exploration games, and that they may hurt performance on easy exploration games relative to epsilon-greedy. Hyperparameters for each bonus method were tuned on Montezuma's Revenge and disclosed in Appendix B.

Significance. If the findings are reliable, the paper provides a valuable cautionary benchmark for the exploration community: it isolates the exploration bonus from confounding factors such as architecture, training budget, and evaluation protocol, and it shows that popular bonus methods can underperform a simple epsilon-greedy baseline when compared fairly. The study is well controlled, uses a standard framework (Dopamine), and follows recommended ALE evaluation practices (sticky actions, no life-loss termination). The negative results, if confirmed, would push the field toward more robust evaluation and away from single-game benchmarks. However, the paper currently lacks the statistical machinery needed to support the word 'significantly' in its central claims, and one of the evaluated methods (RND) is an incomplete version of the published algorithm.

major comments (3)
  1. [Abstract; Section 5.1-5.3; Figures 1, 2, 4, 5] The abstract and Section 5 repeatedly use the word 'significantly' (e.g., 'do not provide significantly improved performance' and 'outperforms other methods by a significant margin on GRAVITAR') in what a reader will naturally interpret as a statistical claim. However, the paper reports only means over five seeds, with no error bars, confidence intervals, or hypothesis tests in Figures 1, 2, and 4. The paper itself acknowledges in Appendix A and Figure 5 that the return on Montezuma's Revenge has high variance because the reward is a step function. With n=5 and high variance, observed gaps such as CTS reaching roughly 5000 points versus RND/ICM around 2500 points are not necessarily beyond sampling noise. The claim that epsilon-greedy is on par with, or better than, bonus methods on hard exploration games such as Gravitar could be a false negative due to low statistical power. The central negative result is therefore asserted rather than demonstrated. I would like to see either per-seed results with standard errors or confidence bands, a statistical test (e.g., paired bootstrap or Mann-Whitney on final scores), or a careful rewording that explicitly says the comparisons are descriptive and not statistically significant.
  2. [Section 5.1; Appendix B.5] The RND result is based on an implementation that uses only the Eq. (3) bonus and omits 'other techniques presented in the same paper that were shown to be critical to the final performance of the algorithm.' This is disclosed, but the abstract and conclusion nonetheless treat RND as one of the 'recently developed bonuses' that does not help. If the omitted techniques are indeed critical, then the reported RND performance may not be representative of the published RND method, and the negative finding for RND would be an artifact of the incomplete implementation. The paper should either implement RND faithfully, or explicitly restrict the claim to the bare bonus version and discuss how the missing components could change the ranking.
  3. [Section 5.2; Appendix B] The paper evaluates all bonus methods with hyperparameters tuned on Montezuma's Revenge, and the same game is then used as the headline evaluation. This design actually favors the bonus methods, so the negative result on Montezuma's Revenge is not a circularity artifact. However, the cross-game generalization claim is undercut by the lack of uncertainty quantification: the statement that 'performance of each method on MONTEZUMA'S REVENGE does not correlate with performance on other hard exploration problems' is based on visual inspection of mean curves across five seeds. Without error bars or a rank correlation measure with uncertainty, the observed differences in ranking could easily be noise. The authors should either provide a more explicit analysis of the stability of cross-game rankings or soften the claim to a qualitative observation.
minor comments (4)
  1. [Abstract; Section 1; Appendix B.1] There are several typos: 'the the set' in the abstract and Section 1, 'incentives exploration' should be 'incentivize exploration', and '1M framces' in Appendix B.1 should be '1M frames'.
  2. [Section 5.3] The sentence 'Despite its limited performance on MONTEZUMA'S REVENGE NoisyNets gave the most consistent results across our evaluation despite its limited performance on MONTEZUMA'S REVENGE' contains a duplicated phrase and should be rewritten for clarity.
  3. [Section 3.3; Section 3.4] In Section 3.3, 'highly newly visited states?' has an unnecessary question mark; in Section 3.4, 'Though is does not generate' should be 'Though it does not generate'.
  4. [Figure 2; Section 5.2] Figure 2 mixes hard exploration games (Venture, Gravitar) with easy games (Asterix, Seaquest) but the caption does not say so; the text in Section 5.2 says 'Training curves for few games are shown in Figure 2' and the remaining ones are in Appendix A, which is confusing because the easy-game curves are not part of the hard-exploration discussion. Clarify the figure layout and caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose claims are supported by out-of-sample evaluations, and the in-game hyperparameter tuning favors rather than forces the reported negative results.

full rationale

This paper contains no derivation chain in which a prediction is constructed from its own inputs. The only candidate issue is that all bonus-method hyperparameters were tuned on Montezuma's Revenge (Appendix B) and the same game is then used as a benchmark; however, the paper does not present the Montezuma's Revenge result as a prediction from that tuning. If anything, tuning on the target game biases the comparison in favor of the bonus methods, so the negative finding that bonuses do not significantly help is not an artifact of a fitted input being renamed as a prediction. The generalization claims to the other hard exploration games and to the easy Atari games are genuinely out-of-sample, since hyperparameters were fixed after Montezuma's Revenge tuning. The paper's citations to the authors' own prior work (Bellemare et al. 2016; Ostrovski et al. 2017; Machado et al. 2018b) are used to justify the pseudo-count baseline and the evaluation protocol, but these are not load-bearing in a circular way: the benchmark results are actually unfavorable to pseudo-counts, which is inconsistent with a self-citation chain being used to force a favorable conclusion. The abstract's use of 'significantly' without formal significance tests or error bars is a statistical-power concern, not a circularity concern, and the paper itself acknowledges high variance on Montezuma's Revenge (Appendix A). No equation is reused as an output, no fitted parameter is relabeled as a prediction, and no uniqueness claim is imported from the authors' prior work. The comparison is self-contained against the fixed Rainbow agent and the common 200M-frame budget, so the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical study. Its conclusions rest on a set of protocol choices and tuned hyperparameters rather than a mathematical derivation. The free parameters above are the per-method bonus scales and auxiliary hyperparameters selected by training on Montezuma's Revenge; the axioms are the testbed and statistical assumptions that must hold for the negative result to generalize.

free parameters (4)
  • CTS bonus scale beta = 0.0005
    Tuned on Montezuma's Revenge final performance after 200M frames (Appendix B.3.1).
  • PixelCNN bonus scale beta and decay c = beta=0.1, c=1.0
    Tuned via sweep over beta and c on Montezuma's Revenge (Appendix B.3.2).
  • ICM weight alpha and bonus scale beta = alpha=0.005, beta=0.005
    Tuned via sweeps over alpha and beta on Montezuma's Revenge (Appendix B.4).
  • RND bonus scale beta and optimizer learning rate = beta=0.0001, lr=0.0002
    Tuned via sweeps on Montezuma's Revenge (Appendix B.5).
assumptions (4)
  • domain assumption The ALE with sticky actions and life-loss termination disabled is an appropriate evaluation testbed for exploration.
    Adopted from Machado et al. (2018b); all results depend on this protocol choice.
  • domain assumption Rainbow (Dopamine implementation) is a suitable fixed base agent for comparing exploration bonuses.
    The paper's protocol fixes the base agent; generalizing to other agents is untested.
  • domain assumption Five random seeds are sufficient to compare methods on ALE games.
    No significance tests or confidence intervals are reported; the paper acknowledges high variance on Montezuma's Revenge.
  • ad hoc to paper Tuning each method's hyperparameters on Montezuma's Revenge gives a fair comparison on that game and other games.
    This is the paper's specific protocol; it favors the bonus methods on the target game while making other-game comparisons out-of-sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment." pith.science (2026). https://pith.science/paper/WFFV5RGV

@misc{pith2026190802388,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFFV5RGV}},
  note         = {Machine review of arXiv:1908.02388}
}
abstract

This paper provides an empirical evaluation of recently developed exploration algorithms within the Arcade Learning Environment (ALE). We study the use of different reward bonuses that incentives exploration in reinforcement learning. We do so by fixing the learning algorithm used and focusing only on the impact of the different exploration bonuses in the agent's performance. We use Rainbow, the state-of-the-art algorithm for value-based agents, and focus on some of the bonuses proposed in the last few years. We consider the impact these algorithms have on performance within the popular game Montezuma's Revenge which has gathered a lot of interest from the exploration community, across the the set of seven games identified by Bellemare et al. (2016) as challenging for exploration, and easier games where exploration is not an issue. We find that, in our setting, recently developed bonuses do not provide significantly improved performance on Montezuma's Revenge or hard exploration games. We also find that existing bonus-based methods may negatively impact performance on games in which exploration is not an issue and may even perform worse than $\epsilon$-greedy exploration.

Figures

Figures reproduced from arXiv: 1908.02388 by the authors.

Figure 1
Figure 1. A comparison of different exploration methods on MON￾TEZUMA’S REVENGE. Sample efficiency is a key objective for exploration methods, yet, because published agents are often trained under differ￾ent regimes it is often not possible to directly compare their performance. They often employ different reinforcement learning algorithms, varying quantity of training frames or inconsistent hyperparameter tuning. As a remedy… view at source ↗
Figure 2
Figure 2. Evaluation of different bonus-based exploration methods on the ALE. VENTURE and GRAVITAR are hard exploration games whereas ASTERIX and SEAQUEST are easy ones. 5.3. ALE training set While the benefit of exploration bonuses has been shown on a few games they can also have a negative impact by skew￾ing the reward landscape. To get a more complete picture, we also evaluated our agents on the original Atari train￾ing se… view at source ↗
Figure 3
Figure 3. Results of different bonus-based exploration methods on hard exploration games. The relative ranking of methods differs from the one observed on MONTEZUMA’S REVENGE. We find that -greedy also performs competitively. This suggests that previous claims of progress in these games has been driven by more advanced reinforcement learning algorithms, not necessarily better exploration strategies. 0 50 100 150 200 0 10000 … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Training curves on MONTEZUMA’S REVENGE B. Hyperparameter tuning Except for NoisyNets, all other methods are tuned with respect to their final performance on MONTEZUMA’S REVENGE after training on 200 million frames on five runs. B.1. Rainbow and Atari preprocessing We u…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

    cs.LG 2019-09 conditional novelty 6.0 of 10

    R2D3, a recurrent distributed Q-learning agent that samples a small ratio of expert demonstrations alongside its own experience, solves six of eight hard-exploration procedural tasks where baseline agents fail.

  2. A survey on intrinsic motivation in reinforcement learning

    cs.LG 2019-08 accept novelty 4.0 of 10

    A survey that classifies intrinsic motivation methods in deep RL as knowledge acquisition or skill learning and proposes their unification through information compression.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [3]

    The relative ranking of methods differs from the one observed on MONTEZUMA ’S REVENGE

    Results of different bonus-based exploration methods on hard exploration games. The relative ranking of methods differs from the one observed on MONTEZUMA ’S REVENGE . We find that ϵ-greedy also performs competitively. This suggests that previous claims of progress in these games has been driven by more advanced reinforcement learning algorithms, not neces...

    work page 2000

  2. [4]

    C., Bellemare, M

    Machado, M. C., Bellemare, M. G., and Bowling, M. Count- Based Exploration with the Successor Representation. CoRR, abs/1807.11622, 2018a. Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial In...

    arXiv

  3. [5]

    Training curves on MONTEZUMA ’S REVENGE B. Hyperparameter tuning Except for NoisyNets, all other methods are tuned with respect to their final performance on MONTEZUMA ’S REVENGE after training on 200 million frames on five runs. B.1. Rainbow and Atari preprocessing We used the standard architecture and Atari preprocessing from Mnih et al. (2015). Following...

    work page 2015

  4. [7]

    Priori- tized experience replay.arXiv preprint arXiv:1511.05952,

    Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori- tized experience replay.arXiv preprint arXiv:1511.05952,

    arXiv

  5. [8]

    C., Levine, S., and Abbeel, P

    Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing ex- ploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814,

    arXiv

  6. [9]

    Randomized Value Functions via Multiplicative Normalizing Flows

    Touati, A., Satija, H., Romoff, J., Pineau, J., and Vincent, P. Randomized value functions via multiplicative normaliz- ing flows. arXiv preprint arXiv:1806.02315,

    work page Pith review arXiv

  7. [10]

    Pixel recurrent neural networks

    Van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759,

    arXiv

  8. [11]

    Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment A. Additional figures 0 50 100 150 2000 5 10 15 20 25 30 35Average Score Freeway CTS PixelCNN NoisyNetworks ϵ-greedy RND ICM 0 50 100 150 2000 500 1000 1500 2000 2500 Gravitar CTS PixelCNN NoisyNetworks ϵ-greedy RND ICM 0 50 100 150 200 0 10000 20000 30000 40000 Private Eye CTS...

    work page 2000

Show all 14 references
  1. [13]

    Rainbow with ϵ-greedy exploration performs as well as other more complex exploration method

    Evaluation of different bonus-based exploration methods on the Atari training set, except for FREEWAY all these games were classified as easy exploration problems. Rainbow with ϵ-greedy exploration performs as well as other more complex exploration method. The variance of the r...

    2000

  2. [2015]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 1928–1937,

    1928

  3. [2016]

    The uncertainty bellman equation and exploration

    O’Donoghue, B., Osband, I., Munos, R., and Mnih, V . The uncertainty bellman equation and exploration. arXiv preprint arXiv:1709.05380,

    arXiv

  4. [2017]

    Burda, Y ., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355,

    arXiv

  5. [2018]

    Dora the explorer: Directed outreaching reinforcement action- selection

    Choshen, L., Fox, L., and Loewenstein, Y . Dora the explorer: Directed outreaching reinforcement action- selection. arXiv preprint arXiv:1804.04012,

    arXiv

  6. [2019]

    S., Moitra, S., Gelada, C., Kumar, S., and Belle- mare, M

    Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Belle- mare, M. G. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110,

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.