{"id":"225f97a3-f116-4d42-b912-a5aa4a6ea07b","arxiv_id":"1908.02388","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Under a fixed Rainbow agent and equal training budget, bonus-based exploration methods offer no significant improvement over epsilon-greedy on Montezuma's Revenge or other hard-exploration Atari games, and can degrade easy-game performance.","lead":"This paper compares several popular exploration bonuses for Atari games, all built on the same Rainbow agent and trained for the same number of frames. It finds that none reliably beat simple epsilon-greedy exploration, and that bonuses can hurt on easier games.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract claims bonuses 'do not provide significantly improved performance' but no significance tests or error bars are reported; with five seeds and acknowledged high variance, the negative result is underpowered and the wording overreaches.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the experimental protocol lacks the statistical power needed to support the abstract's use of 'significantly.' I agree with the reader's analysis and with the conditional acceptance verdict. The paper is a well-controlled benchmark with a clear protocol, and the direction of the findings may well be correct, but the central claim as worded is not backed by significance testing. With only five seeds, the absence of a significant improvement is not evidence of no improvement, and the high variance acknowledged for Montezuma's Revenge compounds the problem. The paper also discloses the RND implementation limitation, which is relevant but less fundamental than the power issue because it affects one method rather than the entire comparison. The proposed concrete test—computing confidence intervals and power from per-seed data—would directly resolve whether the negative result is robust or an artifact of underpowered comparisons. Since the reader already recommended CONDITIONAL acceptance and my stress-test confirms that the concern is real but addressable, I recommend keeping the verdict unchanged rather than escalating to rejection. The authors should either add significance testing/error bars or soften the 'significantly' language in the abstract.","tokens_in":8677,"tokens_out":5475,"duration_ms":54123,"concrete_test":"Obtain the per-seed final scores at 200 million frames for every method and game (from the authors or by digitizing the published training curves, e.g., Appendix Figure 5). For each bonus method versus epsilon-greedy, compute the paired difference across the five seeds and run a two-sided paired permutation test (or Wilcoxon signed-rank test), and also compute a 95% bootstrap confidence interval for the mean difference. Separately, compute the number of seeds needed to achieve 80% power to detect the observed effect size at alpha=0.05. If any confidence interval excludes zero, then the abstract's 'no significant improvement' claim is falsified for that comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, as stated in the abstract, is that 'recently developed bonuses do not provide significantly improved performance on Montezuma's Revenge or hard exploration games.' This uses the word 'significantly' in a statistical sense, yet the paper reports only means over five seeds without error bars, confidence intervals, or hypothesis tests (Figures 1, 2, 4; Section 5). The paper itself notes that 'the variance of the return on Montezuma's Revenge is high because the reward is a step function' (Appendix A, Figure 5). With n=5 and high variance, an observed gap between, e.g., CTS reaching roughly 5000 points and RND/ICM around 2500 points on Montezuma's Revenge may not be statistically distinguishable from noise. Likewise, the claim that epsilon-greedy is 'on par' or better on hard exploration games such as Gravitar could be a false negative caused by low statistical power rather than a true equivalence. Additionally, all bonus-method hyperparameters were tuned on Montezuma's Revenge (Appendix B), which increases the risk of overfitting to that game and makes cross-game comparisons noisier; this is a design choice that further weakens the strength of the negative conclusion about hard exploration games. Because the paper never conducts a significance test, the central claim is asserted rather than demonstrated. The concern is not that the direction of the results is necessarily wrong, but that the evidence does not support the 'significantly' language, and the absence of a significant difference with n=5 is weak evidence of no effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks several bonus-based exploration methods (pseudo-counts with CTS, PixelCNN, ICM, RND) against NoisyNets and epsilon-greedy, all built on the same Rainbow agent from the Dopamine framework, trained for 200 million frames on the Arcade Learning Environment. The evaluation covers Montezuma's Revenge, the hard exploration games of Bellemare et al. (2016), and a set of easier Atari games. The central claims are that, under this controlled protocol, the evaluated bonuses do not provide significantly improved performance on Montezuma's Revenge or other hard exploration games, and that they may hurt performance on easy exploration games relative to epsilon-greedy. Hyperparameters for each bonus method were tuned on Montezuma's Revenge and disclosed in Appendix B.","tokens_in":9029,"tokens_out":4377,"duration_ms":48725,"significance":"If the findings are reliable, the paper provides a valuable cautionary benchmark for the exploration community: it isolates the exploration bonus from confounding factors such as architecture, training budget, and evaluation protocol, and it shows that popular bonus methods can underperform a simple epsilon-greedy baseline when compared fairly. The study is well controlled, uses a standard framework (Dopamine), and follows recommended ALE evaluation practices (sticky actions, no life-loss termination). The negative results, if confirmed, would push the field toward more robust evaluation and away from single-game benchmarks. However, the paper currently lacks the statistical machinery needed to support the word 'significantly' in its central claims, and one of the evaluated methods (RND) is an incomplete version of the published algorithm.","major_comments":[{"comment":"The abstract and Section 5 repeatedly use the word 'significantly' (e.g., 'do not provide significantly improved performance' and 'outperforms other methods by a significant margin on GRAVITAR') in what a reader will naturally interpret as a statistical claim. However, the paper reports only means over five seeds, with no error bars, confidence intervals, or hypothesis tests in Figures 1, 2, and 4. The paper itself acknowledges in Appendix A and Figure 5 that the return on Montezuma's Revenge has high variance because the reward is a step function. With n=5 and high variance, observed gaps such as CTS reaching roughly 5000 points versus RND/ICM around 2500 points are not necessarily beyond sampling noise. The claim that epsilon-greedy is on par with, or better than, bonus methods on hard exploration games such as Gravitar could be a false negative due to low statistical power. The central negative result is therefore asserted rather than demonstrated. I would like to see either per-seed results with standard errors or confidence bands, a statistical test (e.g., paired bootstrap or Mann-Whitney on final scores), or a careful rewording that explicitly says the comparisons are descriptive and not statistically significant.","section":"Abstract; Section 5.1-5.3; Figures 1, 2, 4, 5"},{"comment":"The RND result is based on an implementation that uses only the Eq. (3) bonus and omits 'other techniques presented in the same paper that were shown to be critical to the final performance of the algorithm.' This is disclosed, but the abstract and conclusion nonetheless treat RND as one of the 'recently developed bonuses' that does not help. If the omitted techniques are indeed critical, then the reported RND performance may not be representative of the published RND method, and the negative finding for RND would be an artifact of the incomplete implementation. The paper should either implement RND faithfully, or explicitly restrict the claim to the bare bonus version and discuss how the missing components could change the ranking.","section":"Section 5.1; Appendix B.5"},{"comment":"The paper evaluates all bonus methods with hyperparameters tuned on Montezuma's Revenge, and the same game is then used as the headline evaluation. This design actually favors the bonus methods, so the negative result on Montezuma's Revenge is not a circularity artifact. However, the cross-game generalization claim is undercut by the lack of uncertainty quantification: the statement that 'performance of each method on MONTEZUMA'S REVENGE does not correlate with performance on other hard exploration problems' is based on visual inspection of mean curves across five seeds. Without error bars or a rank correlation measure with uncertainty, the observed differences in ranking could easily be noise. The authors should either provide a more explicit analysis of the stability of cross-game rankings or soften the claim to a qualitative observation.","section":"Section 5.2; Appendix B"}],"minor_comments":[{"comment":"There are several typos: 'the the set' in the abstract and Section 1, 'incentives exploration' should be 'incentivize exploration', and '1M framces' in Appendix B.1 should be '1M frames'.","section":"Abstract; Section 1; Appendix B.1"},{"comment":"The sentence 'Despite its limited performance on MONTEZUMA'S REVENGE NoisyNets gave the most consistent results across our evaluation despite its limited performance on MONTEZUMA'S REVENGE' contains a duplicated phrase and should be rewritten for clarity.","section":"Section 5.3"},{"comment":"In Section 3.3, 'highly newly visited states?' has an unnecessary question mark; in Section 3.4, 'Though is does not generate' should be 'Though it does not generate'.","section":"Section 3.3; Section 3.4"},{"comment":"Figure 2 mixes hard exploration games (Venture, Gravitar) with easy games (Asterix, Seaquest) but the caption does not say so; the text in Section 5.2 says 'Training curves for few games are shown in Figure 2' and the remaining ones are in Appendix A, which is confusing because the easy-game curves are not part of the hard-exploration discussion. Clarify the figure layout and caption.","section":"Figure 2; Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be an ICML workshop paper. The experimental setup is sound and the negative results are potentially important, but the missing statistical support is a serious issue for a journal-level claim that bonuses 'do not provide significantly improved performance.' The RND caveat also needs to be addressed properly. I would be willing to look at a revision that adds uncertainty quantification or softens the claims, and that either implements RND fully or explicitly bounds the scope of the RND conclusion. The paper would also benefit from a statement about code availability, as the use of Dopamine should make this straightforward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should look at this if you care about how exploration methods are evaluated. The paper does something overdue: it takes CTS, PixelCNN, ICM, RND, and NoisyNets, puts them on the same Rainbow agent, trains all for 200M frames on the same game set, and reports the results. That is the right way to benchmark exploration, and it is the paper's main value.\n\nWhat it finds is more nuanced than the abstract lets on. On Montezuma's Revenge, bonus methods clearly beat epsilon-greedy—CTS reaches about 5000 points, RND and ICM around 2500, while epsilon-greedy is near zero. The real negative result is on the other hard exploration games (Gravitar, Venture, Freeway, etc.), where epsilon-greedy is competitive, and on the easy games, where bonuses often hurt. So the useful takeaway is not \"bonuses don't work\"—it's \"bonuses tuned on one game don't transfer, and they can distort the reward landscape on easy games.\"\n\nThe protocol is well controlled and the authors are transparent about details: hyperparameters are disclosed, sticky actions are used, and the RND variant omits components the original authors called critical. The tuning on Montezuma's Revenge is a limitation, but it biases in favor of the bonus methods, which makes the lack of transfer more convincing, not less.\n\nThe soft spots are real. The abstract says \"do not provide significantly improved performance\" but no significance tests, confidence intervals, or error bars are shown. With five seeds and acknowledged high variance on Montezuma's Revenge, the wording overreaches. The paper should either run proper tests or use language like \"no consistent improvement.\" The RND caveat is honest but means the comparison of RND is incomplete. And the dual use of Montezuma's Revenge for tuning and evaluation, while conservative in direction, still limits what you can conclude about that game.\n\nOverall, this is a serious empirical contribution that deserves a proper refereeing. I'd send it to review, with the request that the authors tighten the statistical presentation and align the abstract with what the data actually shows. It's a useful paper for anyone working on exploration or benchmarking RL agents, and I'd cite it for the controlled experimental setup.","headline":"A genuinely useful controlled benchmark whose headline claim overreaches: the bonus methods do help on Montezuma's Revenge; what they don't do is transfer, and the absence of error bars or significance tests makes the 'no significant improvement' wording unsupported.","tokens_in":9534,"tokens_out":4022,"would_cite":true,"duration_ms":40806,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that recent exploration bonuses add little value over simple epsilon-greedy exploration in the Arcade Learning Environment once the base agent and training budget are fixed.","keywords":["reinforcement learning","exploration bonus","Arcade Learning Environment","Montezuma's Revenge","pseudo-counts","random network distillation","intrinsic curiosity","epsilon-greedy benchmark"],"falsifier":"Inspect the per-seed training curves in the paper's Appendix Figure 5: if the spread across five runs is wide enough that every method's best and worst runs overlap with epsilon-greedy's, then the mean curves cannot support a significant-difference claim; a rerun with paired bootstrap confidence intervals on 30 seeds would decide it.","tokens_in":8500,"feed_emoji":"🕹️","tokens_out":6975,"duration_ms":68415,"temperature":0.7,"pith_summary":"This paper asks whether the exploration bonuses added to recent reinforcement-learning agents earn their keep. Fixing one strong value-based agent, a 200-million-frame training budget, and tuning every bonus on Montezuma's Revenge, the authors compare pseudo-counts, curiosity-driven intrinsic rewards, and random network distillation against plain epsilon-greedy exploration and parameter-noise exploration. Their finding is that, in this setting, none of the bonuses delivers a significant improvement on Montezuma's Revenge or on the broader set of hard-exploration Atari games, and epsilon-greedy remains competitive throughout. On easier Atari games the bonuses often hurt performance, suggesting the extra reward signal can distort learning where exploration is not the bottleneck. If correct, the results imply that a strong base algorithm plus simple exploration can match or beat methods whose published gains were measured with different base agents, longer training, or extra tricks.","feed_headline":"Exploration bonuses fail to beat epsilon-greedy in Atari benchmark","feed_subtitle":"With one fixed agent and a 200M-frame budget, recent novelty bonuses add little and can hurt easy games.","key_machinery":"The load-bearing object is the reward-bonus decomposition $r_t = e_t + \\beta i_t$, where the agent is trained on the sum of the environment's extrinsic reward and an intrinsic exploration bonus scaled by $\\beta$. Each method under test is the choice of $i_t$: an inverse-density pseudo-count for CTS and PixelCNN, a forward-model prediction error for ICM, and the prediction error of a fixed random target network for RND. What carries the argument is the experimental harness that keeps the base agent identical across methods, fixes the frame budget at 200M, and tunes only $\\beta$ (and a few per-method constants) on Montezuma's Revenge, so any observed difference is attributable to the bonus rather than to a different learner. NoisyNets, which adds parameter noise rather than a bonus, is included as a comparison that optimizes the true reward.","core_discovery":"Under a fixed evaluation protocol, the paper's central discovery is that recent bonus-based exploration methods do not deliver the performance gains they are usually credited with. With the Rainbow agent and 200 million frames of training, CTS pseudo-counts reach about 5000 points on Montezuma's Revenge but do not significantly outpace other bonuses; on five of six hard-exploration games, every bonus method performs on par with epsilon-greedy, and on Gravitar epsilon-greedy even wins by a clear margin. On the easier Atari training-set games, pseudo-count methods end up worse than epsilon-greedy on almost every game, while RND and ICM match but do not exceed it. The paper concludes that previous progress on these games was likely driven by stronger reinforcement-learning algorithms and better credit assignment, not by the exploration bonuses themselves.","pith_inferences":["An obvious extension is to ablate the base agent: replace Rainbow with a plain DQN and rerun the same bonus comparison; if bonuses show a larger gap over epsilon-greedy there, the paper's negative result is specific to strong value-based agents rather than to the bonuses themselves.","The negative impact on easy games is measured with bonuses tuned for Montezuma's Revenge; tuning each bonus on each easy game could shrink the gap, so the harm may be a tuning-transfer artifact rather than an intrinsic property of bonus methods.","Because the protocol uses a fixed 200M-frame budget, the results do not speak to regimes of much longer training or to bonus methods combined with population-based or off-policy replay; a longer-budget replication could change the ranking.","For practitioners, the actionable reading is to benchmark plain epsilon-greedy before deploying a curiosity or novelty bonus, since the bonus imposes extra compute and can degrade easy-game performance."],"forward_implications":["With a fixed Rainbow agent and a 200M-frame budget, epsilon-greedy exploration is a competitive baseline that bonus methods fail to beat on the hard-exploration game set.","Tuning a bonus on Montezuma's Revenge does not transfer: the relative ranking of methods on that game does not predict rankings on other hard-exploration games.","Adding novelty bonuses to easy-exploration games can reduce performance relative to epsilon-greedy, so the bonus is not a free improvement.","NoisyNets, which explores in parameter space and optimizes the true environment reward, is the most consistent method across games, which points to exploration mechanism rather than reward modification as the more reliable direction.","Published state-of-the-art scores on Montezuma's Revenge are likely confounded by the base agent, longer training, or additional techniques, not by the exploration bonus alone."],"supporting_citations":[{"why":"Defines the hard-exploration game set and the pseudo-count framework underpinning the CTS/PixelCNN bonuses.","marker":"Bellemare et al. (2016)"},{"why":"Introduces ICM, the curiosity module whose bonus is evaluated.","marker":"Pathak et al. (2017)"},{"why":"Introduces RND, whose prediction-error bonus is evaluated.","marker":"Burda et al. (2019)"},{"why":"Defines Rainbow, the fixed value-based agent that carries every bonus.","marker":"Hessel et al. (2018)"},{"why":"Defines NoisyNets, the parameter-noise exploration strategy compared against bonus methods.","marker":"Fortunato et al. (2018)"},{"why":"Provides the Arcade Learning Environment and its games.","marker":"Bellemare et al. (2013)"},{"why":"Supplies the sticky-action protocol and evaluation recommendations the benchmark follows.","marker":"Machado et al. (2018b)"},{"why":"Extends pseudo-counts to neural density models, the basis of the PixelCNN variant.","marker":"Ostrovski et al. (2017)"}],"fun_headline_variants":["Bonus exploration methods fall short in Atari benchmark","New exploration bonuses fail to top epsilon-greedy in Atari","Atari study: bonuses don't beat basic epsilon-greedy","Bonus-based exploration gains vanish in ALE benchmark","No gain from exploration bonuses in Rainbow Atari study"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on five random seeds per method, with hyperparameters tuned on Montezuma's Revenge and no significance tests on the reported curves; if those seeds give an underpowered comparison, real bonus improvements could be missed.","fun_headline_variants_meta":{"raw":{"variants":["Bonus exploration methods fall short in Atari benchmark","New exploration bonuses fail to top epsilon-greedy in Atari","Atari study: bonuses don't beat basic epsilon-greedy","Bonus-based exploration gains vanish in ALE benchmark","No gain from exploration bonuses in Rainbow Atari study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2643,"prompt_tokens":902,"completion_tokens":1741,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1661}},"tokens_in":518,"tokens_out":1741,"duration_ms":12189,"temperature":1.0,"reasoning_tokens":1661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:45:23.514543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the per-seed training curves in the paper's Appendix Figure 5: if the spread across five runs is wide enough that every method's best and worst runs overlap with epsilon-greedy's, then the mean curves cannot support a significant-difference claim; a rerun with paired bootstrap confidence intervals on 30 seeds would decide it.","supporting_citations":[],"review_version":1}