{"id":"f7b995f6-a380-411e-8ed0-3c2a472a832d","arxiv_id":"1908.06040","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":14,"one_line_summary":"A hybrid of Double Q-learning and Deep Recurrent Q-learning achieves higher reported scores on four Atari games than either method alone, but the evaluation lacks statistical rigor.","lead":"This paper combines two existing Atari-playing algorithms, Double Q-learning and a recurrent (LSTM) variant of Deep Q-learning, into one model called DRDQN. The author reports that this hybrid scores higher than the individual methods on four Atari games, though without error bars or code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DRDQN superiority claim rests on a single table of scores with no seeds, error bars, or defined scoring protocol; until those numbers are reproduced with a fixed evaluation procedure, the central comparison is untestable.","rationale":"The paper's intended contribution is an empirical combination of Double Q-Learning and recurrent networks, and its central claim is that DRDQN obtains the best scores on four Atari games. For that claim to hold, the numbers in Table I must be reliable estimates of algorithm performance. The weakest point is that no evaluation protocol is specified: there are no multiple seeds, no error bars, no description of how each score was aggregated, and no code or data release. The internal inconsistencies in Table II (episodes vs. iterations vs. training frequency) mean the experiments cannot be reproduced or even unambiguously described. The Pong scores being far above the standard Atari scale further suggests that the score variable itself is not the usual ALE return, so the comparison to published DQN/DDQN results in the paper is not meaningful. This is not a disagreement with a field consensus; it is a correctness risk about whether the reported measurements measure what they claim. The proposed check — a multi-seed rerun with an explicit evaluation protocol — would directly settle whether DRDQN's advantage is real. Since the reader already rejected the paper on essentially this basis, the verdict remains unchanged. No formal verification, code, or parameter-free derivation is offered to independently support the empirical claim, so the table is the only evidence for the headline result.","tokens_in":3534,"tokens_out":3757,"duration_ms":41329,"concrete_test":"Obtain or reimplement the four algorithms following Table II exactly, run each on SpaceInvaders, Enduro, Pong, and Beam Rider for 10M frames (not episodes) with at least 10 independent seeds, and evaluate each seed with a fixed protocol: undiscounted episode return over a fixed number of no-op starts and a fixed time limit. Report mean and standard error per model per game, and compute a paired significance test between DRDQN and the best baseline. If the DRDQN advantage is not significant or reverses under this protocol, the Table I claim fails; if it survives, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV reports one score per model per game in Table I, but the paper never states how those scores were computed: best episode, mean over episodes, undiscounted total return, or normalized score. The text says results were obtained 'using 10M (10 million) episodes,' while Table II labels the same quantity 'Iterations 10 000000' and describes it as 'number of batch iterations to the learning process' — episodes, frames, and gradient updates are conflated. The hyperparameter table is internally inconsistent: 'Training Frequency 4' is described as 'Repeat each action selected by the agent this many times' (a frame skip), while 'Update Frequency 10000' is described as 'number of actions by agent between successive SGD updates' and 'Y Update Frequency 40000' governs target network updates; the actual training loop cannot be reconstructed. Moreover, the Pong scores (DQN=65, DRQN=39, DDQN=44, DRDQN=74) are far above the standard Atari Pong reward scale where published DQN scores are typically near 20-21, suggesting either a different reward/evaluation convention or an implementation mismatch. Because the central claim that DRDQN outperforms all baselines depends entirely on these four rows, the absence of a scoring protocol, seeds, error bars, and code makes the claim unsupported rather than merely noisy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Deep Recurrent Double Q-Learning (DRDQN), which combines Double Q-learning with an LSTM-based recurrent convolutional network, and reports results on four Atari games (SpaceInvaders, Enduro, Pong, Beam Rider) against DQN, DDQN, and DRQN. The only quantitative evidence is Table I, which lists a single score per model per game and claims DRDQN achieves the best scores on all four games. The manuscript also provides a hyperparameter list in Table II and describes the training setup in Section III, but it does not define the evaluation protocol, report multiple runs, or provide error bars.","tokens_in":3871,"tokens_out":2714,"duration_ms":23197,"significance":"If the claimed superiority of DRDQN were substantiated with a rigorous evaluation, combining recurrent memory with Double Q-learning could be a useful practical variant for Atari-like environments. The paper does give a concrete model description, a hyperparameter table, and results on four games, which is a start. However, the central comparison rests on a single unreplicated table of scores with an undefined scoring protocol and internally inconsistent hyperparameters, so the claimed advantage is not currently established. The paper also makes no code or data available, which further limits its reproducibility.","major_comments":[{"comment":"The central claim that DRDQN outperforms DQN, DDQN, and DRQN on all four games rests on a single score per model per game with no seeds, no error bars, and no statement of the scoring protocol (best episode, mean episode return, undiscounted total reward, or normalized score). Without this information, the four rows of Table I cannot be verified or compared against published Atari results, and the reported differences could be noise or artifacts of a particular run.","section":"Section IV, Table I"},{"comment":"The hyperparameter table is internally inconsistent and prevents reconstruction of the training loop. 'Iterations 10 000000' is described as 'number of batch iterations to the learning process' in Table II, but Section IV states results were obtained 'using 10M (10 million) episodes,' conflating episodes, frames, and gradient updates. In addition, 'Training Frequency 4' is described as 'Repeat each action selected by the agent this many times' (a frame skip), while 'Update Frequency 10000' is described as 'number of actions by agent between successive SGD updates'; the relationship between these quantities and the target network update frequency is not specified, so the experimental procedure is irreproducible.","section":"Section IV, Table II"},{"comment":"The hyperparameters are described as 'the better set (in our case),' indicating that they were selected after experimentation. Because the same table is used for all reported results and there is no held-out validation or multiple seeds, the reported improvements may reflect tuning to the four specific games rather than a general advantage of the proposed algorithm. This result-dependent selection undermines the external validity of the comparison.","section":"Section III, Table II"},{"comment":"The reported Pong scores (DQN=65, DRQN=39, DDQN=44, DRDQN=74) are far above the standard Atari Pong reward scale, where published DQN agents typically score around 20-21 (for example, in the reference [11] cited in the paper). This large discrepancy suggests either a different reward or evaluation convention than the standard ALE setting, or an implementation mismatch, and it directly contradicts the paper's statement that results are compared with Mnih et al. [11] to 'verify correct behavior of learning process.'","section":"Section IV, Table I"}],"minor_comments":[{"comment":"The text attributes the Double Q-learning idea to 'Hado et al. [5]' and then cites reference [5] as Mnih et al.'s DQN paper; the correct reference for Double Q-learning is van Hasselt et al. [9], and 'Hado' is a given name rather than a surname.","section":"Section II.B"},{"comment":"The DRQN work is attributed to 'Mathew et al. [6]' but reference [6] is the Hausknecht and Stone paper; the author name should be corrected to 'Hausknecht et al.'","section":"Section II.C"},{"comment":"Figure 2's caption refers to 'Deep Recurrent Q-Learning model (DQRN)' but the standard abbreviation is DRQN, and the surrounding text uses DRQN; the caption should be corrected.","section":"Section II and Figure 2"},{"comment":"The abstract and introduction claim the work is 'based on Deep Recurrent Q-Learning proposed by DeepMind used in AlphaZero and Go,' which is inaccurate: AlphaZero does not use the DRQN architecture, and DRQN is due to Hausknecht and Stone, not DeepMind; this statement should be corrected.","section":"Abstract and Introduction"},{"comment":"The phrase 'using 10M (10 million) episodes' is inconsistent with Table II's 'Iterations 10 000000' described as 'number of batch iterations'; the manuscript should define the units of training length consistently (e.g., frames, episodes, or gradient updates).","section":"Section IV"},{"comment":"Reference [3] is incomplete, giving only 'Reinforcement Learning Architectures' without venue, year, or page numbers; the citation should be completed.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper is very short and reads like an extended abstract. The core evaluation is a single table of unreplicated scores with no protocol, which cannot support the central claim. The inconsistencies in the hyperparameter table and the implausible Pong scores indicate that the experiments are not reported to the standard expected for a refereed venue. While the combination of DRQN with Double Q-learning is a reasonable idea to explore, the current manuscript does not provide evidence that it works, and the missing evaluation infrastructure would require a major new experimental effort to fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a desk reject. The paper proposes DRDQN, which is exactly what it sounds like: take the Double Q-learning target and put it into DRQN. That is a natural extension and the authors say so plainly. The prose is short and readable, and they are honest that their hyperparameters are 'the better set (in our case)' and that they couldn't run more games due to equipment. Credit is due for the transparency.\n\nThe problem is the evidence. The entire claim rests on Table I: one score per model per game, with no seeds, error bars, or stated evaluation protocol. We don't know if these are best episodes, mean returns, or something else. Worse, the numbers suggest a different reward convention: DQN at 65 on Pong is far above the standard ALE score range (typically around 20 for a trained agent). That alone makes the comparison suspect. The hyperparameter table conflates units — '10M episodes' in the text vs. '10,000,000 iterations' in the table, 'Training Frequency 4' described as frame skip, 'Update Frequency 10000' described as actions between SGD updates. You cannot reconstruct the training loop. And there is no code or data.\n\nNone of this is fixable by a small revision. The baseline numbers need to be reproduced under a fixed protocol, with multiple seeds and error bars, and the Pong scoring mystery needs to be resolved. The novelty is also thin: a direct substitution of one target into an existing architecture, with no theoretical or architectural contribution. The citation pattern is mostly okay, though the DDQN reference is misattributed to Mnih et al. in the background. The paper would not survive referee scrutiny even if the experiments were cleaned up, because there is essentially no new knowledge beyond 'this variant works on a few games, maybe.'\n\nSerious thinker? I'd say no — the internal inconsistencies in the experimental description and the implausible scores suggest the authors do not have a solid grasp of the ALE evaluation framework. That's a harsh call, but the work is incoherent on its own terms.\n\nBottom line: not for reading group, not citable, desk reject. If the authors redo the experiments properly and release code, there might be a short workshop paper in it, but not as it stands.","headline":"A routine DDQN+DRQN combination whose only evidence is an unreproducible four-row table; desk-reject.","tokens_in":4356,"tokens_out":2938,"would_cite":false,"duration_ms":27529,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes deep recurrent double Q-learning (DRDQN) and claims it achieves the best scores on four Atari games compared with DQN, DRQN, and DDQN.","keywords":["deep reinforcement learning","double Q-learning","recurrent Q-learning","LSTM","Atari games","DRDQN","experience replay","partially observable MDPs"],"falsifier":"Run each algorithm on the same four games with at least five random seeds under a fixed evaluation protocol and the paper's hyperparameters. If DRDQN's mean or median score does not beat DDQN's with non-overlapping error bars on all four games, the central claim fails. A second check: replace the LSTM layer with a feedforward layer of the same size; if that network matches DRDQN, the recurrent component is not responsible for the improvement.","tokens_in":3346,"feed_emoji":"🎮","tokens_out":7648,"duration_ms":60600,"temperature":0.7,"pith_summary":"The paper proposes deep recurrent double Q-learning (DRDQN), an agent that stacks an LSTM layer on a convolutional network and trains with the double Q-learning update. It reports that after 10 million training episodes, DRDQN achieves the highest raw scores among DQN, DRQN, DDQN, and DRDQN on SpaceInvaders, Enduro, Pong, and Beam Rider. The point is to show that recurrence and overestimation correction can be combined into one architecture that improves on either ingredient alone. The motivation is that different Atari games respond to different architectures, so a hybrid may cover more games.","feed_headline":"DRDQN outscores three RL baselines on four Atari games","feed_subtitle":"Adding an LSTM layer to double Q-learning lifts scores across all four tested Atari games.","key_machinery":"The central object is the DRDQN architecture: a convolutional stack (three Conv2D layers) feeding a 512-unit LSTM with tanh activation, a 128-unit ReLU dense layer, and a linear action-value output layer. The training update is the double Q-learning target $Y_t = R_{t+1} + \\gamma Q\\bigl(S_{t+1}, \\arg\\max_a Q(S_{t+1}, a; \\theta_t); \\theta'_t\\bigr)$, which decouples action selection from action evaluation to counter overestimation. The LSTM is intended to preserve information across frames, addressing the partial observability that plain DQN ignores. Together they form the mechanism the paper credits for its reported score improvements.","core_discovery":"The central claim is that DRDQN obtains the best scores for SpaceInvaders, Enduro, Pong, and Beam Rider compared with DQN, DRQN, and DDQN, as reported in Table I. In those runs, DRDQN scores 2450 on SpaceInvaders, 1698 on Enduro, 74 on Pong, and 876 on Beam Rider, while each baseline scores lower on every one of the four games. The paper presents this as evidence that the combination of a recurrent LSTM layer with the double Q-learning target improves performance on these games, even as it concedes that no single method will be best for every Atari game.","pith_inferences":["If the comparison were repeated with multiple random seeds and error bars, the DRDQN advantage might shrink or disappear because the paper reports no variance; that is a direct test the author did not run.","The same architecture should transfer naturally to partially observable Atari variants (for example, screen flicker or single-frame observations), where LSTM memory is known to help; a positive result there would strengthen the paper's interpretation.","The paper leaves implicit that its hyperparameters were tuned on these games; a stronger claim would require testing whether DRDQN's advantage survives without per-game tuning.","Clustering the full Atari suite by which algorithm improves scores, as the author suggests for future work, could turn this four-game observation into a predictive rule for when recurrence plus double Q-learning helps."],"forward_implications":["On the four tested games, DRDQN outperforms all three baselines in raw score, which makes it a viable candidate when a practitioner expects temporal dependencies in the observations.","Because DRDQN is assembled from existing public components, the stated hyperparameter list and network diagram are enough for another group to reproduce the comparison.","The paper's conclusion that different game sets favor different methods implies that architecture selection should be guided by game characteristics rather than a one-size-fits-all model.","The reported scores give a new data point for the broader project of combining recurrence with overestimation-reducing targets in value-based reinforcement learning."],"supporting_citations":[{"why":"Defines the DQN algorithm and experience replay that serve as the primary baseline and training framework.","marker":"[5]"},{"why":"Introduces DRQN, the recurrent baseline that DRDQN extends with double Q-learning.","marker":"[6]"},{"why":"Provides the convolutional recurrent architecture (CNN with LSTM) that the proposed model modifies.","marker":"[7]"},{"why":"Defines the LSTM memory cell used in the recurrent layer of DRDQN.","marker":"[8]"},{"why":"Introduces double Q-learning, whose target decomposition DRDQN adopts.","marker":"[9]"},{"why":"Supplies the Arcade Learning Environment used to run the four Atari games.","marker":"[10]"},{"why":"Provides human-level DQN scores used to check that the learning process behaves correctly.","marker":"[11]"}],"fun_headline_variants":["DRDQN beats DQN, DRQN, DDQN on all four Atari games","LSTM double Q-learning tops three baselines on four Atari titles","New DRDQN scores highest on SpaceInvaders, Enduro, Pong, Beam Rider","Recurrent double Q-learning outperforms three rivals across four Atari games"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on a single training run per algorithm with one hand-chosen hyperparameter set, so the score gaps could be noise or tuning artifacts rather than a real advantage.","fun_headline_variants_meta":{"raw":{"variants":["DRDQN beats DQN, DRQN, DDQN on all four Atari games","LSTM double Q-learning tops three baselines on four Atari titles","New DRDQN scores highest on SpaceInvaders, Enduro, Pong, Beam Rider","Recurrent double Q-learning outperforms three rivals across four Atari games"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1191,"prompt_tokens":781,"completion_tokens":410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":397,"tokens_out":410,"duration_ms":4208,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:56:55.813646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each algorithm on the same four games with at least five random seeds under a fixed evaluation protocol and the paper's hyperparameters. If DRDQN's mean or median score does not beat DDQN's with non-overlapping error bars on all four games, the central claim fails. A second check: replace the LSTM layer with a feedforward layer of the same size; if that network matches DRDQN, the recurrent component is not responsible for the improvement.","supporting_citations":[{"cited_title":"Playing Atari with Deep Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Defines the DQN algorithm and experience replay that serve as the primary baseline and training framework."},{"cited_title":"Deep Recurrent Q-Learning for Partially Observable MDPs","cited_arxiv_id":null,"evidence_quote":"Introduces DRQN, the recurrent baseline that DRDQN extends with double Q-learning."},{"cited_title":"Deep Q-Learning with Recurrent Neural Networks","cited_arxiv_id":null,"evidence_quote":"Provides the convolutional recurrent architecture (CNN with LSTM) that the proposed model modifies."},{"cited_title":"“Long short-term memory“","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM memory cell used in the recurrent layer of DRDQN."},{"cited_title":"Deep Reinforcement Learning with Double Q-learning","cited_arxiv_id":null,"evidence_quote":"Introduces double Q-learning, whose target decomposition DRDQN adopts."},{"cited_title":"Bellemare, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the Arcade Learning Environment used to run the four Atari games."},{"cited_title":"Human-level control through deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides human-level DQN scores used to check that the learning process behaves correctly."}],"review_version":1}