Pith. sign in

REVIEW 4 major objections 6 minor 12 references

Performing Deep Recurrent Double Q-Learning for Atari Games

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes deep recurrent double Q-learning (DRDQN) and claims it achieves the best scores on four Atari games compared with DQN, DRQN, and DDQN.

desk verdict A routine DDQN+DRQN combination whose only evidence is an unreproducible four-row table; desk-reject. read the letter →

arxiv 1908.06040 v2 pith:F6HYKRUB submitted 2019-08-16 cs.LG cs.AIcs.NEstat.ML

Felipe Moreno-Vera This is my paper · ORCID
classification cs.LGcs.AIcs.NEstat.ML
keywords deepreinforcementlearningdoubleQ-learningrecurrentLSTMAtarigamesDRDQNexperiencereplaypartiallyobservableMDPs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes deep recurrent double Q-learning (DRDQN), an agent that stacks an LSTM layer on a convolutional network and trains with the double Q-learning update. It reports that after 10 million training episodes, DRDQN achieves the highest raw scores among DQN, DRQN, DDQN, and DRDQN on SpaceInvaders, Enduro, Pong, and Beam Rider. The point is to show that recurrence and overestimation correction can be combined into one architecture that improves on either ingredient alone. The motivation is that different Atari games respond to different architectures, so a hybrid may cover more games.

What carries the argument

The central object is the DRDQN architecture: a convolutional stack (three Conv2D layers) feeding a 512-unit LSTM with tanh activation, a 128-unit ReLU dense layer, and a linear action-value output layer. The training update is the double Q-learning target $Y_t = R_{t+1} + \gamma Q\bigl(S_{t+1}, \arg\max_a Q(S_{t+1}, a; \theta_t); \theta'_t\bigr)$, which decouples action selection from action evaluation to counter overestimation. The LSTM is intended to preserve information across frames, addressing the partial observability that plain DQN ignores. Together they form the mechanism the paper credits for its reported score improvements.

What would settle it

Run each algorithm on the same four games with at least five random seeds under a fixed evaluation protocol and the paper's hyperparameters. If DRDQN's mean or median score does not beat DDQN's with non-overlapping error bars on all four games, the central claim fails. A second check: replace the LSTM layer with a feedforward layer of the same size; if that network matches DRDQN, the recurrent component is not responsible for the improvement.

Watch

Extended reading notes

Core claim

The central claim is that DRDQN obtains the best scores for SpaceInvaders, Enduro, Pong, and Beam Rider compared with DQN, DRQN, and DDQN, as reported in Table I. In those runs, DRDQN scores 2450 on SpaceInvaders, 1698 on Enduro, 74 on Pong, and 876 on Beam Rider, while each baseline scores lower on every one of the four games. The paper presents this as evidence that the combination of a recurrent LSTM layer with the double Q-learning target improves performance on these games, even as it concedes that no single method will be best for every Atari game.

Load-bearing premise

The entire comparison rests on a single training run per algorithm with one hand-chosen hyperparameter set, so the score gaps could be noise or tuning artifacts rather than a real advantage.

Editorial extensions

If this is right

  • On the four tested games, DRDQN outperforms all three baselines in raw score, which makes it a viable candidate when a practitioner expects temporal dependencies in the observations.
  • Because DRDQN is assembled from existing public components, the stated hyperparameter list and network diagram are enough for another group to reproduce the comparison.
  • The paper's conclusion that different game sets favor different methods implies that architecture selection should be guided by game characteristics rather than a one-size-fits-all model.
  • The reported scores give a new data point for the broader project of combining recurrence with overestimation-reducing targets in value-based reinforcement learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the comparison were repeated with multiple random seeds and error bars, the DRDQN advantage might shrink or disappear because the paper reports no variance; that is a direct test the author did not run.
  • The same architecture should transfer naturally to partially observable Atari variants (for example, screen flicker or single-frame observations), where LSTM memory is known to help; a positive result there would strengthen the paper's interpretation.
  • The paper leaves implicit that its hyperparameters were tuned on these games; a stronger claim would require testing whether DRDQN's advantage survives without per-game tuning.
  • Clustering the full Atari suite by which algorithm improves scores, as the author suggests for future work, could turn this four-game observation into a predictive rule for when recurrence plus double Q-learning helps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Deep Recurrent Double Q-Learning (DRDQN), which combines Double Q-learning with an LSTM-based recurrent convolutional network, and reports results on four Atari games (SpaceInvaders, Enduro, Pong, Beam Rider) against DQN, DDQN, and DRQN. The only quantitative evidence is Table I, which lists a single score per model per game and claims DRDQN achieves the best scores on all four games. The manuscript also provides a hyperparameter list in Table II and describes the training setup in Section III, but it does not define the evaluation protocol, report multiple runs, or provide error bars.

Significance. If the claimed superiority of DRDQN were substantiated with a rigorous evaluation, combining recurrent memory with Double Q-learning could be a useful practical variant for Atari-like environments. The paper does give a concrete model description, a hyperparameter table, and results on four games, which is a start. However, the central comparison rests on a single unreplicated table of scores with an undefined scoring protocol and internally inconsistent hyperparameters, so the claimed advantage is not currently established. The paper also makes no code or data available, which further limits its reproducibility.

major comments (4)
  1. [Section IV, Table I] The central claim that DRDQN outperforms DQN, DDQN, and DRQN on all four games rests on a single score per model per game with no seeds, no error bars, and no statement of the scoring protocol (best episode, mean episode return, undiscounted total reward, or normalized score). Without this information, the four rows of Table I cannot be verified or compared against published Atari results, and the reported differences could be noise or artifacts of a particular run.
  2. [Section IV, Table II] The hyperparameter table is internally inconsistent and prevents reconstruction of the training loop. 'Iterations 10 000000' is described as 'number of batch iterations to the learning process' in Table II, but Section IV states results were obtained 'using 10M (10 million) episodes,' conflating episodes, frames, and gradient updates. In addition, 'Training Frequency 4' is described as 'Repeat each action selected by the agent this many times' (a frame skip), while 'Update Frequency 10000' is described as 'number of actions by agent between successive SGD updates'; the relationship between these quantities and the target network update frequency is not specified, so the experimental procedure is irreproducible.
  3. [Section III, Table II] The hyperparameters are described as 'the better set (in our case),' indicating that they were selected after experimentation. Because the same table is used for all reported results and there is no held-out validation or multiple seeds, the reported improvements may reflect tuning to the four specific games rather than a general advantage of the proposed algorithm. This result-dependent selection undermines the external validity of the comparison.
  4. [Section IV, Table I] The reported Pong scores (DQN=65, DRQN=39, DDQN=44, DRDQN=74) are far above the standard Atari Pong reward scale, where published DQN agents typically score around 20-21 (for example, in the reference [11] cited in the paper). This large discrepancy suggests either a different reward or evaluation convention than the standard ALE setting, or an implementation mismatch, and it directly contradicts the paper's statement that results are compared with Mnih et al. [11] to 'verify correct behavior of learning process.'
minor comments (6)
  1. [Section II.B] The text attributes the Double Q-learning idea to 'Hado et al. [5]' and then cites reference [5] as Mnih et al.'s DQN paper; the correct reference for Double Q-learning is van Hasselt et al. [9], and 'Hado' is a given name rather than a surname.
  2. [Section II.C] The DRQN work is attributed to 'Mathew et al. [6]' but reference [6] is the Hausknecht and Stone paper; the author name should be corrected to 'Hausknecht et al.'
  3. [Section II and Figure 2] Figure 2's caption refers to 'Deep Recurrent Q-Learning model (DQRN)' but the standard abbreviation is DRQN, and the surrounding text uses DRQN; the caption should be corrected.
  4. [Abstract and Introduction] The abstract and introduction claim the work is 'based on Deep Recurrent Q-Learning proposed by DeepMind used in AlphaZero and Go,' which is inaccurate: AlphaZero does not use the DRQN architecture, and DRQN is due to Hausknecht and Stone, not DeepMind; this statement should be corrected.
  5. [Section IV] The phrase 'using 10M (10 million) episodes' is inconsistent with Table II's 'Iterations 10 000000' described as 'number of batch iterations'; the manuscript should define the units of training length consistently (e.g., frames, episodes, or gradient updates).
  6. [References] Reference [3] is incomplete, giving only 'Reinforcement Learning Architectures' without venue, year, or page numbers; the citation should be completed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical comparison is not a derivation, and tuning admission does not constitute a prediction-from-fit.

full rationale

This paper makes no formal derivation chain that could be circular. It proposes an architecture that combines DRQN's LSTM layer with the Double Q-Learning target, then reports empirical scores in Table I. The combination is definitional but not circular: the reported scores are experimental outputs, not quantities that are fed back into the algorithm's equations. The only selection step is the statement in Section III that the hyperparameters are 'the better set (in our case),' which admits tuning but does not rename a fitted parameter as a prediction; no generalization claim is made beyond the four games tested. References [1] and [2] are author self-citations, but they appear only as background examples of reinforcement learning applications and are not load-bearing for the proposed model. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result occurs. Concerns about missing seeds, error bars, ambiguous evaluation protocol, and reproducibility are correctness and evidence-quality issues, not circularity, and per the reviewing rules they do not raise the circularity score.

Assumptions & free parameters 14 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical comparison, so the main burden falls on the experimental setup. The hyperparameters are all manually chosen and form the 'better set' described in Section III; no new theoretical axioms or invented entities are introduced beyond standard reinforcement learning assumptions.

free parameters (14)
  • Iterations = 10,000,000
    Manually set training duration as part of the 'better set' hyperparameters.
  • Mini-batch size = 32
    Chosen by hand; affects stochastic gradient updates.
  • Memory buffer size = 900,000
    Manually set; determines experience replay capacity.
  • Learning rate = 0.00025
    Chosen as part of the 'better set'; controls update magnitude.
  • Training frequency = 4
    Number of frames between SGD updates, manually chosen.
  • Target network update frequency = 40,000
    How often the target network is refreshed, manually set.
  • Update frequency = 10,000
    Actions between successive SGD updates, manually chosen.
  • Replay start size = 50,000
    Number of experiences before learning begins, manually set.
  • Exploration max = 1.0
    Starting epsilon value, manually chosen.
  • Exploration min = 0.1
    Final epsilon value after annealing, manually chosen.
  • Exploration steps = 850,000
    Number of frames over which epsilon anneals, manually set.
  • Discount factor gamma = 0.99
    Standard value but manually selected; affects target computation.
  • LSTM hidden size = 512
    Architecture choice for recurrent layer, manually selected; changes model capacity.
  • Dense layer size = 128
    Architecture choice for the penultimate layer, manually selected.
assumptions (3)
  • domain assumption Atari environments satisfy the Markov assumption for Q-learning
    Q-learning updates assume a Markov decision process; Atari games are only approximately Markov when using raw frames, and the paper does not address partial observability beyond adding LSTM.
  • domain assumption The CNN features are sufficient state representations
    The paper relies on a convolutional network to extract meaningful state features from frames, assuming these features capture all information needed for value estimation.
  • standard math Standard RL assumptions: stationary environment, bounded rewards, discounting
    These are implicit in the Q-learning updates and are not stated or verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performing Deep Recurrent Double Q-Learning for Atari Games." pith.science (2026). https://pith.science/paper/F6HYKRUB

@misc{pith2026190806040,
  author       = {Pith},
  title        = {Pith review of: Performing Deep Recurrent Double Q-Learning for Atari Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6HYKRUB}},
  note         = {Machine review of arXiv:1908.06040}
}
read the original abstract

Currently, many applications in Machine Learning are based on define new models to extract more information about data, In this case Deep Reinforcement Learning with the most common application in video games like Atari, Mario, and others causes an impact in how to computers can learning by himself with only information called rewards obtained from any action. There is a lot of algorithms modeled and implemented based on Deep Recurrent Q-Learning proposed by DeepMind used in AlphaZero and Go. In this document, We proposed Deep Recurrent Double Q-Learning that is an implementation of Deep Reinforcement Learning using Double Q-Learning algorithms and Recurrent Networks like LSTM and DRQN.

Figures

Figures reproduced from arXiv: 1908.06040 by the authors.

Figure 1
Figure 1. Deep Mind DQN algorithm with experience replay [5]. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Deep Q-Learning with Recurrent Neural Networks model (DQRN) [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 2
Figure 2. Deep Q-Learning with Recurrent Neural Networks model Deep [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Convolutional Networks proposed in our models. [PITH_FULL_IMAGE:figures/full_fig_p002_4.png]
Figure 6
Figure 6. Figure 6: DRDQN Accuracy. REFERENCES [1] Leon-Vera, Leonardo and Moreno-Vera, Felipe, Car Monitoring System in Apartments Garages by Small Autonomous Car Using Deep Learning, Annual International Symposium on Information Management and Big Data, Springer, 2018. [2] Leon-Vera, Le…
Figure 5
Figure 5. Figure 5: DDQN Accuracy. CONCLUSIONS We present a model based on DRQN and Double Q￾Learning combined to get a better performance in some games, using LSTM and CNN to analyze frames. We notice that each method could be good for a set of specific Atari games and other similar game…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [11]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg & Demis Hassabis, "Human-level control through deep reinforcement learning"

    work page

  2. [1]

    Leon-Vera, Leonardo and Moreno-Vera, Felipe, Car Monitoring System in Apartments’ Garages by Small Autonomous Car Using Deep Learning, Annual International Symposium on Information Management and Big Data, Springer, 2018

    work page 2018

  3. [2]

    Leon-Vera, Leonardo and Moreno-Vera, Felipe, Sistema de Monitoreo de Autos por Mini-Robot inteligente utilizando Tecnicas de Vision Computacional en Garaje Subterraneo, LACCEI, 2018

    work page 2018

  4. [3]

    Reinforcement Learning Architectures

    Richard S. Sutton, "Reinforcement Learning Architectures"

    work page

  5. [4]

    “Reinforcement Learning: An Introduction“

    Richard Sutton and Andrew Barto. “Reinforcement Learning: An Introduction“. MIT Press, 1998

    work page 1998

  6. [5]

    Playing Atari with Deep Reinforcement Learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Daan Wierstra, Alex Graves, Ioannis Antonoglou, Martin Riedmiller, "Playing Atari with Deep Reinforcement Learning", In NIPS Deep Learning Workshop 2013

    work page 2013

  7. [6]

    Deep Recurrent Q-Learning for Partially Observable MDPs

    Matthew Hausknecht and Peter Stone, "Deep Recurrent Q-Learning for Partially Observable MDPs", In AAAI Fall Symposium Series 2015

    work page 2015

  8. [7]

    Deep Q-Learning with Recurrent Neural Networks

    Clare Chen, Vincent Ying, Dillon Laird, "Deep Q-Learning with Recurrent Neural Networks"

    work page

Show all 12 references
  1. [8]

    “Long short-term memory“

    Hochreiter and Schmidhuber. “Long short-term memory“. Neural Comput. 9(8):1735-1780

  2. [9]

    Deep Reinforcement Learning with Double Q-learning

    Hado van Hasselt, Arthur Guez and David Silver, "Deep Reinforcement Learning with Double Q-learning"

  3. [10]

    Bellemare, Y

    M. Bellemare, Y. Naddaf, J. Veness and M. Bowling, “The arcade learning enviroment. An evaluation platform for general agents“. In Journal of Artificial Intelligence Research, 47:253-279, 2013

    2013

  4. [12]

    Ԅ; ˏ 9"j8 6 tzyy l6@ kp' 7l6 d2X Q q Gr P m6o&EQo>qD0 dY ٗ_ ڵk (Y ` ^ݻw ޽ xSSS:j|Ǐ|YUnKh0 `` 歷2 ap8 4M˲o 9r xUUj H / Q eCP4 T < _ \ Ȋ p8l0 X t:L&F oQw 7 h a&6c(ʲl27 D

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEco...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.