REVIEW 4 major objections 4 minor 27 references
Superstition in the Network: Deep Reinforcement Learning Plays Deceptive Games
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A deep reinforcement learner, A2C, misses optimal play on nearly every level of four deceptive games and invents a superstitious ritual.
desk verdict A solid, modest empirical study of one A2C configuration on four hand-designed deceptive games, with a genuinely interesting superstition result, but the title and conclusion overgeneralize from A2C to deep RL as a class. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the deceptive game suite itself, combined with the A2C training protocol. Each of the four games is a minimal environment whose reward schedule punishes the very assumption a reward-driven learner must make: that immediate positive signals can be trusted as guides. DeceptiCoins uses an irreversible early choice, WaferThinMints makes the same item switch from positive to lethal, Flower makes fast collection destructive, and Invest makes the agent incur a penalty before a delayed payoff. The machinery does its work by making the failure visible and repeatable: the same superstitious corner-waiting behavior appears across independent training runs in Invest, and the Flower agent's performance declines as it gets better. Comparing these behaviors with forward-model planners isolates which failures come from the learning rule rather than from the games themselves.
What would settle it
Train A2C on DeceptiCoins level 3 for fifty million frames instead of five million. If it then consistently takes the higher-reward right-side path, the reported failure is a training-budget artifact rather than a robust deception. Likewise, give A2C a direct encoding of the avatar's fullness meter in WaferThinMints; if that alone yields safe mint collection, then that deception depends on the pixel-only representation rather than the reward structure itself.
Extended reading notes
Core claim
The central discovery is that a standard deep reinforcement learner, A2C, reliably misses the optimal policy in four deceptive games, and the way it fails is informative. In DeceptiCoins, the agent converges to a suboptimal path once the level is large enough; in WaferThinMints, it learns that mints are good but cannot master the rule that the tenth mint kills it, and the stochastic mint-placing waiter keeps the game from converging; in Flower, the agent's improving skill backfires because collecting seedlings prevents them from maturing, so its score drops as training progresses; and in Invest, it learns to invest only with the quickest-paying banker and then runs to a fixed corner to wait for the payout, an arbitrary ritual the authors call superstitious. The paper argues these four failure modes correspond to distinct deceptions—lack of hierarchical understanding, subverted generalization, delayed reward, and delayed gratification—and uses six planning agents that rely on a forward model as a comparison to show that model-free learning fails differently from planning with a forward model.
Load-bearing premise
The conclusion would collapse if a different deep RL configuration did not fail: the paper uses one algorithm—A2C with a fixed architecture, default hyperparameters, and five million training frames—and assumes the four games isolate the intended deceptions rather than confounding pixel observation or randomness.
Editorial extensions
If this is right
- If one standard model-free deep RL algorithm can be reliably deceived by such simple reward structures, then strong performance on standard arcade benchmarks is not evidence that a method handles delayed or trap-like rewards.
- The Flower result implies that training can make a reward-driven agent monotonically worse: improving its ability to collect flowers destroys the flowers' growth, so competence and score move in opposite directions.
- The Invest result implies that deep RL policies can encode stable but causally false associations, since separate training runs produced the same arbitrary waiting-spot ritual.
- Because planning agents with forward models fail on different games than the learning agent does, deceptive games can separate failures of credit assignment from failures of search or representation.
- The proposed typology gives algorithm builders four concrete stressors—hierarchical abstraction, generalization under changing effects, delayed reward, and delayed gratification—to test before deploying an agent in an environment with adversarial or natural reward traps.
Reading between the lines
- A natural extension is to test value-based methods and recurrent policies on the same four games; if they also fail, the result generalizes from actor-critic training to model-free deep RL broadly, while if they succeed it means the deception is specific to A2C's update rule or architecture.
- The superstition in Invest suggests a cheap diagnostic for causal confusion: remove the spatial cue by making the waiting spot irrelevant, and see whether the learned policy collapses or adapts; this would test whether the policy truly depends on the spurious location.
- The delayed-reward game could be made progressively harder by randomizing the investment payoff time, which would tell whether the agent can learn to wait at all or would simply develop longer and more elaborate superstitions.
- In real deployments where rewards are naturally delayed—such as medical or financial decisions—this paper implies that a model-free agent trained on reward alone may lock onto accidental correlates and should be audited for behavior that has no causal link to outcomes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies failure modes of deep reinforcement learning by introducing four 'deceptive games' implemented in the GVGAI framework: DeceptiCoins, WaferThinMints, Flower, and Invest. The authors train A2C (OpenAI Baselines, default Atari-style hyperparameters, five seeds, five million frames) on these games, evaluate each agent on 150 episodes, and compare the resulting average scores against six planning agents. They report that A2C converges to suboptimal policies in most of the games, including a 'superstitious' corner-waiting behavior in Invest, and propose an initial typology of deceptions: lack of hierarchical understanding, subverted generalization, delayed reward, and delayed gratification. The central qualitative claim is that deep reinforcement learners are easily deceived by reward structures that lead away from optimal policies.
Significance. If the empirical findings are supported, this is a useful contribution to the growing literature on failure modes of deep RL. The paper's strengths are its use of a consistent evaluation setup across four games, a comparison with six planning agents that differ in mechanism, and a concrete proposal of deception categories that could guide future benchmark design. The reported consistency across five training seeds is encouraging. However, the paper's central conclusion is broader than its evidence: only one algorithm (A2C) with one hyperparameter configuration is tested, the quantitative results lack variance measures and statistical tests, and the most distinctive behavioral claims are based on qualitative video inspection. These issues need to be addressed before the general claim about 'deep reinforcement learners' can be accepted.
major comments (4)
- [Conclusion / Abstract] The conclusion that 'deep reinforcement learners are easily deceived' is not supported by evidence from a single algorithm, A2C, with one default hyperparameter configuration (Methods: 'Instead of tuning the algorithms for the games, we designed the games for the algorithms'). The Discussion itself concedes that 'numerous modifications' of deep RL could potentially solve these games. Please either test at least one additional algorithm (e.g., DQN or PPO) under comparable conditions, or narrow the claim in the abstract and conclusion to 'this A2C configuration is easily deceived by these games.'
- [Table 1 / Comparison with planning algorithms] Table 1 reports average scores across evaluation runs, but no error bars, standard deviations, per-seed breakdowns, or statistical tests are provided for any algorithm. This makes it impossible to assess whether the A2C scores (e.g., DeceptiCoins level 2: 3.79 vs. optimal 5.0; Invest: 69.6 vs. Return42's 190.12) are reliable differences or within-run noise. Please report variance measures and clarify how the five independently trained A2C agents are aggregated into the single A2C row.
- [Invest / Results] The 'superstition' claim is load-bearing for the paper's title and discussion, but it rests entirely on qualitative video inspection. The text states that the agent 'runs to the far right corner and waits' and that the time to reach the corner 'correlates with the arrival of the delayed reward,' yet no quantitative evidence is provided. Please include position-over-time traces, waiting-time statistics, and, for all five seeds, a demonstration that the corner waiting is temporally correlated with reward payout. Without such data, the claim cannot be independently evaluated.
- [WaferThinMints / Results and Methods] The paper explicitly states that WaferThinMints 'did not converge' and 'might have benefited from more training time.' Interpreting this as evidence of deception is undercut by the acknowledged non-convergence: a truncated training run may fail for reasons unrelated to the game's reward structure. Please provide a convergence criterion, longer training results, or an explicit analysis of learning dynamics. Similarly, the Flower claim that 'the agent gets worse over time' is supported only by a training curve that is not shown; include the training curves with error bars for all seeds.
minor comments (4)
- [Methods / Reproducibility] The paper does not provide code, game descriptions, or evaluation scripts. Since the central evidence is empirical and behavioral, releasing the VGDL game files and the A2C evaluation code would substantially strengthen the work and allow replication.
- [Comparison with planning algorithms] The comparison between A2C and planning agents is described as 'apples and oranges,' but the paper does not specify how much computation the planning agents are allowed per episode. A sentence clarifying the total planning budget relative to the RL training budget would help readers interpret the comparison.
- [Deceptive Games / DeceptiCoins] The optimal scores for DeceptiCoins are not given explicitly. Reporting the maximum achievable score for each level would make Table 1 more interpretable, because the reader cannot tell whether a score of 2.0 out of a possible 5.0 represents a partial or total failure.
- [References] The reference to Anderson et al. 2018 is cited for the previous deceptive-games suite, but the current paper does not clearly state which games are new in this work and which were introduced previously. Please clarify the relationship.
Circularity Check
No significant circularity: the A2C results are fresh empirical observations, and the self-citation to prior deceptive-game work supplies inputs rather than the paper's conclusions.
full rationale
This paper does not contain a derivation chain in which a predicted quantity is equivalent to an input by construction. The central claim, that A2C fails to find optimal policies on the four games, is supported by new training runs with fixed OpenAI Baselines hyperparameters, five random seeds, and 150 evaluation episodes per agent; none of these results are fitted parameters renamed as predictions. The deceptive games themselves come from Anderson et al. 2018, which overlaps in authorship, but that citation supplies game implementations and design motivations, not the conclusion that A2C is deceived. The 'optimal strategy' definitions are author-specified experimental targets, not quantities derived from the algorithm, so defining failure relative to them is a measurement design rather than a circular reduction. The paper's own Discussion concedes that 'numerous modifications' of deep RL could potentially solve these games, which narrows the scope of the generalization but does not make the empirical finding circular. The only self-citations are normal provenance for the GVGAI setup and prior deceptive-game benchmarks, and they are not load-bearing evidence for the observed A2C behavior. Accordingly, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- WaferThinMints mint limit =
9
- Invest banker payoff schedules =
Green: 3->5 after 30 ticks; Red: 7->15 after 60 ticks; Blue: 5->10 after 90 ticks
- DeceptiCoins level 3 size =
not specified numerically in the paper
- Training duration =
5,000,000 frames
assumptions (4)
- domain assumption A2C with Atari-style pixel inputs and default hyperparameters represents deep reinforcement learning broadly.
- domain assumption The four custom games are valid instantiations of the named deception categories and are not confounded by observation or stochasticity.
- domain assumption Video-based qualitative analysis is sufficient to infer learned behavior such as superstition.
- domain assumption Planning agents and A2C are comparable despite different resource budgets.
Cite this review
Pith. "Pith review of Superstition in the Network: Deep Reinforcement Learning Plays Deceptive Games." pith.science (2026). https://pith.science/paper/5OSDD2G6
@misc{pith2026190804436,
author = {Pith},
title = {Pith review of: Superstition in the Network: Deep Reinforcement Learning Plays Deceptive Games},
year = {2026},
howpublished = {\url{https://pith.science/paper/5OSDD2G6}},
note = {Machine review of arXiv:1908.04436}
}
read the original abstract
Deep reinforcement learning has learned to play many games well, but failed on others. To better characterize the modes and reasons of failure of deep reinforcement learners, we test the widely used Asynchronous Actor-Critic (A2C) algorithm on four deceptive games, which are specially designed to provide challenges to game-playing agents. These games are implemented in the General Video Game AI framework, which allows us to compare the behavior of reinforcement learning-based agents with planning agents based on tree search. We find that several of these games reliably deceive deep reinforcement learners, and that the resulting behavior highlights the shortcomings of the learning algorithm. The particular ways in which agents fail differ from how planning-based agents fail, further illuminating the character of these algorithms. We propose an initial typology of deceptions which could help us better understand pitfalls and failure modes of (deep) reinforcement learning.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Anderson, D.; Stephenson, M.; Togelius, J.; Salge, C.; Levine, J.; and Renz, J. 2018. Deceptive games. In International Conference on the Applications of Evolutionary Computation , 376--391. Springer
work page 2018
-
[3]
P.; Brundage, M.; and Bharath, A
Arulkumaran, K.; Deisenroth, M. P.; Brundage, M.; and Bharath, A. A. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34(6):26--38
work page 2017
-
[4]
G.; Naddaf , Y.; Veness , J.; and Bowling , M
Bellemare , M. G.; Naddaf , Y.; Veness , J.; and Bowling , M. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47:253--279
work page 2013
-
[5]
Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. Openai gym. arXiv preprint arXiv:1606.01540
arXiv 2016
-
[6]
Dhariwal, P.; Hesse, C.; Klimov, O.; Nichol, A.; Plappert, M.; Radford, A.; Schulman, J.; Sidor, S.; and Wu, Y. 2017. Openai baselines. https://github.com/openai/baselines
work page 2017
-
[7]
M.; Schaul, T.; Thompson, T.; and Togelius, J
Ebner, M.; Levine, J.; Lucas, S. M.; Schaul, T.; Thompson, T.; and Togelius, J. 2013. Towards a video game description language. In Dagstuhl Follow-Ups , volume 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik
work page 2013
-
[8]
Ecoffet, A.; Huizinga, J.; Lehman, J.; Stanley, K. O.; and Clune, J. 2019. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995
arXiv 2019
Show all 27 references
-
[9]
Hume, D. 1739. A Treatise of Human Nature . Oxford University Press
-
[10]
Justesen, N.; Bontrager, P.; Togelius, J.; and Risi, S. 2017. Deep learning for video game playing. arXiv preprint arXiv:1708.07902
2017 arXiv
-
[11]
R.; Bontrager, P.; Khalifa, A.; Togelius, J.; and Risi, S
Justesen, N.; Torrado, R. R.; Bontrager, P.; Khalifa, A.; Togelius, J.; and Risi, S. 2018. Procedural level generation improves generality of deep reinforcement learning. arXiv preprint arXiv:1806.10729
2018 arXiv
-
[12]
M.; Miikkulainen, R.; Schaul, T.; and Thompson, T
Levine, J.; Bates Congdon, C.; Ebner, M.; Kendall, G.; Lucas, S. M.; Miikkulainen, R.; Schaul, T.; and Thompson, T. 2013. General video game playing. Artificial and Computational Intelligence in Games
2013
-
[13]
B.; and Raskoff Zeiss, A
Mischel, W.; Ebbesen, E. B.; and Raskoff Zeiss, A. 1972. Cognitive and attentional mechanisms in delay of gratification. Journal of personality and social psychology 21(2):204
1972
-
[14]
A.; Veness, J.; Bellemare, M
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529
2015
-
[15]
P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K
Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , 1928--1937
2016
-
[16]
OpenAI. 2018. Openai five. https://blog.openai.com/openai-five/
2018
-
[17]
M.; and Schaul, T
Perez-Liebana, D.; Samothrakis, S.; Togelius, J.; Lucas, S. M.; and Schaul, T. 2016. General video game ai: Competition, challenges and opportunities. In Thirtieth AAAI Conference on Artificial Intelligence
2016
-
[18]
Rodriguez Torrado, R.; Bontrager, P.; Togelius, J.; Liu, J.; and Perez-Liebana, D. 2018. Deep reinforcement learning for general video game ai. In Computational Intelligence and Games (CIG), 2018 IEEE Conference on . IEEE
2018
-
[19]
Russell, B. 1912. The Problems of Philosophy . Williams and Norgate. chapter On Induction
1912
-
[20]
Schaul, T. 2013. A video game description language for model-based or interactive learning . In IEEE Conference on Computatonal Intelligence and Games, CIG
2013
-
[21]
Skinner, B. F. 1948. 'superstition'in the pigeon. Journal of experimental psychology 38(2):168
1948
-
[22]
S., and Barto, A
Sutton, R. S., and Barto, A. G. 1998. Reinforcement learning: An introduction . MIT press
1998
-
[23]
Sutton, R. S. 1992. Introduction: The Challenge of Reinforcement Learning . Boston, MA: Springer US. 1--3
1992
-
[24]
Togelius, J.; Schaul, T.; Wierstra, D.; Igel, C.; Gomez, F.; and Schmidhuber, J. 2009. Ontogenetic and phylogenetic reinforcement learning. K \"u nstliche Intelligenz 23(3):30--33
2009
-
[25]
Vinyals, O.; Babuschkin, I.; Chung, J.; Mathieu, M.; Jaderberg, M.; Czarnecki, W. M.; Dudzik, A.; Huang, A.; Georgiev, P.; Powell, R.; Ewalds, T.; Horgan, D.; Kroiss, M.; Danihelka, I.; Agapiou, J.; Oh, J.; Dalibard, V.; Choi, D.; Sifre, L.; Sulsky, Y.; Vezhnevets, S.; Molloy,...
2019
-
[26]
Wilson, D., and Sicart, M. 2010. Now it's personal: on abusive game design. In Proceedings of the International Academic Conference on the Future of Game Design and Technology , 40--47. ACM
2010
-
[27]
H., and Macready, W
Wolpert, D. H., and Macready, W. G. 1997. No free lunch theorems for optimization. IEEE transactions on evolutionary computation 1(1):67--82
1997
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.