{"id":"8972faa1-fb16-4e2a-975c-7b5c544f5171","arxiv_id":"1908.04436","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A2C deep reinforcement learning reliably fails on four specially designed deceptive games, sometimes learning superstitious behaviors, and its failure modes differ from planning agents.","lead":"This paper tests A2C deep reinforcement learning on four games designed to trick game-playing agents, and finds the algorithm falls for the traps in most of them. It compares those failures with planning algorithms and sketches a typology of game deceptions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion that 'deep reinforcement learners are easily deceived' is not supported by evidence from a single algorithm (A2C) with one hyperparameter set; a second algorithm test would settle the scope.","rationale":"The reader identified the representativeness of A2C as the weakest assumption, and I agree: the paper's empirical base—one algorithm, one architecture, one hyperparameter set, five seeds, four reverse-designed games—cannot bear the weight of the phrase 'deep reinforcement learners are easily deceived.' The Methods are transparent ('Instead of tuning the algorithms for the games, we designed the games for the algorithms'), and the Discussion explicitly notes that newer deep RL modifications might solve these games, so the authors are aware of the scope problem. The conditional verdict is the right level of confidence: the within-experiment results are plausible (A2C scores in Table 1 are below the best planning agents on most games, and the Flower and Invest training curves show the predicted non-monotonic behavior), but the external generalization is not yet established. The proposed test—running a second major algorithm with standard settings—would directly settle whether the failure is an A2C quirk or a broader deep RL tendency. Adding variance statistics and quantitative behavioral traces would further harden the paper, especially for the superstition claim, but those are secondary to the generalization issue in the central conclusion.","tokens_in":10605,"tokens_out":7996,"duration_ms":88781,"concrete_test":"Train at least one independent deep RL algorithm (e.g., PPO or DQN) on the same four GVGAI games with a standard, untuned-for-these-games configuration, using a comparable training budget and at least five seeds, and report per-seed final scores with mean and standard deviation. If the second algorithm also fails to reach the optimal policy on the same games, the conclusion that deep RL is deceived gains support; if it solves Invest or Flower, the claim must be narrowed to A2C or to this A2C configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that deep reinforcement learners are easily deceived—rests on a single model-free algorithm, A2C, trained with one OpenAI Baselines configuration, five random seeds, and no reported variance in Table 1. A2C is known to be sensitive to hyperparameters and architecture choices, so a failure of this particular configuration does not establish a general property of deep RL. The paper's own Discussion concedes that 'numerous modifications' of deep RL could potentially solve these games, which undercuts the breadth of the Conclusion's wording. The four games are bespoke and reverse-designed for this setup, making the result a valid demonstration that this A2C configuration falls for these reward structures, but not a demonstration about the class of deep reinforcement learners. In addition, the most distinctive subclaim—the corner-waiting 'superstition' in Invest—is inferred from video inspection without quantitative position traces or waiting-time statistics, so it cannot be independently evaluated from the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies failure modes of deep reinforcement learning by introducing four 'deceptive games' implemented in the GVGAI framework: DeceptiCoins, WaferThinMints, Flower, and Invest. The authors train A2C (OpenAI Baselines, default Atari-style hyperparameters, five seeds, five million frames) on these games, evaluate each agent on 150 episodes, and compare the resulting average scores against six planning agents. They report that A2C converges to suboptimal policies in most of the games, including a 'superstitious' corner-waiting behavior in Invest, and propose an initial typology of deceptions: lack of hierarchical understanding, subverted generalization, delayed reward, and delayed gratification. The central qualitative claim is that deep reinforcement learners are easily deceived by reward structures that lead away from optimal policies.","tokens_in":10786,"tokens_out":3114,"duration_ms":35088,"significance":"If the empirical findings are supported, this is a useful contribution to the growing literature on failure modes of deep RL. The paper's strengths are its use of a consistent evaluation setup across four games, a comparison with six planning agents that differ in mechanism, and a concrete proposal of deception categories that could guide future benchmark design. The reported consistency across five training seeds is encouraging. However, the paper's central conclusion is broader than its evidence: only one algorithm (A2C) with one hyperparameter configuration is tested, the quantitative results lack variance measures and statistical tests, and the most distinctive behavioral claims are based on qualitative video inspection. These issues need to be addressed before the general claim about 'deep reinforcement learners' can be accepted.","major_comments":[{"comment":"The conclusion that 'deep reinforcement learners are easily deceived' is not supported by evidence from a single algorithm, A2C, with one default hyperparameter configuration (Methods: 'Instead of tuning the algorithms for the games, we designed the games for the algorithms'). The Discussion itself concedes that 'numerous modifications' of deep RL could potentially solve these games. Please either test at least one additional algorithm (e.g., DQN or PPO) under comparable conditions, or narrow the claim in the abstract and conclusion to 'this A2C configuration is easily deceived by these games.'","section":"Conclusion / Abstract"},{"comment":"Table 1 reports average scores across evaluation runs, but no error bars, standard deviations, per-seed breakdowns, or statistical tests are provided for any algorithm. This makes it impossible to assess whether the A2C scores (e.g., DeceptiCoins level 2: 3.79 vs. optimal 5.0; Invest: 69.6 vs. Return42's 190.12) are reliable differences or within-run noise. Please report variance measures and clarify how the five independently trained A2C agents are aggregated into the single A2C row.","section":"Table 1 / Comparison with planning algorithms"},{"comment":"The 'superstition' claim is load-bearing for the paper's title and discussion, but it rests entirely on qualitative video inspection. The text states that the agent 'runs to the far right corner and waits' and that the time to reach the corner 'correlates with the arrival of the delayed reward,' yet no quantitative evidence is provided. Please include position-over-time traces, waiting-time statistics, and, for all five seeds, a demonstration that the corner waiting is temporally correlated with reward payout. Without such data, the claim cannot be independently evaluated.","section":"Invest / Results"},{"comment":"The paper explicitly states that WaferThinMints 'did not converge' and 'might have benefited from more training time.' Interpreting this as evidence of deception is undercut by the acknowledged non-convergence: a truncated training run may fail for reasons unrelated to the game's reward structure. Please provide a convergence criterion, longer training results, or an explicit analysis of learning dynamics. Similarly, the Flower claim that 'the agent gets worse over time' is supported only by a training curve that is not shown; include the training curves with error bars for all seeds.","section":"WaferThinMints / Results and Methods"}],"minor_comments":[{"comment":"The paper does not provide code, game descriptions, or evaluation scripts. Since the central evidence is empirical and behavioral, releasing the VGDL game files and the A2C evaluation code would substantially strengthen the work and allow replication.","section":"Methods / Reproducibility"},{"comment":"The comparison between A2C and planning agents is described as 'apples and oranges,' but the paper does not specify how much computation the planning agents are allowed per episode. A sentence clarifying the total planning budget relative to the RL training budget would help readers interpret the comparison.","section":"Comparison with planning algorithms"},{"comment":"The optimal scores for DeceptiCoins are not given explicitly. Reporting the maximum achievable score for each level would make Table 1 more interpretable, because the reader cannot tell whether a score of 2.0 out of a possible 5.0 represents a partial or total failure.","section":"Deceptive Games / DeceptiCoins"},{"comment":"The reference to Anderson et al. 2018 is cited for the previous deceptive-games suite, but the current paper does not clearly state which games are new in this work and which were introduced previously. Please clarify the relationship.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but its broad title-level claim 'Superstition in the Network' and the abstract's 'deep reinforcement learners are easily deceived' exceed what can be concluded from a single algorithm with one configuration and no variance reporting. The missing quantitative support for the superstition claim is especially important given the paper's framing. I would be comfortable with a revised version that narrows the claims, adds variance measures and quantitative behavioral traces, and ideally includes a second algorithm. The current version is more suitable for a workshop than for a main-track journal unless these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper shows, convincingly, that one specific deep RL algorithm—A2C with default Atari hyperparameters—falls for four hand-designed reward traps, including a genuinely odd corner-waiting superstition in Invest. That is what the evidence supports. It does not support the broader headline that \"deep reinforcement learners are easily deceived,\" and the paper's own Discussion quietly walks that back by noting that numerous RL modifications could potentially solve these games.\n\nWhat is actually new: the A2C results are absent from Anderson et al. 2018, which only tested planning agents. The four-category typology is a useful frame, even if the categories overlap a bit. Credit where due: the games are reverse-designed for the algorithm rather than tuning the algorithm to the games, which is a legitimate benchmark methodology. Five training seeds and 150 evaluation runs per game give the qualitative failure modes some solidity. The corner-waiting superstition, observed across all five seeds, is the most interesting thing in the paper—it is a clean behavioral signature of a credit-assignment problem, and the Skinner pigeon analogy is apt. The comparison with planning agents is honest, including the apples-to-oranges caveat about forward models and training time.\n\nSoft spots, in rough order. First, the generalization from A2C to \"deep RL\" is a stretch: this is one algorithm, one architecture, one hyperparameter set, and A2C is known to be sensitive to those choices. The stress-test note is right here. Second, the superstition claim rests on video inspection rather than quantitative position traces or waiting-time statistics; I believe it, but the paper asks the reader to trust the authors' eyes. Third, there are no error bars, no significance tests, no code or data release, and WaferThinMints never converged. Those are real presentation gaps, not evidence that the observations are fabricated. The Flower result—score declining as the agent gets faster at collecting—is plausible and actually a nice example of a competence trap, but it is read off training curves without showing variance.\n\nI think the central qualitative result holds: A2C reliably falls for delayed-gratification and delayed-reward structures in these games, and its failure modes differ from planners. The work is not circular, because the games come from prior work but the A2C observations are independent. The scope claim needs reining in, ideally by adding a second algorithm (DQN or PPO) and releasing code or quantitative behavioral metrics.\n\nThis paper deserves serious review. It is a useful cautionary example for anyone working on RL evaluation, reward design, or safety. I would send it to a competent referee for revision rather than desk reject it, and I would bring it to a reading group as a short, discussable case study. I would cite it in my own work as evidence about this A2C configuration, not about deep RL as a whole.","headline":"A solid, modest empirical study of one A2C configuration on four hand-designed deceptive games, with a genuinely interesting superstition result, but the title and conclusion overgeneralize from A2C to deep RL as a class.","tokens_in":11316,"tokens_out":2219,"would_cite":true,"duration_ms":26049,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep reinforcement learner, A2C, misses optimal play on nearly every level of four deceptive games and invents a superstitious ritual.","keywords":["deep reinforcement learning","A2C","deceptive games","reward structure","credit assignment","delayed reward","superstition in learned policies","general video game playing"],"falsifier":"Train A2C on DeceptiCoins level 3 for fifty million frames instead of five million. If it then consistently takes the higher-reward right-side path, the reported failure is a training-budget artifact rather than a robust deception. Likewise, give A2C a direct encoding of the avatar's fullness meter in WaferThinMints; if that alone yields safe mint collection, then that deception depends on the pixel-only representation rather than the reward structure itself.","tokens_in":10412,"feed_emoji":"🎮","tokens_out":10284,"duration_ms":100508,"temperature":0.7,"pith_summary":"This paper tries to establish that deep reinforcement learning can be reliably deceived by the reward structure of an environment, not just by sparse rewards or partial observability. To show this, the authors designed four small games—DeceptiCoins, WaferThinMints, Flower, and Invest—each built around a distinct way of tempting an agent away from optimal play, and trained a widely used model-free actor-critic algorithm, A2C, on pixels. They report that A2C fails to find the optimal policy in every game except one level of one game, and that watching the trained agents reveals the mechanism of failure. The paper also proposes an initial typology of deceptions: lack of hierarchical understanding, subverted generalization, delayed reward, and delayed gratification. If the claim is right, benchmark results on standard arcade games overstate how robust current reward-driven learning is.","feed_headline":"Reward-driven AI fails four deceptive games and learns a superstition","feed_subtitle":"A standard actor-critic agent misses the optimal policy on every game but one level; its waiting ritual is pure learned superstition.","key_machinery":"The load-bearing object is the deceptive game suite itself, combined with the A2C training protocol. Each of the four games is a minimal environment whose reward schedule punishes the very assumption a reward-driven learner must make: that immediate positive signals can be trusted as guides. DeceptiCoins uses an irreversible early choice, WaferThinMints makes the same item switch from positive to lethal, Flower makes fast collection destructive, and Invest makes the agent incur a penalty before a delayed payoff. The machinery does its work by making the failure visible and repeatable: the same superstitious corner-waiting behavior appears across independent training runs in Invest, and the Flower agent's performance declines as it gets better. Comparing these behaviors with forward-model planners isolates which failures come from the learning rule rather than from the games themselves.","core_discovery":"The central discovery is that a standard deep reinforcement learner, A2C, reliably misses the optimal policy in four deceptive games, and the way it fails is informative. In DeceptiCoins, the agent converges to a suboptimal path once the level is large enough; in WaferThinMints, it learns that mints are good but cannot master the rule that the tenth mint kills it, and the stochastic mint-placing waiter keeps the game from converging; in Flower, the agent's improving skill backfires because collecting seedlings prevents them from maturing, so its score drops as training progresses; and in Invest, it learns to invest only with the quickest-paying banker and then runs to a fixed corner to wait for the payout, an arbitrary ritual the authors call superstitious. The paper argues these four failure modes correspond to distinct deceptions—lack of hierarchical understanding, subverted generalization, delayed reward, and delayed gratification—and uses six planning agents that rely on a forward model as a comparison to show that model-free learning fails differently from planning with a forward model.","pith_inferences":["A natural extension is to test value-based methods and recurrent policies on the same four games; if they also fail, the result generalizes from actor-critic training to model-free deep RL broadly, while if they succeed it means the deception is specific to A2C's update rule or architecture.","The superstition in Invest suggests a cheap diagnostic for causal confusion: remove the spatial cue by making the waiting spot irrelevant, and see whether the learned policy collapses or adapts; this would test whether the policy truly depends on the spurious location.","The delayed-reward game could be made progressively harder by randomizing the investment payoff time, which would tell whether the agent can learn to wait at all or would simply develop longer and more elaborate superstitions.","In real deployments where rewards are naturally delayed—such as medical or financial decisions—this paper implies that a model-free agent trained on reward alone may lock onto accidental correlates and should be audited for behavior that has no causal link to outcomes."],"forward_implications":["If one standard model-free deep RL algorithm can be reliably deceived by such simple reward structures, then strong performance on standard arcade benchmarks is not evidence that a method handles delayed or trap-like rewards.","The Flower result implies that training can make a reward-driven agent monotonically worse: improving its ability to collect flowers destroys the flowers' growth, so competence and score move in opposite directions.","The Invest result implies that deep RL policies can encode stable but causally false associations, since separate training runs produced the same arbitrary waiting-spot ritual.","Because planning agents with forward models fail on different games than the learning agent does, deceptive games can separate failures of credit assignment from failures of search or representation.","The proposed typology gives algorithm builders four concrete stressors—hierarchical abstraction, generalization under changing effects, delayed reward, and delayed gratification—to test before deploying an agent in an environment with adversarial or natural reward traps."],"supporting_citations":[{"why":"Defines the deceptive games and reports how planning agents fail on them, providing the suite and comparison baseline this paper extends.","marker":"[Anderson et al. 2018]"},{"why":"Introduces the asynchronous actor-critic method and the network architecture that A2C uses here.","marker":"[Mnih et al. 2016]"},{"why":"Establishes the pixel-based deep reinforcement learning training setup for general video game AI that the authors reuse without tuning.","marker":"[Rodriguez Torrado et al. 2018]"},{"why":"Supplies the animal-learning concept of superstition that the paper uses to interpret the agent's arbitrary waiting behavior in Invest.","marker":"[Skinner 1948]"},{"why":"Provides the reinforcement-learning formalism and the credit-assignment problem that the deceptive reward delays exploit.","marker":"[Sutton and Barto 1998]"},{"why":"Gives the no-free-lunch rationale for why algorithms tailored to a problem class are vulnerable when their implicit assumptions are subverted.","marker":"[Wolpert and Macready 1997]"},{"why":"Provides the marshmallow-experiment analogue that frames the delayed-gratification failure in Flower.","marker":"[Mischel, Ebbesen, and Raskoff Zeiss 1972]"}],"fun_headline_variants":["AI learns superstition in deceptive games, misses optimal play","Actor-critic agent tricked by deceptive games into rituals","Reinforcement learning exposed: four deceptive games beat A2C","Deep RL fails deceptive games, picks up superstitious habits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion would collapse if a different deep RL configuration did not fail: the paper uses one algorithm—A2C with a fixed architecture, default hyperparameters, and five million training frames—and assumes the four games isolate the intended deceptions rather than confounding pixel observation or randomness.","fun_headline_variants_meta":{"raw":{"variants":["AI learns superstition in deceptive games, misses optimal play","Actor-critic agent tricked by deceptive games into rituals","Reinforcement learning exposed: four deceptive games beat A2C","Deep RL fails deceptive games, picks up superstitious habits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":2969,"prompt_tokens":905,"completion_tokens":2064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1994}},"tokens_in":521,"tokens_out":2064,"duration_ms":13542,"temperature":1.0,"reasoning_tokens":1994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:42:12.047548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train A2C on DeceptiCoins level 3 for fifty million frames instead of five million. If it then consistently takes the higher-reward right-side path, the reported failure is a training-budget artifact rather than a robust deception. Likewise, give A2C a direct encoding of the avatar's fullness meter in WaferThinMints; if that alone yields safe mint collection, then that deception depends on the pixel-only representation rather than the reward structure itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the deceptive games and reports how planning agents fail on them, providing the suite and comparison baseline this paper extends."},{"cited_title":"P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K","cited_arxiv_id":null,"evidence_quote":"Introduces the asynchronous actor-critic method and the network architecture that A2C uses here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the pixel-based deep reinforcement learning training setup for general video game AI that the authors reuse without tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the animal-learning concept of superstition that the paper uses to interpret the agent's arbitrary waiting behavior in Invest."},{"cited_title":"S., and Barto, A","cited_arxiv_id":null,"evidence_quote":"Provides the reinforcement-learning formalism and the credit-assignment problem that the deceptive reward delays exploit."},{"cited_title":"H., and Macready, W","cited_arxiv_id":null,"evidence_quote":"Gives the no-free-lunch rationale for why algorithms tailored to a problem class are vulnerable when their implicit assumptions are subverted."},{"cited_title":"B.; and Raskoff Zeiss, A","cited_arxiv_id":null,"evidence_quote":"Provides the marshmallow-experiment analogue that frames the delayed-gratification failure in Flower."}],"review_version":1}