{"id":"1f4a56f1-3018-4dff-a526-7ef9ee9f2b11","arxiv_id":"2505.21731","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Deep and symbolic RL agents fail to maintain performance on simplified versions of Atari training tasks, whereas humans adapt, revealing reliance on shortcuts.","lead":"After training on Atari games, reinforcement learning agents lose much of their score when the games are made simpler, while human players adapt easily. The paper introduces HackAtari, a suite of simplified game variations, as a diagnostic for whether agents learned the right strategies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'systematic' claim rests on a criterion-selected 15-of-224 subset, and the human simplification validation is order-confounded; random-sample re-evaluation is needed before accepting the central claim.","rationale":"The reader's weakest assumption—that the 15 human-validated variations are unrepresentative and selected around expected agent failures—is the same load-bearing concern I identify. The paper's own wording in Section 4 confirms that selection was conditioned on expected divergence, so the human study cannot independently validate the simplification premise for the full benchmark. Figure 3 also uses only 32 of 224 variants, so even the aggregate RL evidence is not a random or exhaustive sample. I am not claiming the paper is internally inconsistent; the issue is external validity of the headline 'systematic' claim. A random-sample re-evaluation would settle this directly: if agent drops persist on a representative sample and humans still adapt under a counterbalanced protocol, the central claim would be substantially strengthened. If not, the abstract and title should be narrowed to the selected variants. The benchmark itself remains a useful contribution, and the paper is honest about some limitations (e.g., lack of standardized difficulty scaling in Section 7), but the central human-agent divergence claim currently rests on a selection procedure that predicts the result. This does not change the reader's CONDITIONAL verdict: the paper should be revised to qualify the claim or provide the random-sample evidence, and the code/models should be released to make such checks possible.","tokens_in":33560,"tokens_out":6502,"duration_ms":74742,"concrete_test":"Select a random sample of, say, 40 HackAtari variants from the full 224, excluding the 15 used in Figure 4. Evaluate DQN, PPO, and IMPALA on original and modified tasks (30 episodes x 3 seeds). For a randomly chosen subset of 20 of these 40, run a pre-registered human study with counterbalanced order—half of participants see the modified task first—and at least 8 participants per variant, measuring raw-score change and normalized performance change. If the three algorithms show large negative performance change on a comparable fraction of the random sample and humans maintain or improve under counterbalancing, the 'systematic' claim is supported; if drops concentrate in the originally selected variants or human improvement disappears under counterbalancing, the central claim does not generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference—that performance drops on HackAtari variants reveal a consistent reliance on shortcuts—requires two premises: (i) the tested variants are genuine simplifications of the training task, and (ii) the tested variants are representative of HackAtari rather than chosen to make agents fail. Premise (i) is tested on only 15 of 224 variants, and Section 4 states these were selected 'for which most agents exhibit performance drops while we expect humans to improve or maintain their performances.' The human study therefore confirms the selection criterion; it does not validate the broader claim. Premise (ii) is unsupported: Figure 3 aggregates just 32 variants over 17 games, and no analysis is reported for the remaining ~192 variants. It may be true that most HackAtari variants are genuine simplifications, but a selection procedure conditioned on expected agent drops and expected human improvement can produce exactly the observed pattern even if RL agents generalize well on the majority of variations. Moreover, the human protocol always presents the original evaluation before the modified evaluation, with no counterbalancing, so human 'improvement' on the modified task is partly confounded with practice or fatigue. The load-bearing check is therefore whether agent drops and human-maintained performance reproduce on a random or held-out sample of HackAtari variants, not only on the criterion-selected subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces HackAtari, a RAM-based modification framework for the Arcade Learning Environment containing over 224 variations across more than 33 games, and uses it to argue that deep RL agents rely on shortcuts. The authors evaluate six deep RL algorithms and several object-centric agents on a subset of 17 games (32 variations), reporting that agents' performance drops on simplified variants while a human study of 128 participants on 15 game-variation pairs shows that humans mostly maintain or improve performance. They conclude that training and testing in the same environment is insufficient to obtain agents with human-like generalization, and they release the benchmark and promise to release code and checkpoints. The paper includes extended results in appendices, including raw scores for all agents and human participants.","tokens_in":33786,"tokens_out":5270,"duration_ms":54210,"significance":"If the central claim were established, this would be a useful benchmark and a cautionary result: simplification testing is underused in RL, and the open release of 224 RAM-based variations would support robustness evaluation. The paper's strengths include the use of IQM with bootstrap confidence intervals, multiple algorithms and seeds, and a reasonably sized human study with performance-contingent bonuses. However, the headline claim of systematic performance drops is not yet supported by the aggregated evidence, and the human validation is selection-confounded. The paper is best viewed as a benchmark resource plus a promising but incomplete diagnostic study; the central conclusion requires additional representative sampling and per-variation analysis before it can be accepted.","major_comments":[{"comment":"The 15 human-validated variations were explicitly selected \"for which most agents exhibit performance drops while we expect humans to improve or maintain their performances.\" Consequently, the human study confirms the selection criterion rather than validating the broader claim that HackAtari variations are simplifications. Since Figure 3 aggregates only 32 of the 224 variations and no random or held-out subset is analyzed, the paper's \"systematic\" conclusion is not established for the benchmark as a whole; a random sample of variations, or an explicit limitation of the claim to the selected subset, is required.","section":"Section 4 (Q2) and Appendix F.2"},{"comment":"The human protocol always presents the original game before the modified game (Free Training, Eval1 original, Eval2 modified), with no counterbalancing across participants. Human improvement in Eval2 is therefore confounded with practice or fatigue, and the paper itself invokes fatigue to explain the MsPacman drop. A counterbalanced order, or an analysis of score trajectories within each phase, is needed before human adaptation can be attributed to the simplification itself rather than to continued practice.","section":"Appendix F.2 and Section 4 (human protocol)"},{"comment":"Table 5 contains numerous agent-variation entries with large improvements, for example C51 on Asterix obelix (7,766 to 81,812), MDQN on Bowling top pins (33 to 181), and IMPALA on StarGunner static flyers (166,881 to 878,106), and Figure 3 reports that IMPALA maintains superhuman average performance. The aggregated IQM in Figure 3 can mask these positive cases, so the abstract's claim that agents \"systematically exhibit huge performance drops\" requires a per-variation distribution, such as the proportion of drops and the median drop, rather than a single aggregate over games.","section":"Table 5 and Figure 3"},{"comment":"The inference from performance drops to shortcut learning requires that the tested variations are genuine simplifications that preserve task semantics, but two of the 15 human-validated variations, BankHeist two police cars and MsPacman set level 1, produce human performance decreases or deep-agent scores near the random baseline; the text itself describes BankHeist as a decrease. These variations should be separated from strict simplifications, and the shortcut-learning interpretation should be applied only to variations with verified human non-decrease; otherwise the observed drop could reflect sensitivity to RAM-level distribution shift rather than shortcut reliance.","section":"Figures 4 and 5, Table 7"},{"comment":"The object-centric comparison in Figure 5 is confounded by training budget: Appendix B states that Binary Mask PPO and Planes PPO were trained for only 40 million frames, whereas the pixel PPO baseline and the deep agents were trained for 200 million frames. Since Figure 5 uses this comparison to conclude that object-centric inductive biases are insufficient, the training budgets must be matched or explicitly accounted for in the analysis.","section":"Appendix B and Figure 5"}],"minor_comments":[{"comment":"The main text says agents were evaluated \"on the 30 modifications,\" while the Figure 3 caption says \"32 task variations,\" and Appendix C mentions 50 game configurations; these counts should be harmonized.","section":"Section 4 and Figure 3 caption"},{"comment":"The caption says \"gameplay modifications (left)\" twice; the second \"left\" should likely be \"right,\" since the figure places visual perturbations on the left and gameplay modifications on the right.","section":"Figure 5 caption"},{"comment":"The main text says \"We randomly selected 134 users,\" while the preceding paragraph and Appendix F report 128 participants after exclusions; clarify the recruitment sample versus the analysis sample.","section":"Section 4 and Appendix F"},{"comment":"The HNS metric is defined with human and random reference scores from Badia et al. (2020a), but Table 7 also reports in-house random scores that differ substantially (e.g., Bowling random 63.12 versus Badia random 23.1); state explicitly which random baseline is used for normalization in each figure.","section":"Appendix A and Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a machine learning venue, and the HackAtari resource itself is valuable. The major comments above are fixable with additional experiments and re-analysis, but the central claim as stated is stronger than the current evidence supports. I would also ask the authors to provide the anonymized repository and model checkpoints during revision, since Section E promises release only after acceptance, which hampers reproducibility review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nQuick take: HackAtari is a genuinely useful resource, and the simplification axis is a nice complement to the usual robustness benchmarks. But the paper's headline claim—that RL agents 'systematically' fail on simpler tasks because they rely on shortcuts—is not supported by its own tables, and the human validation is selection-confounded. I'd send it to review, but it needs a serious rewrite of the claims.\n\nWhat's new: the first systematic ALE variation suite aimed at simplifications rather than complexifications. Over 200 RAM-based variations across 33 games, with code promised. The multi-agent evaluation (value-based, policy-gradient, IMPALA, object-centric, SCoBots) is broad, and the object-centric comparison is a useful data point. The human study on 15 games is a reasonable attempt to ground what counts as 'simpler.' That alone is worth having.\n\nWhere it's soft. First, the abstract says agents 'systematically exhibit huge performance drops,' but raw tables show several sizable improvements—C51 on Asterix obelix, MDQN on Bowling top pins, IMPALA on StarGunner static flyers—and IMPALA's aggregate stays superhuman on modified tasks. 'Systematic' is doing too much work. Second—and the stress-test note is right—the 15 variations for the human study were chosen, as Section 4 states, 'for which most agents exhibit performance drops while we expect humans to improve or maintain their performances.' So the human validation confirms the selection criterion; it does not validate the full suite. To support the central claim, the authors need to show that a random or held-out sample of HackAtari produces the same pattern, or at least report the full distribution of performance changes across all 224 variations. Third, the human protocol always presents original before modified, with no counterbalancing, so part of the human 'improvement' could be practice. Fixable but real. Finally, a performance drop on a RAM alteration is not direct evidence of 'shortcut learning'—it could be ordinary sensitivity to input distribution shift. The Pong prior work supports the shortcut story there; extending that inference to all these games needs more mechanism or more controls.\n\nWho it's for: anyone building or evaluating RL agents on ALE, and researchers working on generalization diagnostics. The benchmark deserves a serious referee: the resource is real and the direction is plausible, but the paper as written overclaims. I'd ask for the abstract to be calibrated, an unbiased or full-suite analysis, and counterbalanced human data before acceptance.","headline":"HackAtari is a useful new benchmark, but the 'systematic shortcut learning' headline is stronger than the paper's own tables support.","tokens_in":34349,"tokens_out":2679,"would_cite":true,"duration_ms":24941,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep RL agents systematically collapse on simpler versions of their training tasks, a new benchmark shows.","keywords":["deep reinforcement learning","generalization","shortcut learning","HackAtari","Arcade Learning Environment","task simplification","object-centric RL","human study"],"falsifier":"Run the same trained agents on a set of RAM perturbations that are equally large in RAM space but semantically meaningless for the game, such as randomizing unused memory cells or changing the color of irrelevant background objects. If the agents' normalized scores drop by the same magnitude on these controls as on the HackAtari simplifications, then the drops would be explained by input sensitivity rather than by reliance on task-specific shortcuts, contradicting the paper's interpretation.","tokens_in":33354,"feed_emoji":"🕹️","tokens_out":6374,"duration_ms":65460,"temperature":0.7,"pith_summary":"The paper introduces HackAtari, a suite of over 224 RAM-level modifications of Atari 2600 games, mostly simplifications that preserve the task's goal while making it easier. It argues that deep RL agents trained on the original games collapse on these simpler variants, while non-expert humans maintain or improve their scores on most tested variations. The performance drops are read as evidence that the agents learned shortcuts tied to the training environment rather than the underlying game logic. Object-centric agents are more robust to visual changes but still fail on gameplay changes. The paper concludes that matching human scores on the training environment is not evidence of human-like reasoning, and that evaluation on held-out task variations should be standard practice.","feed_headline":"RL agents flop on easier versions of their own games","feed_subtitle":"HackAtari benchmark: 224 simplified Atari variants, human-verified, expose shortcut-policy collapse.","key_machinery":"The load-bearing object is HackAtari, a benchmark of over 224 controlled variations of Arcade Learning Environment games, implemented by directly overwriting the games' Random Access Memory. RAM alteration is the mechanism because the original games are proprietary; it lets the authors remove or freeze enemies, change colors, stop cars, and alter physics in ways that preserve task semantics while reducing difficulty. The second load-bearing piece is the performance-change metric normalized against random scores, which measures an agent's relative drop from original to modified environment, together with a human study on 15 of the variations that validates the chosen variants as true simplifications.","core_discovery":"The central discovery is that task simplification, not complexification, exposes misalignment in deep RL. Using RAM alterations to make games objectively easier—removing enemies, stopping cars, aligning ice floes, shifting shields by a pixel—the paper finds that the aggregated normalized score of every evaluated deep agent drops substantially, often toward random, while humans improve or hold their performance on 13 of the 15 tested variations. Because the variations were chosen to be simplifications for humans, the agents' failure is interpreted not as an inability to cope with harder tasks but as reliance on shortcuts: they succeed in the original environment for the wrong reasons. The pattern holds across value-based, policy-gradient, and object-centric agents, with object-centric representations reducing sensitivity to visual changes but not resolving gameplay-logic failures. In-distribution evaluation with human-normalized scores is therefore insufficient to certify human-like intelligence.","pith_inferences":["The RAM-alteration technique could be reused to build semantic control conditions for other emulated benchmarks, separating shortcut reliance from mere input sensitivity.","Fine-tuning an agent on a single simplified variant and measuring how quickly it recovers could reveal whether failures come from missing features in the policy or from an inability to reuse learned skills.","The 224-variation suite could be distilled into a standardized, human-validated subset for reporting generalization, making agent comparisons more meaningful across studies."],"forward_implications":["A superhuman score on the original environment no longer counts as evidence of task understanding, because held-out simplifications can falsify it.","Evaluation protocols for general RL should include both complexifications and simplifications, since only simplifications clearly separate poor adaptation from misaligned policies.","Object-centric inductive biases improve robustness to visual changes but are not sufficient for generalization to gameplay-logic changes.","Agents that pass a broad set of semantically validated simplifications are more likely to be aligned with the intended task goals.","Reward maximization during training alone is not enough to produce policies that generalize the way humans do."],"supporting_citations":[{"why":"Introduces DQN and the claim that RL agents must generalize from past experience, which is the paper's target.","marker":"Mnih et al. (2015)"},{"why":"Defines the Arcade Learning Environment that HackAtari extends.","marker":"Bellemare et al. (2013)"},{"why":"Supplies the expert human scores used to compute Expert-Human Normalized Scores.","marker":"Badia et al. (2020a)"},{"why":"Shows DQN fails to zero-shot generalize to complexified tasks, the contrast case for the new simplification results.","marker":"Farebrother et al. (2018)"},{"why":"Defines shortcut learning, the phenomenon the paper uses to interpret performance drops.","marker":"Geirhos et al. (2020)"},{"why":"Provides the earlier Pong shortcut evidence and the SCoBots agent architecture evaluated in this study.","marker":"Delfosse et al. (2024b)"},{"why":"Supplies the IQM and bootstrap confidence interval methodology used to aggregate results.","marker":"Agarwal et al. (2021)"},{"why":"Sets the evaluation protocol with frameskip, sticky actions, and no-ops used for training and testing.","marker":"Machado et al. (2018)"},{"why":"Supplies the object-centric attention agents whose generalization is compared against deep baselines.","marker":"Blüml et al. (2025)"}],"fun_headline_variants":["Simplicity stumps deep RL agents","RL agents fail when games get easier","HackAtari: easier games drop RL performance","Shortcut reliance exposed by simplified Atari","Deep RL can't handle simpler versions of tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 15 human-tested variations are true simplifications that preserve the task's meaning, so the agents' performance drops on them indicate shortcut reliance rather than mere sensitivity to altered inputs.","fun_headline_variants_meta":{"raw":{"variants":["Simplicity stumps deep RL agents","RL agents fail when games get easier","HackAtari: easier games drop RL performance","Shortcut reliance exposed by simplified Atari","Deep RL can't handle simpler versions of tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1548,"prompt_tokens":854,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":470,"tokens_out":694,"duration_ms":7606,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:23:36.179627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trained agents on a set of RAM perturbations that are equally large in RAM space but semantically meaningless for the game, such as randomizing unused memory cells or changing the color of irrelevant background objects. If the agents' normalized scores drop by the same magnitude on these controls as on the HackAtari simplifications, then the drops would be explained by input sensitivity rather than by reliance on task-specific shortcuts, contradicting the paper's interpretation.","supporting_citations":[{"cited_title":"A., Veness, J., Bellemare, M","cited_arxiv_id":null,"evidence_quote":"Introduces DQN and the claim that RL agents must generalize from past experience, which is the paper's target."},{"cited_title":"C., and Bowling, M","cited_arxiv_id":null,"evidence_quote":"Shows DQN fails to zero-shot generalize to complexified tasks, the contrast case for the new simplification results."},{"cited_title":"S., Brendel, W., Bethge, M., and Wichmann, F","cited_arxiv_id":null,"evidence_quote":"Defines shortcut learning, the phenomenon the paper uses to interpret performance drops."},{"cited_title":"C., Bellemare, M","cited_arxiv_id":null,"evidence_quote":"Sets the evaluation protocol with frameskip, sticky actions, and no-ops used for training and testing."}],"review_version":1}