{"id":"94a3b0e7-e685-4705-b827-0f38dfc0a3cf","arxiv_id":"2507.00030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A DQN augmented with a contextual bandit that selects action durations is reported to outperform fixed and discrete action-repeat baselines on five Atari games.","lead":"This paper adds a contextual bandit module to a DQN so the agent chooses both an action and how many frames to repeat it, and reports higher Atari scores on 3 of 5 games. The supporting evidence is a single table without error bars or code, and one analysis table contains internally inconsistent percentages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's duration percentages sum to >100% for three of five games, making the empirical results physically impossible and the central performance claim unsubstantiated.","rationale":"I read the paper as a claim that a contextual-bandit module for action duration improves DQN on Atari. The only evidence is Tables 1 and 2. Table 1 is the primary evidence; Table 2 is meant to explain the mechanism. The percentages in Table 2 cannot be correct as stated, because the three listed buckets are disjoint and their percentages exceed 100 for the majority of games. This indicates a data handling error or fabrication, and it undermines the trustworthiness of the entire experimental section. Without reliable numbers, there is no basis for the central claim. The reader's weakest_assumption about Q-value-difference bias is a theoretical concern, but it is secondary: even a theoretically flawed reward could in principle produce real improvements, while impossible statistics make the reported improvements unverifiable. My proposed check—recomputing the percentages from raw logs—directly tests the integrity of the data. If the error is just a typo in the table, the authors can correct it; if not, the result is unsupported. Given the paper provides no code, seeds, or error bars, the internal inconsistency tips the balance toward rejection rather than conditional acceptance.","tokens_in":3453,"tokens_out":6744,"duration_ms":59453,"concrete_test":"Ask the authors to release the raw per-episode logs (or re-run the experiments) and recompute the duration percentages for each game. Specifically, compute the fraction of all selected durations that fall into the 1–5, 6–7, and 8–11 frame buckets. For each game, the three fractions plus the fraction for durations 12–20 must sum to 1. If the recomputed fractions still yield sums of 1.10, 1.04, and 1.10 for Seaquest, Enduro, and Q*Bert, the reported table is impossible and the empirical claim lacks support. Additionally, verify that Table 1's scores can be reproduced from the raw episode returns with at least 10 independent training seeds.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that Bandit-DQN outperforms static and dynamic ARR baselines—rests entirely on the experimental results in Section 4. Table 2 (Section 4.2) reports the percentage of short (1–5), medium (6–7), and long (8–11) durations selected by Bandit-DQN. These three non-overlapping categories partition the reported duration range, so the entries must sum to 100% within rounding error. However, Seaquest sums to 110% (54+30+26), Enduro to 104% (34+25+45), and Q*Bert to 110% (49+30+31). Only Space Invaders and Alien sum to 100%. This is not a rounding artifact; it is a mathematical impossibility for a distribution. It indicates that the reported duration statistics were not actually computed from the agent's behavior, which casts serious doubt on the accuracy of the scores in Table 1. If the analysis table is unreliable, the reported performance improvements—and therefore the paper's main contribution—are not credible. Even if the bandit reward signal were theoretically sound, the empirical evidence currently on the page cannot support the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Bandit-DQN, a DQN extension that uses a contextual bandit to select an action duration d in {1,...,d_max} for every selected action, with the bandit trained by policy gradient on a reward equal to the change in the DQN's own Q-values. The method is evaluated on five Atari 2600 games against static action repetition rates and a dynamic frameskip baseline (DFDQN), and the authors claim significant performance improvements. The central contribution is empirical: an adaptive duration selection mechanism that is claimed to outperform static and dynamic baselines on Seaquest, Space Invaders, Enduro, Alien, and Q*Bert.","tokens_in":3697,"tokens_out":3749,"duration_ms":37600,"significance":"If the empirical claims were properly supported, adaptive action-duration selection would be a useful direction for improving sample efficiency and temporal control in deep RL, and the contextual-bandit formulation is a natural extension of dynamic action repetition. However, the paper provides no theoretical analysis, no code, no error bars or multiple seeds, and the reported duration statistics in Table 2 are internally inconsistent. The main evidence for the headline claim is therefore not credible as presented. The paper does identify a relevant problem and proposes a simple, potentially useful architecture; those strengths are acknowledged, but they do not compensate for the current lack of reliable experimental support.","major_comments":[{"comment":"The three duration categories Short (1-5), Medium (6-7), and Long (8-11) partition the integer durations 1-11, so the percentages for each game must sum to 100% within rounding error. However, Seaquest sums to 54+30+26=110%, Enduro to 34+25+45=104%, and Q*Bert to 49+30+31=110%. This is mathematically impossible for a distribution over disjoint categories and indicates that the reported statistics were not actually computed from the agent's behavior. The table also omits the range 12-20 entirely, even though d_max=20, with no explanation. Because this analysis table is used to explain the performance gains, its unreliability directly undermines the empirical claims in Table 1.","section":"Section 4.2, Table 2"},{"comment":"The Q*Bert row reports Bandit-DQN at 3500 versus DFDQN at 1150, a 204% improvement that is wildly larger than the improvements in any other game. No mechanism for this outlier is given, no error bars or standard deviations are reported, and the number is consistent with a typo. Since Table 1 is the only evidence for the central claim that Bandit-DQN outperforms all baselines, the authors must provide raw per-seed results, standard deviations, and statistical tests across at least five independent runs before the claim can be evaluated.","section":"Section 4.1, Table 1"},{"comment":"The bandit reward rb = Q(s_{t+dt}, a'; theta) - Q(s_t, a_t; theta) is a difference of Q-value estimates from the same network that is being trained. This objective is not the environment reward, is not discounted, and has no theoretical guarantee of correlating with true return. During early training Q-values are inaccurate, and because theta changes simultaneously, the policy-gradient update on the bandit is biased with respect to any fixed reward signal. The paper provides no ablation comparing this heuristic with alternatives (e.g., using the environment reward accumulated over the duration), so the reported improvements could result from exploiting Q-value estimation error rather than from better temporal control. This is a load-bearing assumption and needs either a proof of correctness or an ablation that separates the effect of the reward signal from the effect of adaptive durations.","section":"Section 3.1, Eq. (1)"},{"comment":"The text states that the authors 'report the best testing epoch score.' Choosing the maximum testing score across training epochs is a form of test-set selection that can substantially inflate reported performance and makes comparisons across methods unreliable, especially without multiple seeds. Combined with the absence of error bars, statistical tests, and a fixed evaluation protocol, the claim that 'Bandit-DQN outperforms all baselines' is not statistically substantiated. The authors should report the mean and standard deviation over seeds, ideally with a fixed policy snapshot or a final-policy evaluation.","section":"Section 4, Experimental protocol"}],"minor_comments":[{"comment":"The notation 'd∈Z+' should be replaced with 'd ∈ {1,...,d_max}' to match the actual arm set, and the probability notation should be typeset as π_b(d_t | s_t; θ_b).","section":"Section 3.1"},{"comment":"The transition stored in the replay memory includes d_t, but the DQN update described on line 11 does not use d_t in any way. Please clarify how the chosen duration affects the Q-learning loss, or remove d_t from the stored tuple if it is not used.","section":"Algorithm 1, line 10"},{"comment":"The abstract says the method offers 'finer-grained temporal control without predefined rates,' but the durations are still drawn from a discrete set {1,...,20}. Please rephrase to avoid implying a continuous range.","section":"Abstract and Section 2"},{"comment":"The sentence 'Our work is the first to apply contextual bandits to action duration selection in DRL' is a strong novelty claim without a comprehensive literature survey. Please soften it or provide more evidence that no prior work exists.","section":"Section 2"},{"comment":"Several references use 'and others' instead of listing authors, and reference [8] is described only by a title with no clear venue. Please complete the reference information; also, [5] is a news-recommendation paper and is not directly an RL method, so consider citing more relevant contextual-bandit RL literature.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is empirical, but the experimental section as written contains a mathematically impossible table and an unexplained outlier. These are fixable with a rerun and proper statistics, so rejection is not warranted, but the authors must supply correct data, multiple seeds, and an ablation for the bandit reward. The novelty claim about being 'first' should also be checked carefully given the related literature on dynamic action repetition and temporal abstractions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea—letting a contextual bandit pick action durations on top of DQN, with a Q-value-difference reward—is a reasonable incremental extension of DAR. It's not a paradigm shift, but it's a clean combination of existing components, and the paper is clearly written and honestly positioned against the DAR baseline. The Q-value-difference reward is a genuine heuristic that could work in practice, though it has no theoretical grounding.\n\nThe problem is the evidence. Table 2 reports the percentage of short, medium, and long durations selected by Bandit-DQN. These categories are non-overlapping and should sum to 100% within rounding. For Seaquest they sum to 110%, Enduro to 104%, Q*Bert to 110%. That is mathematically impossible for a distribution. It suggests the table was not actually computed from agent behavior, and that casts doubt on the scores in Table 1. The Q*Bert score is also an outlier (3500 vs. 1150 for DFDQN) with no explanation. There are no error bars, no seeds, no statistical tests, and no code release. For a paper whose contribution is empirical, this is a load-bearing flaw.\n\nI want to be fair: the idea deserves a serious look, and the flaws are fixable in principle. The authors should release code, run multiple seeds, report variance, correct the table, and clarify why durations 12–20 (given d_max=20) are absent from the analysis. They should also address the bandit's reliance on potentially noisy Q-values early in training.\n\nWho is this for? Someone working on temporal abstraction in DRL might find the bandit formulation interesting, but only after the empirical claims are made credible. I'd send it to peer review rather than desk-reject, because the mechanism is novel enough and the problems are correctable. But I'd make it clear that the current data cannot support the reported gains.","headline":"The bandit-based duration selection is a plausible incremental idea, but the empirical section as printed is not credible: Table 2's percentages sum to over 100% for three games, which is impossible for a distribution.","tokens_in":4179,"tokens_out":1771,"would_cite":false,"duration_ms":19051,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes adding a contextual bandit to DQN to learn how many frames each action should be repeated, and reports that this adaptive duration selection beats static and discrete repetition baselines on three Atari games.","keywords":["contextual bandits","action repetition","temporal abstraction","deep reinforcement learning","DQN","Atari 2600","adaptive action duration"],"falsifier":"One concrete test is to train the same architecture but replace the bandit reward with the actual return difference $r_t + \\gamma \\max_a Q(s_{t+d_t}, a) - Q(s_t, a_t)$; if the score gains vanish or reverse, the Q-difference signal is the source of the improvement. A second, simpler falsifier is to run the full method but sample durations uniformly at random from the same learned marginal distribution; if performance does not drop, then the state-dependent duration choices are not doing the work.","tokens_in":3265,"feed_emoji":"🎮","tokens_out":3309,"duration_ms":49295,"temperature":0.7,"pith_summary":"The paper claims that letting an agent choose, per state, how many frames to repeat an action improves performance in dynamic environments. It augments a Deep Q-Network with a contextual bandit module that outputs a distribution over durations from 1 to 20 frames, trained on the change in the DQN's own Q-values after executing that duration. On five Atari 2600 games, the method reportedly outperforms DQN with fixed frame skips and a discrete dynamic repetition baseline, including 15% and 10% score gains over DFDQN in Seaquest and Enduro. The significance is that temporal control emerges from state context rather than manual or fixed settings, which matters for real-time applications.","feed_headline":"Bandit learns how long to hold each action, beats fixed Atari baselines","feed_subtitle":"Adding a contextual bandit to DQN to pick frame counts per state improves scores in Seaquest, Space Invaders, and Enduro.","key_machinery":"The central object is a contextual bandit module attached to DQN that outputs a probability distribution over durations $d \\in \\{1,\\dots,d_{\\max}\\}$ conditioned on the state, with $d_{\\max}=20$. It is trained by policy gradient on the reward $r_b = Q(s_{t+d_t}, a'; \\theta) - Q(s_t, a_t; \\theta)$, the change in the DQN's predicted Q-value after executing action $a_t$ for $d_t$ frames. This Q-difference signal is what lets the bandit learn which durations are useful in different states without external labels, and it is the mechanism that carries the argument from architecture to reported performance.","core_discovery":"The central claim is that a contextual bandit can learn action durations and that this adaptive repetition improves over fixed and discrete repetition baselines. The paper demonstrates this in Atari 2600 games, where Bandit-DQN achieves higher average episode scores than DQN with ARR=4, DQN with ARR=20, and DFDQN in Seaquest, Space Invaders, and Enduro. The paper interprets the learned duration distributions as evidence of game-specific temporal strategies, such as short durations in Space Invaders for quick reflexes and long durations in Enduro for sustained driving, explaining the performance gains.","pith_inferences":["The Q-difference reward is a self-referential signal: early in training, when Q-values are inaccurate, the bandit may reinforce estimates that are wrong in the same way, so the reported improvements could partly stem from exploiting Q-value noise rather than true temporal benefits.","The Q*Bert result is an outlier: the score jumps from 1150 (DFDQN) to 3500, far larger than gains in other games and not explained by the modest duration distribution shift, suggesting a possible evaluation variance or a non-temporal mechanism.","A testable extension is to replace the bandit reward with the actual return difference $r_t + \\gamma \\max_a Q(s_{t+d}, a) - Q(s_t, a_t)$; if the method's advantage disappears, the Q-value-change signal is the critical component, not the adaptivity itself.","The discrete duration set (1 to 20) is still a coarse approximation of continuous time; extending the bandit to output a continuous duration or to modulate the frame count mid-action could be a natural next step."],"forward_implications":["If the reported gains hold, adaptive duration selection is a lightweight extension to DQN that improves scores while potentially reducing computation by repeating actions that do not need fine-grained control.","The learned duration distributions imply that agents can acquire game-specific temporal policies, such as short bursts for shooting and long stretches for driving, without explicit supervision.","The approach extends Dynamic Action Repetition by replacing a discrete set of repetition rates with a range of durations, which may transfer to tasks where optimal durations vary continuously.","The method is directly compatible with other DQN variants and could be ported to policy-based algorithms, as the paper suggests for A3C."],"supporting_citations":[{"why":"Provides the Arcade Learning Environment used to evaluate all methods.","marker":"[1]"},{"why":"Establishes that frame skip rates significantly affect Atari performance, motivating action repetition as a design parameter.","marker":"[2]"},{"why":"Introduces Dynamic Action Repetition (DFDQN), the main baseline that the proposed Bandit-DQN must beat, and supplies the experimental setup with d_max=20.","marker":"[4]"},{"why":"Supplies the contextual-bandit formulation that the duration-selection module is modeled on.","marker":"[5]"},{"why":"Defines the DQN algorithm and architecture that the paper augments, and is the source of the Q-values used to compute the bandit reward.","marker":"[6]"}],"fun_headline_variants":["Contextual bandit tunes action duration in DQN, bests Atari baselines","Adaptive action repetition boosts DQN on Atari games","Bandit-driven action durations improve deep RL in dynamic tasks","DQN with learned action lengths beats static repetition in Atari","Action duration via contextual bandit lifts DRL scores on Atari"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bandit is trained on the change in the DQN's own Q-values, and this signal is assumed to be a reliable indicator of how good a chosen duration is, but during early training these Q-values are inaccurate and the bandit may chase estimation errors instead of true returns.","fun_headline_variants_meta":{"raw":{"variants":["Contextual bandit tunes action duration in DQN, bests Atari baselines","Adaptive action repetition boosts DQN on Atari games","Bandit-driven action durations improve deep RL in dynamic tasks","DQN with learned action lengths beats static repetition in Atari","Action duration via contextual bandit lifts DRL scores on Atari"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2268,"prompt_tokens":816,"completion_tokens":1452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":1360}},"tokens_in":432,"tokens_out":1452,"duration_ms":10508,"temperature":1.0,"reasoning_tokens":1360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:08:43.814754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test is to train the same architecture but replace the bandit reward with the actual return difference $r_t + \\gamma \\max_a Q(s_{t+d_t}, a) - Q(s_t, a_t)$; if the score gains vanish or reverse, the Q-difference signal is the source of the improvement. A second, simpler falsifier is to run the full method but sample durations uniformly at random from the same learned marginal distribution; if performance does not drop, then the state-dependent duration choices are not doing the work.","supporting_citations":[{"cited_title":"Bellemare and others","cited_arxiv_id":null,"evidence_quote":"Provides the Arcade Learning Environment used to evaluate all methods."},{"cited_title":"Frame skip is a powerful parameter for learning to play Atari","cited_arxiv_id":null,"evidence_quote":"Establishes that frame skip rates significantly affect Atari performance, motivating action repetition as a design parameter."},{"cited_title":"Lakshminarayanan and others","cited_arxiv_id":null,"evidence_quote":"Introduces Dynamic Action Repetition (DFDQN), the main baseline that the proposed Bandit-DQN must beat, and supplies the experimental setup with d_max=20."},{"cited_title":"A contextual-bandit approach to personalized news article recommendation","cited_arxiv_id":null,"evidence_quote":"Supplies the contextual-bandit formulation that the duration-selection module is modeled on."},{"cited_title":"Human-level control through deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Defines the DQN algorithm and architecture that the paper augments, and is the source of the Q-values used to compute the bandit reward."}],"review_version":1}