{"id":"8fa2446a-77c3-4693-9350-3d8db10326b0","arxiv_id":"2509.04815","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"ACED-DQN combines heterogeneous DQN variants with loss-based reliability weighting and experience assignment, but the paper's own ablation indicates that arbitration control is not the key factor behind the performance gain.","lead":"ACED-DQN is an ensemble of five DQN variants whose actions are chosen by weighting each agent's Q-values by a reliability score derived from recent training loss, aiming to fight catastrophic forgetting in continual reinforcement learning. The paper reports a 50.5 percent win rate across 26 modified Atari games, but the authors' own ablation shows that replacing the reliability weighting with random agent selection performs almost as well.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own ablation contradicts the central claim: random arbitration performs similarly, so arbitration control is not shown to be the key contributing factor.","rationale":"The reader's verdict already flags the random-selection ablation as a contradiction, but their weakest_assumption focuses on the MSE-reliability premise rather than on the direct policy-level control. I view the random ablation as the more load-bearing concern because it does not depend on whether MSE tracks optimality: even if losses were perfectly informative, the ablation suggests reliability-weighted aggregation adds little over random aggregation. This is an internal inconsistency, not a disagreement with external consensus, and it is testable with the released code. The reader's CONDITIONAL verdict already requires reframing the contribution and adding significance tests, so my analysis does not shift the verdict; it sharpens the condition by specifying the decisive comparison and the exact reporting gap.","tokens_in":21864,"tokens_out":6697,"duration_ms":58113,"concrete_test":"Using the released GitHub code, re-run the Section 4.3 comparison and report the exact ACED-DQN (random) win rate and per-game per-block mean returns, then compute a paired Wilcoxon signed-rank test (or 95% bootstrap confidence interval) over the 26 games x 5 blocks between ACED-DQN and ACED-DQN (random). If p > 0.05 or the confidence interval includes zero, the paper's own comparison falsifies the claim that arbitration control is the key contributing factor; contribution bullet 3 should be revised to credit diversified DQN training with agency-based sampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 1 contribution bullet 3 states that 'ablation studies indicate that arbitration control is the key contributing factor to its success.' Section 4.3 directly undercuts this: ACED-DQN with random agent selection 'performed similarly,' and the text concludes that the gains arise 'not solely from arbitration control, but also from diversified DQNs during training—likely due to the agency-based sampling.' Because ACED-DQN (random) keeps the same heterogeneous DQN ensemble and agency-based sampling and changes only the reliability-weighted aggregation of Eqs. (1)-(3) to a random selector, it is the cleanest control for the paper's headline novelty. The random variant's win rate is not reported, and no paired significance test or confidence interval accompanies the 47.6% figure. The paper therefore contains an internal inconsistency: its own ablation is a counterexample to the stated central claim. The separate assertion in Section 4.4.1 that least MSE 'guarantees' the best agent is only needed if arbitration adds value; the random ablation tests the policy-level question directly and currently fails to support the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ACED-DQN, a continual RL method that combines an ensemble of five DQN variants (DQN, Double DQN, Dueling DQN, Noisy DQN, and Distributional DQN) with an arbitration mechanism that weights each agent's Q-values by a softmax over its recent TD loss (Eqs. 1-3), smoothed by an exponential moving average and clipped, and with an agency-based replay sampling scheme that attributes transitions to the most reliable agent (Algorithm 2). The evaluation uses 26 Atari games with five 200K-step blocks varying the reward objective and the probability of random actions. The paper reports win rates of ACED-DQN over Rainbow, SUNRISE, and an ACED-Rainbow variant, and an ablation with random arbitration, which is described as performing similarly to the full method. A supplementary theoretical analysis attempts to justify divergence among DQN variants through biased Bellman operators and update-noise variance.","tokens_in":22091,"tokens_out":8714,"duration_ms":72122,"significance":"If the main claims were fully supported, the paper would offer a useful, biologically inspired solution to the curse of diversity in ensemble RL and demonstrate that a diversified DQN ensemble with reliability-based arbitration can outperform strong baselines in a continual Atari benchmark. Strengths include the public code repository, the nontrivial 26-game continual benchmark design, and the attempt to ablate both arbitration and agency-based sampling. The present manuscript, however, does not yet establish the central claim: the headline comparisons lack statistical inference, and the random-arbitration ablation contradicts the contribution statement. The significance is therefore conditional on substantial revision.","major_comments":[{"comment":"The ablation reported in Section 4.3 directly undermines the contribution claim in Section 1 that \"arbitration control is the key contributing factor to its success.\" The text states that ACED-DQN with random agent selection \"performed similarly\" and attributes the gains \"not solely from arbitration control, but also from diversified DQNs during training—likely due to the agency-based sampling.\" Since ACED-DQN (random) retains the diversified DQN ensemble and the agency-based sampling of Algorithm 2 and replaces only the reliability weighting of Eqs. (1)-(3) with a random selector, it is the cleanest control for the paper's headline novelty. The random variant's win rate is not reported, and no paired significance test or confidence interval accompanies the 47.6% figure. The authors should either report the random-ablation numbers with confidence intervals and significance tests, or revise the contribution claim; as written, the paper contains an internal inconsistency in its central message.","section":"Section 4.3 vs. Section 1"},{"comment":"The headline win-rate comparisons (ACED-DQN 50.5%, ACED-Rainbow 22.6%, SUNRISE 16.3%, Rainbow 10.6%) are reported without confidence intervals, standard errors, or significance tests, and the same is true for the 47.6% win rate in Section 4.3. Given 26 environments and 30 evaluation runs, a paired test across games (e.g., Wilcoxon signed-rank or a bootstrap confidence interval on the win-rate difference) is needed to support the abstract's claim of \"significant performance improvements.\" Without such inference, the stated superiority over Rainbow and SUNRISE is not statistically established.","section":"Table 1, Section 4.2, and Figure 6"},{"comment":"The load-bearing premise that \"an agent's least MSE guarantees that the agent is the best in an ensemble\" is asserted without evidence. The reliability signal is the training TD loss, and no analysis shows that lower MSE tracks task-appropriate optimality under non-stationary reward objectives. The sensitivity of Eqs. (1)-(2) to the temperature T, the momentum parameter gamma, the clipping bounds Rmin/Rmax, and especially the rescaling factor alpha for Distributional DQN (Supplementary Eq. (10)) is not studied; if the rescaling makes losses non-comparable across heterogeneous architectures, the arbitration can systematically favor the wrong agent. A sensitivity analysis and a direct comparison of arbitration against random selection with error bars would be needed to establish that the mechanism works as intended.","section":"Section 4.4.1"},{"comment":"The inequality (E[||M_{t+1}||_2])^2 <= E[||M_{t+1}||_2^2] = Var(M_{t+1}) is not a valid basis for the claim that larger noise variance leads to larger expected noise magnitude. Jensen's inequality gives an upper bound on E[||M||] in terms of E[||M||^2], not a lower bound, and E[||M||^2] equals Var(M) only in the scalar zero-mean case; for vector-valued martingale differences, E[||M||^2] is the sum of component variances. The argument therefore does not establish the stated relationship between variance and tracking error, and it should either be corrected or removed from the theoretical justification.","section":"Supplementary Section 2.4"},{"comment":"The Top-2 match analysis is not interpretable as reported. Under the natural reading of the metric (the overlap between two top-2 sets, normalized by 2), the random permutation baseline of 0.41 +/- 0.04 is essentially the chance level for two random 2-subsets drawn from 5 agents (expected ratio 0.4), while the proposed method's 0.51 +/- 0.01 is only slightly above chance. The Wilcoxon test with W = 0 and p = 1.86e-9 is driven by the tiny variance across many comparisons rather than by a meaningful effect size, and the paper does not define the match metric explicitly. The authors should state the exact metric, report the explicit chance level, and provide game-level confidence intervals before claiming that agency-based sampling has a \"stronger correspondence with the actual top-performing agents.\"","section":"Section 4.4.2 and Figure 8"}],"minor_comments":[{"comment":"There is a typo in \"probabiltiy\" in the paragraph introducing the probability of random actions.","section":"Section 3.1"},{"comment":"The text refers to \"Table 4.2\" where it should refer to Table 1, and the caption of Table 1 does not state the number of runs or confidence intervals; please add these details.","section":"Section 4.2 and Table 1"},{"comment":"There are formatting issues such as \"A veraged DQN\" with an errant space and \"A TTACHMENT\" in some reference URLs; a careful proofreading pass is needed.","section":"Section 2.2 and References"},{"comment":"The symbol T is used for both the action-selection temperature and the agency-sampling temperature, even though the two values differ (0.3 and 0.8); please use distinct names such as T_act and T_sample to avoid confusion.","section":"Section 3.2.2 and Algorithm 2"},{"comment":"The sentence \"we plotted two data points for each blocks, resulting 10 markers\" is unclear; please clarify what the two markers per block represent.","section":"Figure 5 caption"},{"comment":"The term \"win rate\" is used throughout but never precisely defined; please state whether it is computed per game, per block, or per evaluation run, and how ties are handled.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a publicly available implementation and a clear experimental protocol, but the mismatch between the contribution claim (arbitration is key) and the Section 4.3 ablation result is the central problem. I recommend major revision rather than rejection because the issue is addressable by reporting the random-ablation numbers with confidence intervals, adding paired significance tests for the headline tables, and correcting the theoretical supplement. I would also ask the editor to require the authors to disclose the omitted rescaling factor alpha and its value, since the comparability of losses across DQN variants is load-bearing for the arbitration mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a paper where the main claim fails on the authors' own evidence. They state in the introduction that arbitration control is the key contributing factor, but their Section 4.3 ablation shows ACED-DQN with random agent selection performing similarly. That is the direct control, and it undercuts the whole selling point. The paper would be better framed as evidence for a diversified DQN ensemble with agency-based sampling, which is what the data actually supports.\n\nCredit where it is due: combining heterogeneous DQN variants with reliability-weighted selection and per-agent experience assignment is a new combination. The continual Atari benchmark, with alternating objectives and action randomness across five blocks, is a real contribution; adapting 26 games is not trivial. They released code, which is more than many papers do. The isolated-block study (Section 4.1) reasonably shows that no single variant wins everywhere, which motivates the ensemble approach.\n\nThe soft spots are not minor. The reliability signal is circular by construction: agents train on their own data, so low loss may just mean the agent has specialized, not that it is currently best. Section 4.4.1's claim that 'an agent's least MSE guarantees that the agent is the best in an ensemble' is asserted without proof. The top-2 match analysis is a consistency check on the same runs, not a predictive validation. The supplementary mentions a rescaling factor alpha for Distributional DQN but never reports its value, which hurts reproducibility. The Jensen's inequality step in the supplementary is wrong for nonzero mean: E[||M||^2] is not Var(M) unless the mean is zero. The qualitative conclusion may survive, but the math as written is incorrect.\n\nWho is this for? People working on continual RL with ensembles might find the benchmark and the agency-sampling idea useful. But the current paper needs major revision: report the random ablation's exact win rate with confidence intervals, run a paired test, and either fix the theoretical section or drop it. If the authors reframe the contribution around the diversified ensemble, the paper becomes credible. As it stands, I would not cite it.\n\nRecommendation: send to peer review only if the editor asks the authors to address the contradiction head-on. The kernel is salvageable; the current presentation is not.","headline":"The paper's own ablation defeats its central claim, but the diversified ensemble and continual Atari benchmark are worth a second look.","tokens_in":22606,"tokens_out":4123,"would_cite":false,"duration_ms":34430,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ACED-DQN, an ensemble of five DQN variants weighted by recent prediction error, substantially outperforms Rainbow and SUNRISE in continual Atari environments, with arbitration control identified as the key…","keywords":["continual reinforcement learning","catastrophic forgetting","ensemble reinforcement learning","DQN variants","arbitration control","reliability weighting","agency-based experience replay","Atari benchmark"],"falsifier":"Rank the five ensemble agents by their TD error and by their actual return in each 200K-step block: if the least-error agent is not the top-scoring agent in a substantial fraction of blocks, the arbitration signal is faulty. A direct check is the paper's own random-arbitration variant: if random weights achieve a similar win rate, arbitration is not the source of the gains.","tokens_in":21633,"feed_emoji":"🎮","tokens_out":9238,"duration_ms":72346,"temperature":0.7,"pith_summary":"The paper introduces ACED-DQN, a continual-reinforcement-learning agent built from an ensemble of five DQN variants (DQN, Double DQN, Noisy DQN, Dueling DQN, and Distributional DQN) deliberately trained to have diverse value functions. An arbitration control converts each agent's recent mean-squared TD error into a reliability weight and uses those weights to aggregate Q-values for action selection; an agency-based sampler uses the same signal to assign each experience to the agent most responsible for it. The central claim is that this design lets the ensemble adapt as game objectives and action randomness change every 200K steps, yielding a 50.5% win rate against 10.6% for Rainbow and 16.3% for SUNRISE across 26 Atari games. The paper further argues that no single DQN variant is best across all blocks, so dynamic, reliability-based selection is necessary.","feed_headline":"Arbitrated DQN ensemble wins 50.5% of continual Atari matchups","feed_subtitle":"Five diverse DQN variants, selected by recent error, outperform Rainbow and SUNRISE when game rules shift.","key_machinery":"The load-bearing mechanism is the arbitration control: for each agent $i$, reliability is computed as $R_i^{(t)} = \\mathrm{softmax}(-L_i^{(t)}/T)$ where $L_i$ is the agent's TD mean-squared error and $T$ a temperature; this is smoothed by an exponential moving average, clipped to $[R_{\\min}, R_{\\max}]$, and normalized to weights $w_i$ that aggregate the ensemble Q-value as $Q_{\\mathrm{total}}(s,a) = \\sum_i w_i Q_i(s,a)$, with the action taken as the argmax. A second mechanism, agency-based sampling, uses the same reliability signal to assign each PER-sampled transition to one agent, so each agent trains mainly on transitions it generated. Because Distributional DQN outputs a distribution over returns rather than a scalar, its loss is converted to a scalar MSE and rescaled by a factor $\\alpha$ so that losses are comparable across heterogeneous agents.","core_discovery":"On the paper's own terms, the discovery is that reliability-based arbitration over a deliberately diversified ensemble of DQN variants—rather than a single integrated agent such as Rainbow or a homogeneous ensemble such as SUNRISE—is what enables continual learning under shifting reward functions and action stochasticity. In the continual Atari benchmark, the best-performing variant changes across games and blocks, and the authors report that weighting each variant by the softmax of its recent TD error, then sampling experiences according to a sense of agency, keeps the ensemble tracking the current context. The empirical claim is a win rate of 50.5% against 10.6% for Rainbow and 16.3% for SUNRISE, with the paper stating that ablation studies indicate arbitration control is the key contributing factor. A supplementary theoretical analysis shows that DQN variants diverge because of biased Bellman operators (Double DQN converges to a different fixed point) and variance-induced tracking error in non-stationary environments, which grounds why a diverse, arbitrated ensemble can help.","pith_inferences":["The paper's own ablation, in which ACED-DQN with random action selection performs similarly to arbitrated ACED-DQN, suggests that the headline gains may be driven more by the diversified ensemble and agency-based sampling than by the arbitration weighting itself; this reading runs counter to the contribution bullet that calls arbitration the key factor.","If the link between low TD error and task-specific optimality is weak, arbitration could be made more robust by replacing the hand-tuned softmax temperature, momentum, and clipping with a learned selector, or by defining reliability on expected returns rather than raw loss.","The supplementary divergence analysis is stated for tabular Q-learning, so an open test is whether the five deep variants actually occupy distinct value regions when trained from the same seeds; a direct measurement of value-function distance across blocks would either support or undercut the premise that the ensemble members are truly diverse in the way the theory assumes.","A straightforward extension beyond the paper would be a temperature sweep (from near-zero, hard selection of the least-loss agent, to very large, uniform averaging) to map exactly where arbitration helps in the win-rate curve."],"forward_implications":["If the central claim is correct, continual RL agents can retain old knowledge without explicit task boundaries by adaptively re-weighting a diverse ensemble of value functions.","The reported win-rate margin over Rainbow and SUNRISE suggests that arbitration between heterogeneous value functions can outperform both single-agent integration and homogeneous ensembles on Atari-like discrete control tasks.","Agency-based sampling offers a concrete remedy to the curse of diversity, allowing diverse agents to train on self-generated data rather than highly off-policy shared data.","The framework is claimed to generalize beyond discrete control, for example to continuous control and multi-task RL, by replacing the MSE reliability signal with a policy-appropriate loss."],"supporting_citations":[{"why":"Provides Rainbow, the single-agent baseline that ACED-DQN outperforms, and the integrated DQN improvements it builds on.","marker":"[9]"},{"why":"Provides SUNRISE, the homogeneous-ensemble baseline ACED-DQN is compared against, with UCB exploration and weighted Bellman updates.","marker":"[15]"},{"why":"Documents the curse of diversity in shared-replay ensembles, the problem the agency-based sampling is designed to solve.","marker":"[17]"},{"why":"Introduces the DQN algorithm that anchors the ensemble and supplies the base network architecture.","marker":"[18]"},{"why":"Supplies Distributional DQN (C51), one of the five ensemble members, and the distributional return representation.","marker":"[4]"},{"why":"Supplies Noisy DQN, the ensemble member whose parameter noise drives exploration and often wins the reliability ranking.","marker":"[6]"},{"why":"Supplies Double DQN, the ensemble member that motivates the biased-operator divergence analysis.","marker":"[8]"},{"why":"Defines the Atari 100K benchmark whose 26 games are adapted into the continual block environment.","marker":"[34]"},{"why":"Provides prioritized experience replay, the base sampler on which agency-based transition assignment is built.","marker":"[27]"}],"fun_headline_variants":["Brain-inspired arbitration boosts continual RL","DQN ensemble with arbitration wins continual Atari","Reliability-based arbitration beats Rainbow in CRL","Diversified DQN ensemble plus arbitration for CRL","Arbitration control curbs catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole arbitration rests on the premise that the agent with the smallest recent prediction error is the best agent for the current task; if that link breaks, reliability weighting can promote the wrong policy.","fun_headline_variants_meta":{"raw":{"variants":["Brain-inspired arbitration boosts continual RL","DQN ensemble with arbitration wins continual Atari","Reliability-based arbitration beats Rainbow in CRL","Diversified DQN ensemble plus arbitration for CRL","Arbitration control curbs catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1253,"prompt_tokens":976,"completion_tokens":277,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":207}},"tokens_in":592,"tokens_out":277,"duration_ms":3272,"temperature":1.0,"reasoning_tokens":207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:26:23.524727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rank the five ensemble agents by their TD error and by their actual return in each 200K-step block: if the least-error agent is not the top-scoring agent in a substantial fraction of blocks, the arbitration signal is faulty. A direct check is the paper's own random-arbitration variant: if random weights achieve a similar win rate, arbitration is not the source of the gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides SUNRISE, the homogeneous-ensemble baseline ACED-DQN is compared against, with UCB exploration and weighted Bellman updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Atari 100K benchmark whose 26 games are adapted into the continual block environment."}],"review_version":2}