Pith. sign in

REVIEW 5 major objections 6 minor 35 references

An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that ACED-DQN, an ensemble of five DQN variants weighted by recent prediction error, substantially outperforms Rainbow and SUNRISE in continual Atari environments, with arbitration control identified as the key…

desk verdict The paper's own ablation defeats its central claim, but the diversified ensemble and continual Atari benchmark are worth a second look. read the letter →

arxiv 2509.04815 v1 pith:GCQ6FY6V submitted 2025-09-05 cs.LG cs.MA

Wonseo Jang , Dongjae Kim This is my paper · ORCID
classification cs.LGcs.MA
keywords continualreinforcementlearningcatastrophicforgettingensembleDQNvariantsarbitrationcontrolreliabilityweightingagency-basedexperiencereplayAtaribenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ACED-DQN, a continual-reinforcement-learning agent built from an ensemble of five DQN variants (DQN, Double DQN, Noisy DQN, Dueling DQN, and Distributional DQN) deliberately trained to have diverse value functions. An arbitration control converts each agent's recent mean-squared TD error into a reliability weight and uses those weights to aggregate Q-values for action selection; an agency-based sampler uses the same signal to assign each experience to the agent most responsible for it. The central claim is that this design lets the ensemble adapt as game objectives and action randomness change every 200K steps, yielding a 50.5% win rate against 10.6% for Rainbow and 16.3% for SUNRISE across 26 Atari games. The paper further argues that no single DQN variant is best across all blocks, so dynamic, reliability-based selection is necessary.

What carries the argument

The load-bearing mechanism is the arbitration control: for each agent $i$, reliability is computed as $R_i^{(t)} = \mathrm{softmax}(-L_i^{(t)}/T)$ where $L_i$ is the agent's TD mean-squared error and $T$ a temperature; this is smoothed by an exponential moving average, clipped to $[R_{\min}, R_{\max}]$, and normalized to weights $w_i$ that aggregate the ensemble Q-value as $Q_{\mathrm{total}}(s,a) = \sum_i w_i Q_i(s,a)$, with the action taken as the argmax. A second mechanism, agency-based sampling, uses the same reliability signal to assign each PER-sampled transition to one agent, so each agent trains mainly on transitions it generated. Because Distributional DQN outputs a distribution over returns rather than a scalar, its loss is converted to a scalar MSE and rescaled by a factor $\alpha$ so that losses are comparable across heterogeneous agents.

What would settle it

Rank the five ensemble agents by their TD error and by their actual return in each 200K-step block: if the least-error agent is not the top-scoring agent in a substantial fraction of blocks, the arbitration signal is faulty. A direct check is the paper's own random-arbitration variant: if random weights achieve a similar win rate, arbitration is not the source of the gains.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that reliability-based arbitration over a deliberately diversified ensemble of DQN variants—rather than a single integrated agent such as Rainbow or a homogeneous ensemble such as SUNRISE—is what enables continual learning under shifting reward functions and action stochasticity. In the continual Atari benchmark, the best-performing variant changes across games and blocks, and the authors report that weighting each variant by the softmax of its recent TD error, then sampling experiences according to a sense of agency, keeps the ensemble tracking the current context. The empirical claim is a win rate of 50.5% against 10.6% for Rainbow and 16.3% for SUNRISE, with the paper stating that ablation studies indicate arbitration control is the key contributing factor. A supplementary theoretical analysis shows that DQN variants diverge because of biased Bellman operators (Double DQN converges to a different fixed point) and variance-induced tracking error in non-stationary environments, which grounds why a diverse, arbitrated ensemble can help.

Load-bearing premise

The whole arbitration rests on the premise that the agent with the smallest recent prediction error is the best agent for the current task; if that link breaks, reliability weighting can promote the wrong policy.

Editorial extensions

If this is right

  • If the central claim is correct, continual RL agents can retain old knowledge without explicit task boundaries by adaptively re-weighting a diverse ensemble of value functions.
  • The reported win-rate margin over Rainbow and SUNRISE suggests that arbitration between heterogeneous value functions can outperform both single-agent integration and homogeneous ensembles on Atari-like discrete control tasks.
  • Agency-based sampling offers a concrete remedy to the curse of diversity, allowing diverse agents to train on self-generated data rather than highly off-policy shared data.
  • The framework is claimed to generalize beyond discrete control, for example to continuous control and multi-task RL, by replacing the MSE reliability signal with a policy-appropriate loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own ablation, in which ACED-DQN with random action selection performs similarly to arbitrated ACED-DQN, suggests that the headline gains may be driven more by the diversified ensemble and agency-based sampling than by the arbitration weighting itself; this reading runs counter to the contribution bullet that calls arbitration the key factor.
  • If the link between low TD error and task-specific optimality is weak, arbitration could be made more robust by replacing the hand-tuned softmax temperature, momentum, and clipping with a learned selector, or by defining reliability on expected returns rather than raw loss.
  • The supplementary divergence analysis is stated for tabular Q-learning, so an open test is whether the five deep variants actually occupy distinct value regions when trained from the same seeds; a direct measurement of value-function distance across blocks would either support or undercut the premise that the ensemble members are truly diverse in the way the theory assumes.
  • A straightforward extension beyond the paper would be a temperature sweep (from near-zero, hard selection of the least-loss agent, to very large, uniform averaging) to map exactly where arbitration helps in the win-rate curve.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper proposes ACED-DQN, a continual RL method that combines an ensemble of five DQN variants (DQN, Double DQN, Dueling DQN, Noisy DQN, and Distributional DQN) with an arbitration mechanism that weights each agent's Q-values by a softmax over its recent TD loss (Eqs. 1-3), smoothed by an exponential moving average and clipped, and with an agency-based replay sampling scheme that attributes transitions to the most reliable agent (Algorithm 2). The evaluation uses 26 Atari games with five 200K-step blocks varying the reward objective and the probability of random actions. The paper reports win rates of ACED-DQN over Rainbow, SUNRISE, and an ACED-Rainbow variant, and an ablation with random arbitration, which is described as performing similarly to the full method. A supplementary theoretical analysis attempts to justify divergence among DQN variants through biased Bellman operators and update-noise variance.

Significance. If the main claims were fully supported, the paper would offer a useful, biologically inspired solution to the curse of diversity in ensemble RL and demonstrate that a diversified DQN ensemble with reliability-based arbitration can outperform strong baselines in a continual Atari benchmark. Strengths include the public code repository, the nontrivial 26-game continual benchmark design, and the attempt to ablate both arbitration and agency-based sampling. The present manuscript, however, does not yet establish the central claim: the headline comparisons lack statistical inference, and the random-arbitration ablation contradicts the contribution statement. The significance is therefore conditional on substantial revision.

major comments (5)
  1. [Section 4.3 vs. Section 1] The ablation reported in Section 4.3 directly undermines the contribution claim in Section 1 that "arbitration control is the key contributing factor to its success." The text states that ACED-DQN with random agent selection "performed similarly" and attributes the gains "not solely from arbitration control, but also from diversified DQNs during training—likely due to the agency-based sampling." Since ACED-DQN (random) retains the diversified DQN ensemble and the agency-based sampling of Algorithm 2 and replaces only the reliability weighting of Eqs. (1)-(3) with a random selector, it is the cleanest control for the paper's headline novelty. The random variant's win rate is not reported, and no paired significance test or confidence interval accompanies the 47.6% figure. The authors should either report the random-ablation numbers with confidence intervals and significance tests, or revise the contribution claim; as written, the paper contains an internal inconsistency in its central message.
  2. [Table 1, Section 4.2, and Figure 6] The headline win-rate comparisons (ACED-DQN 50.5%, ACED-Rainbow 22.6%, SUNRISE 16.3%, Rainbow 10.6%) are reported without confidence intervals, standard errors, or significance tests, and the same is true for the 47.6% win rate in Section 4.3. Given 26 environments and 30 evaluation runs, a paired test across games (e.g., Wilcoxon signed-rank or a bootstrap confidence interval on the win-rate difference) is needed to support the abstract's claim of "significant performance improvements." Without such inference, the stated superiority over Rainbow and SUNRISE is not statistically established.
  3. [Section 4.4.1] The load-bearing premise that "an agent's least MSE guarantees that the agent is the best in an ensemble" is asserted without evidence. The reliability signal is the training TD loss, and no analysis shows that lower MSE tracks task-appropriate optimality under non-stationary reward objectives. The sensitivity of Eqs. (1)-(2) to the temperature T, the momentum parameter gamma, the clipping bounds Rmin/Rmax, and especially the rescaling factor alpha for Distributional DQN (Supplementary Eq. (10)) is not studied; if the rescaling makes losses non-comparable across heterogeneous architectures, the arbitration can systematically favor the wrong agent. A sensitivity analysis and a direct comparison of arbitration against random selection with error bars would be needed to establish that the mechanism works as intended.
  4. [Supplementary Section 2.4] The inequality (E[||M_{t+1}||_2])^2 <= E[||M_{t+1}||_2^2] = Var(M_{t+1}) is not a valid basis for the claim that larger noise variance leads to larger expected noise magnitude. Jensen's inequality gives an upper bound on E[||M||] in terms of E[||M||^2], not a lower bound, and E[||M||^2] equals Var(M) only in the scalar zero-mean case; for vector-valued martingale differences, E[||M||^2] is the sum of component variances. The argument therefore does not establish the stated relationship between variance and tracking error, and it should either be corrected or removed from the theoretical justification.
  5. [Section 4.4.2 and Figure 8] The Top-2 match analysis is not interpretable as reported. Under the natural reading of the metric (the overlap between two top-2 sets, normalized by 2), the random permutation baseline of 0.41 +/- 0.04 is essentially the chance level for two random 2-subsets drawn from 5 agents (expected ratio 0.4), while the proposed method's 0.51 +/- 0.01 is only slightly above chance. The Wilcoxon test with W = 0 and p = 1.86e-9 is driven by the tiny variance across many comparisons rather than by a meaningful effect size, and the paper does not define the match metric explicitly. The authors should state the exact metric, report the explicit chance level, and provide game-level confidence intervals before claiming that agency-based sampling has a "stronger correspondence with the actual top-performing agents."
minor comments (6)
  1. [Section 3.1] There is a typo in "probabiltiy" in the paragraph introducing the probability of random actions.
  2. [Section 4.2 and Table 1] The text refers to "Table 4.2" where it should refer to Table 1, and the caption of Table 1 does not state the number of runs or confidence intervals; please add these details.
  3. [Section 2.2 and References] There are formatting issues such as "A veraged DQN" with an errant space and "A TTACHMENT" in some reference URLs; a careful proofreading pass is needed.
  4. [Section 3.2.2 and Algorithm 2] The symbol T is used for both the action-selection temperature and the agency-sampling temperature, even though the two values differ (0.3 and 0.8); please use distinct names such as T_act and T_sample to avoid confusion.
  5. [Figure 5 caption] The sentence "we plotted two data points for each blocks, resulting 10 markers" is unclear; please clarify what the two markers per block represent.
  6. [General] The term "win rate" is used throughout but never precisely defined; please state whether it is computed per game, per block, or per evaluation run, and how ties are handled.

Circularity Check

1 steps flagged · score 4.0 of 10

Agency-sampling validation is self-referential; the main win-rate claim is independent but is contradicted by the paper's own random-selection ablation.

  1. self definitional [Section 4.4.1-4.4.2, with Eq. (1) and Supplementary Eq. (10)]
    "For transitions, we calculated a metric, which is a ratio of reliability to MSE. It validates if the most optimal agent (i.e., least MSE), and therefore it can be used to measure degree of optimality of agents. Note that the models are already trained enough, so that an agent's least MSE guarantees that the agent is the best in an ensemble."

    Reliability R_i = softmax(-L_i/T) (Eq. 1) is, by construction, an order-preserving transform of the training MSE L_i. Section 4.4.1 then declares the 'most optimal agent' to be exactly the least-MSE agent and asserts this equivalence as a 'guarantee.' Under that declared equivalence, selecting the 'highest reliability' agents and comparing them with the 'best' agents compares a ranking with itself; the external return data from Figure 4 cannot falsify the premise because the premise is asserted a priori. The measured top-2 match of 0.51 would actually contradict the 'guarantee,' yet the paper interprets it as validation, showing that the validation is self-referential rather than an independent check.

full rationale

The central performance comparison (Table 1, Figure 5) is an external benchmark: ACED-DQN's win rates against Rainbow, SUNRISE, and ACED-Rainbow are empirical outcomes, not derived from the reliability formula, so the main empirical claim is not circular. The reliability weighting itself is definitional (Eq. 1: R=softmax(-L/T)), but using it in action selection is a design choice, not a prediction. The one genuinely self-referential step is the validation of agency-based sampling in Section 4.4: the paper equates 'best' with least MSE and then measures whether the reliability ranking (also least MSE) matches the best agents, with the equivalence asserted as a 'guarantee' rather than tested. This makes the validation partially circular. Separately, Section 1's claim that 'ablation studies indicate that arbitration control is the key contributing factor to its success' is directly contradicted by Section 4.3, where ACED-DQN with random agent selection 'performed similarly'; this is an internal inconsistency and a correctness risk, but it is not a circular reduction. The unreported rescaling factor alpha (Supplementary Eq. 10) is a reproducibility gap, not a circular step. Overall, the main result is not forced by construction, so the circularity score is moderate.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on a small number of hand-tuned hyperparameters and on the assumption that raw loss values are comparable and informative across heterogeneous DQN architectures. The paper does not supply sensitivity studies or an external validation of the reliability metric.

free parameters (6)
  • Arbitration control temperature T (action selection) = 0.3
    Controls the sharpness of the softmax reliability weighting; chosen ad hoc, no sensitivity analysis is provided.
  • Agency sampling temperature T = 0.8
    Used to convert per-transition losses into probabilities in Algorithm 2; set ad hoc, no sensitivity analysis.
  • Reliability smoothing momentum gamma = 0.6
    Sets the exponential moving average rate for reliability updates; tuned without reported sensitivity.
  • Reliability clipping bounds Rmin, Rmax = [0.2, 0.5]
    Clamps the smoothed reliability values; the narrow range limits how much arbitration can actually favor a single agent.
  • Loss rescaling factor alpha for Distributional DQN = not reported
    Supplementary Eq. (10) introduces alpha to make the C51 MSE comparable to other agents' losses, but no value or tuning procedure is given.
  • Algorithm 2 sizes (pre-sample size M and top-k) = not reported
    These control the agency-based sampling behavior; neither is listed in the hyperparameter table, leaving the procedure underspecified.
assumptions (4)
  • standard math Robbins-Monro stochastic approximation convergence conditions hold (infinite visits, sum alpha = infinity, sum alpha squared < infinity)
    Used in Supplementary Theorem 1 to claim convergence of DQN variants to fixed points; standard, but not verified for deep networks.
  • domain assumption An agent's TD MSE is a valid proxy for its reliability and optimality in the current context
    Section 3.2.1 defines reliability from MSE; Section 4.4.1 assumes least MSE 'guarantees' the agent is best. No evidence is provided, especially when losses are normalized across heterogeneous variants.
  • ad hoc to paper Loss magnitudes of different DQN variants can be made comparable by a single rescaling factor alpha
    Supplementary Section 4.1 uses Eq. (10) with alpha, but alpha is never specified or justified; the two temperatures and clipping range are also chosen manually.
  • domain assumption The continual Atari block design (objective shift and random-action probability) is a meaningful CRL benchmark
    Section 3.1 introduces the benchmark; no validation that these perturbations induce catastrophic forgetting or mimic real continual learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learning." pith.science (2026). https://pith.science/paper/GCQ6FY6V

@misc{pith2026250904815,
  author       = {Pith},
  title        = {Pith review of: An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GCQ6FY6V}},
  note         = {Machine review of arXiv:2509.04815}
}
read the original abstract

Deep reinforcement learning (RL) models, despite their efficiency in learning an optimal policy in static environments, easily loses previously learned knowledge (i.e., catastrophic forgetting). It leads RL models to poor performance in continual reinforcement learning (CRL) scenarios. To address this, we present an arbitration control mechanism over an ensemble of RL agents. It is motivated by and closely aligned with how humans make decisions in a CRL context using an arbitration control of multiple RL agents in parallel as observed in the prefrontal cortex. We integrated two key ideas into our model: (1) an ensemble of RLs (i.e., DQN variants) explicitly trained to have diverse value functions and (2) an arbitration control that prioritizes agents with higher reliability (i.e., less error) in recent trials. We propose a framework for CRL, an Arbitration Control for an Ensemble of Diversified DQN variants (ACED-DQN). We demonstrate significant performance improvements in both static and continual environments, supported by empirical evidence showing the effectiveness of arbitration control over diversified DQNs during training. In this work, we introduced a framework that enables RL agents to continuously learn, with inspiration from the human brain.

Figures

Figures reproduced from arXiv: 2509.04815 by the authors.

Figure 1
Figure 1. A schematic view of conventional ensemble RL and the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Comparison between SUNRISE and ACED-DQN. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Performance of DQN variants in each block [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Learning curves of RL agents. Each panel represents averaged episodic return of ACED-DQN, SUNRISE, Rainbow, and ACED-Rainbow. Although there are five environmental context blocks, we plotted two data points for each blocks, resulting 10 markers on each line. Markers in…
Figure 6
Figure 6. Figure 6: Performance comparison of RL agents on 26 continual Atari tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The effects of Agency-based sampling. Normalized ratio of reliability to MSE in each block is measured to illustrate how ACED-DQN selects different RL agents as an optimal choice. Evaluation is performed across five consecutive 200k-step blocks (B0-B4). Markers and the…
Figure 8
Figure 8. Figure 8: Arbitration control is well-matched to the optimal agent. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 14 canonical work pages

  1. [1]

    D. Abel, G. Deepmind, A. Barreto, B. V. Roy, D. Precup, H. V. Hasselt, and S. Singh. A definition of continual reinforcement learning. 7 2023. ISSN 10495258. URL https://arxiv.org/abs/2307.11046v2

    arXiv 2023

  2. [2]

    Agarwal, D

    R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. 37th International Conference on Machine Learning, ICML 2020, PartF168147-1: 0 92--102, 7 2019. URL https://arxiv.org/abs/1907.04543v4

    arXiv 2020

  3. [3]

    Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

    O. Ansehel, N. Baram, and N. Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. 34th International Conference on Machine Learning, ICML 2017, 1: 0 240--253, 11 2016. URL https://arxiv.org/pdf/1611.01929

    work page Pith review arXiv 2017

  4. [4]

    M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. 34th International Conference on Machine Learning, ICML 2017, 1: 0 693--711, 7 2017. URL https://arxiv.org/abs/1707.06887v1

    arXiv 2017

  5. [5]

    N. D. Daw, S. J. Gershman, B. Seymour, P. Dayan, and R. J. Dolan. Model-based influences on humans' choices and striatal prediction errors. Neuron, 69: 0 1204--1215, 3 2011. ISSN 1097-4199. doi:10.1016/J.NEURON.2011.02.027. URL https://pubmed.ncbi.nlm.nih.gov/21435563/

    work page arXiv 2011

  6. [6]

    Fortunato, M

    M. Fortunato, M. G. Azar, B. Piot, J. Menick, M. Hessel, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy networks for exploration. 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, 6 2017. URL https://arxiv.org/abs/1706.10295v3

    arXiv 2018

  7. [7]

    Gl \"a scher, R

    J. Gl \"a scher, R. Adolphs, H. Damasio, A. Bechara, D. Rudrauf, M. Calamia, L. K. Paul, and D. Tranel. Lesion mapping of cognitive control and value-based decision making in the prefrontal cortex. Proceedings of the National Academy of Sciences, 109 0 (36): 0 14681--14686, 2012

    work page 2012

  8. [8]

    H. V. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. 30th AAAI Conference on Artificial Intelligence, AAAI 2016, pages 2094--2100, 9 2015. ISSN 2159-5399. doi:10.1609/aaai.v30i1.10295. URL https://arxiv.org/abs/1509.06461v3

    arXiv 2016

Show all 35 references
  1. [9]

    Hessel, J

    M. Hessel, J. Modayil, H. V. Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 3215--3222, 10 2017. ISSN 215...

    2018 arXiv

  2. [10]

    Ishfaq, Q

    H. Ishfaq, Q. Cui, V. Nguyen, A. Ayoub, Z. Yang, Z. Wang, D. Precup, and L. F. Yang. Randomized exploration for reinforcement learning with general value function approximation. Proceedings of Machine Learning Research, 139: 0 4607--4616, 6 2021. ISSN 26403498. URL https://arx...

    2021 arXiv

  3. [11]

    Januszewski, M

    P. Januszewski, M. Olko, M. Kr \'o likowski, J. \'S wi a tkowski, M. Andrychowicz, . Kuci \'n ski, and P. Mi o \'s . Continuous control with ensemble deep deterministic policy gradients. 2021. URL https://arxiv.org/abs/2111.15382

    2021 arXiv

  4. [12]

    Khetarpal, M

    K. Khetarpal, M. Riemer, I. Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75: 0 1401--1476, 12 2020. ISSN 10769757. doi:10.1613/JAIR.1.13673. URL https://arxiv.org/abs/2012.13490v2

    2020 arXiv

  5. [13]

    D. Kim, G. Y. Park, J. P. O'Doherty, and S. W. Lee. Task complexity interacts with state-space uncertainty in the arbitration between model-based and model-free learning. Nature Communications 2019 10:1, 10: 0 1--14, 12 2019. ISSN 2041-1723. doi:10.1038/s41467-019-13632-1. URL...

    2019 doi

  6. [14]

    D. Kim, J. Jeong, and S. W. Lee. Prefrontal solution to the bias-variance tradeoff during reinforcement learning. Cell Reports, 37, 12 2021. ISSN 22111247. doi:10.1016/J.CELREP.2021.110185/ATTACHMENT/69B8EC97-BEDE-42D0-82C7-8B177ECAC289/MMC2.PDF. URL http://www.cell.com/articl...

    2021

  7. [15]

    K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. 2021

    2021

  8. [16]

    S. W. Lee, S. Shimojo, and J. P. O'Doherty. Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81: 0 687--699, 2 2014. ISSN 08966273. doi:10.1016/J.NEURON.2013.11.028/ATTACHMENT/168F1AFC-B6B7-4349-B318-A51F17D5FAFB/MMC1.PDF. URL htt...

    2014 doi

  9. [17]

    Z. Lin, P. D'Oro, E. Nikishin, and A. Courville. The curse of diversity in ensemble-based exploration. 12th International Conference on Learning Representations, ICLR 2024, 5 2024. URL https://arxiv.org/abs/2405.04342v1

    2024 arXiv

  10. [18]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. 12 2013. URL https://arxiv.org/abs/1312.5602v1

    2013 arXiv

  11. [19]

    J. W. Moore. What is the sense of agency and why does it matter? Frontiers in psychology, 7: 0 1272, 2016

    2016

  12. [20]

    R. Y. C. Openai, S. S. Openai, P. Abbeel, and J. S. Openai. Ucb exploration via q-ensembles. 6 2017. URL https://arxiv.org/pdf/1706.01502

    2017 arXiv

  13. [21]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems, pages 4033--4041, 2 2016 a . ISSN 10495258. URL https://arxiv.org/abs/1602.04621v3

    2016 arXiv

  14. [22]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016 b

    2016

  15. [23]

    Ostrovski, M

    G. Ostrovski, M. G. Bellemare, A. V. D. Oord, and R. Munos. Count-based exploration with neural density models. 34th International Conference on Machine Learning, ICML 2017, 6: 0 4161--4175, 3 2017. URL https://arxiv.org/pdf/1703.01310

    2017 arXiv

  16. [24]

    O. Peer, C. Tessler, N. Merlis, and R. Meir. Ensemble bootstrapping for q-learning. Proceedings of Machine Learning Research, 139: 0 8454--8463, 2 2021. ISSN 26403498. URL https://arxiv.org/pdf/2103.00445

    2021 arXiv

  17. [25]

    M. F. Rushworth, N. Kolling, J. Sallet, and R. B. Mars. Valuation and decision-making in frontal cortex: one or many serial or parallel systems? Current opinion in neurobiology, 22: 0 946--955, 12 2012. ISSN 1873-6882. doi:10.1016/J.CONB.2012.04.011. URL https://pubmed.ncbi.nl...

    2012

  18. [26]

    Saphal, B

    R. Saphal, B. Ravindran, D. Mudigere, S. Avancha, and B. Kaul. Seerl: Sample efficient ensemble reinforcement learning. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2: 0 1088--1096, 1 2020. doi:10.5555/3463952.3464080. U...

    2020

  19. [27]

    Schaul, J

    T. Schaul, J. Quan, I. Antonoglou, D. Silver, and G. Deepmind. Prioritized experience replay. 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, pages 1--21, 11 2015. URL https://arxiv.org/abs/1511.05952v4

    2016 arXiv

  20. [28]

    Schmitt, M

    S. Schmitt, M. Hessel, and K. Simonyan. Off-policy actor-critic with shared experience replay. 37th International Conference on Machine Learning, ICML 2020, PartF168147-11: 0 8503--8512, 9 2019. URL https://arxiv.org/pdf/1909.11583

    2020 arXiv

  21. [29]

    Sutton and A

    R. Sutton and A. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9: 0 1054--1054, 4 2005. ISSN 1045-9227. doi:10.1109/TNN.1998.712192

    2005

  22. [30]

    R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning 1988 3:1, 3: 0 9--44, 8 1988. ISSN 1573-0565. doi:10.1007/BF00115009. URL https://link.springer.com/article/10.1007/BF00115009

    1988 doi

  23. [31]

    Thangarajah, F

    J. Thangarajah, F. Zambetta, and M. Dann. Exploration in sparse reward games examining and improving exploration effort partitioning. 2018. URL https://studenttheses.uu.nl/handle/20.500.12932/31527

    2018

  24. [32]

    G. M. van de Ven, N. Soures, and D. Kudithipudi. Continual learning and catastrophic forgetting. pages 55--75, 3 2024. doi:10.1007/978-3-031-01581-6_4. URL https://arxiv.org/abs/2403.05175v1

    2024 arXiv

  25. [33]

    Z. Wang, T. Schaul, M. Hessel, H. V. Hasselt, M. Lanctot, and N. D. Frcitas. Dueling network architectures for deep reinforcement learning. 33rd International Conference on Machine Learning, ICML 2016, 4: 0 2939--2947, 11 2015. URL https://arxiv.org/abs/1511.06581v3

    2016 arXiv

  26. [34]

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34: 0 25476--25488, 2021

    2021

  27. [35]

    Borkar and S

    V. Borkar and S. Meyn. The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38 0 (2): 0 447--469, 2000

    2000

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.