Pith. sign in

REVIEW 5 major objections 6 minor 79 references

Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces CL-DC, where an agent's action set grows or shrinks between tasks, and claims that AACL—a framework that decouples policy from actions through a learned action-representation space—outperforms six baselines on three…

desk verdict A useful new problem formulation and a credible first solver, but the contraction-evaluation protocol and code release need to be pinned down before the empirical claims are fully checkable. read the letter →

arxiv 2506.05702 v1 pith:VPIIIN4P submitted 2025-06-06 cs.LG cs.AI

Chaofan Pan , Jiafen Liu , Yanhua Li , Linbo Xiong , Fan Min , Wei Wei , Xin Yang This is my paper
classification cs.LGcs.AI
keywords continuallearningreinforcementdynamicactionspacesrepresentationcatastrophicforgettingpolicygeneralizationself-supervisedelasticweightconsolidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most continual learning assumes the agent's abilities stay fixed while the environment changes. This paper introduces a complementary problem, CL-DC, where the action space itself changes between tasks—actions are added, removed, or both—and asks whether a policy learned for old actions can generalize to new ones. The proposed answer is AACL: train an encoder-decoder that maps actions to and from a learned action-representation space, train the policy only on that space, and when the action set changes, expand or mask the decoder and fine-tune it under an EWC constraint. On a new benchmark built from MiniGrid, Bigfish, and Atlantis, with expansion, contraction, and combined sequences, the authors report that AACL achieves better continual return and forward transfer and lower forgetting than the compared continual-learning baselines. If correct, this makes continuing learning viable for agents whose bodies or interfaces change, such as robots that lose or gain limbs.

What carries the argument

The load-bearing mechanism is an action representation space $\mathcal{E}$: an encoder $f_\phi$ maps an action $a$ to a vector $e$ in $\mathcal{E}$ by predicting the next state $s'$ from $(s,a)$ using self-supervised cross-entropy on reward-free transitions, and a decoder $g_\delta$ maps $e$ back to action probabilities. The policy is trained in $\mathcal{E}$, so its output dimension never changes when the action set changes; only the decoder's output head is expanded for new actions or masked for removed actions. Fine-tuning the decoder is regularized by elastic weight consolidation using the Fisher information of past decoder parameters, while the encoder is fine-tuned freely to keep the representation space plastic. The intended effect is that the policy lives on stable latent dynamics, analogous to the cortical representation of movement, while the decoder plays the role of adaptable motor mapping.

What would settle it

On a small tabular MDP, compute the optimal policy with action set {left, forward} and with {left, forward, sprint}, and check whether the probability of 'left' in every state is identical; if it differs in any state, the optimal-policy-consistency premise fails, and AACL's objective would require unlearning rather than pure generalization. A second check is to train AACL on an expansion sequence where the added action deliberately dominates a shared action, and observe whether the method can or should preserve the old action's probability.

Watch

Extended reading notes

Core claim

The central claim is that a policy can be generalized across different action spaces by decoupling it from the action space: instead of mapping states to individual actions, the policy maps states to points in a learned action-representation space, and a separate decoder maps those points to the probability of each available action. Because the decoder is the only component that touches the action set, expanding the action space means growing the decoder, contracting it means masking outputs, and both changes are followed by EWC-constrained fine-tuning that preserves previously learned mappings. The paper's experiments are intended to show that this separation yields the best balance of stability and plasticity among the compared methods, with forward transfer (zero-shot generalization to unseen action sets) the clearest benefit. The overarching framing is that the CL-DC objective, average expected return over all past action sets, is the right metric for capability-changing lifelong learning.

Load-bearing premise

The framework assumes that for every state, the optimal choice among actions that two tasks share is the same in both tasks, no matter what other actions are available; if adding or removing an action changes which shared action is best, the CL-DC objective becomes ill-posed and the benchmark would not measure pure action-space generalization.

Editorial extensions

If this is right

  • When the action space expands, AACL adds neurons to the decoder without touching the policy, so the agent can start using new actions immediately and evaluate old ones without catastrophic forgetting.
  • When the action space contracts, masking removed actions lets the same policy keep functioning on the smaller action set, which the authors report is the harder direction for all baselines.
  • Training on earlier action sets provides forward transfer: evaluation on later action sets improves before any training on them, so knowledge from old capabilities accelerates learning of new ones.
  • The released benchmark gives future CL-DC work a common set of environments and metrics—continual return, forgetting, and forward transfer—so different methods can be compared on capability changes rather than only environment changes.
  • Because the policy is independent of action count, longer sequences that alternate expansion and contraction can be handled with the same mechanism while maintaining performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats the optimal-policy-consistency assumption as given. If adding or removing an action genuinely changes which shared action is best in a state, the same framework would be forced to unlearn, and the benchmark results would not isolate generalization—so the most informative follow-up is to measure, on each environment, how often adding an action changes the optimal probability of exi
  • The same encoder-decoder decoupling could plausibly be applied to continual classification with open label sets, where the policy is analogous to a feature extractor and the decoder is analogous to the classifier head; the paper's learning of action representations from their state effects, however, is specific to reinforcement learning.
  • The ablation results suggest a sharp division of labor: regularization on the encoder hurts forward transfer, while regularization on the decoder helps stability. This could mean future methods should use asymmetric protection, penalizing or freezing only read-out layers, a pattern worth testing in non-RL continual learning.
  • A direct extension the paper only sketches is continuous action spaces; the same principle would require the decoder to output dimension-wise ranges rather than discrete action probabilities, which may need a fundamentally different representation-learning loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Continual Learning with Dynamic Capabilities (CL-DC), in which the agent's action space changes across tasks while the underlying task logic remains similar. The authors propose AACL, which decouples the policy from the action space by learning an action representation space through a reward-agnostic inverse-dynamics objective (Eq. 3), training the policy on that representation space, and adaptively fine-tuning the decoder under action-space expansion/contraction with EWC regularization (Eq. 6). Experiments are conducted on MiniGrid, Bigfish, and Atlantis under expansion, contraction, and combined sequences, comparing AACL with six continual-learning and deep-RL baselines via continual return, forgetting, and forward transfer. The main claim is that decoupling the policy from the action space enables efficient policy generalization to new action spaces.

Significance. The CL-DC problem formulation is a useful extension of continual reinforcement learning, and the idea of building an action representation space from inverse dynamics is well motivated and technically plausible. The empirical study is broad: three environments, three task-sequence situations, 10 seeds on MiniGrid, 5 seeds on Bigfish and Atlantis, ablations of the regularization variants, longer sequences, hyperparameter sensitivity, and t-SNE visualizations. The paper also includes an explicit limitations section. If the empirical claims are substantiated after resolving the evaluation-protocol ambiguities, the contribution would be a meaningful step for CL under dynamic action spaces. However, the contraction evaluation mechanism is currently underspecified, the optimal-policy consistency assumption is not validated on the chosen benchmarks, and the closest prior method (LAICA) is not compared empirically; these issues need to be addressed before the central claims can be fully accepted.

major comments (5)
  1. [Section 4.3, Algorithm 1 (Appendix B), Tables 1 and 8] The contraction evaluation protocol is not reproducible. Section 4.3 states that on contraction the outputs corresponding to removed actions are masked, and Algorithm 1 returns only the final policy π_Eθ and decoder g_δ after all tasks. However, the contraction results in Tables 1 and 8 report forgetting and forward transfer that require evaluating earlier tasks with larger action spaces, e.g., the 7-Action task after a 7→5→3 sequence. If the final decoder has only unmasked outputs for the current action space, it cannot produce a meaningful distribution over the seven actions of Task 1; if per-task masks or decoder snapshots are stored and switched during evaluation, that mechanism is not described. Please specify (a) how old action spaces are evaluated, (b) whether masked output units remain frozen or are re-enabled, and (c) how probability mass is renormalized during evaluation. Without this clarification, the reported AACL forgetting of 0.04 in Table 1 (contraction) and the corresponding forward-transfer numbers are not reproducible and do not directly support the single-decoder generalization claimed in Section 4.1.
  2. [Appendix A.1, Eq. (9); Section 3.2] The formalization assumes optimal-policy consistency (Eq. 9): for any shared action, the optimal policies of different tasks assign the same probability to that action. This assumption is load-bearing because if it fails, the objective in Eq. (1) may be impossible for any single policy to satisfy on shared actions, and a performance drop after contraction could reflect a changed optimal policy rather than catastrophic forgetting. The benchmark tasks appear likely to violate this assumption: in Bigfish, removing left/right and diagonal actions changes the optimal way to pursue prey, and in MiniGrid, adding diagonal movement actions changes the optimal choice among earlier actions. Please test the assumption empirically—for example, train task-specific policies, evaluate them on shared states, and report the divergence between their action distributions on shared actions—or construct tasks that provably satisfy Eq. (9).
  3. [Section 5.2 and Appendix C.2] The closest prior method for changing action sets, LAICA, is not compared experimentally. The related work identifies LAICA as directly addressing expansion of action spaces, but the experiments compare only against methods designed for static action spaces, adapted by using the largest action space of all tasks. Since the central claim is that AACL handles CL-DC better than existing approaches, the expansion setting at least should include a LAICA-based baseline, or the paper should argue explicitly why LAICA cannot be adapted to the benchmark. The current comparisons do not establish superiority over the most relevant prior method.
  4. [Section 5.1 and Appendix D.2] The baseline adaptation protocol is underspecified. The paper says baselines are adapted to CL-DC by using the largest action space of all tasks, but it does not specify how a baseline acts when the current task's action space is smaller, e.g., in a contraction sequence. If the agent samples from all output units, it executes invalid actions; if it masks and renormalizes, this is an additional mechanism that should be described and applied consistently. The ALL upper-bound baseline also needs a precise definition of how it accesses all actions during contraction tasks. This protocol is needed to interpret the large gaps in Tables 1 and 2.
  5. [Section 4.3, Eq. (6)] The expansion mechanism is internally inconsistent with the EWC regularization. Section 4.3 says that on expansion the parameters of old neurons are fixed and only new neurons are initialized, yet Eq. (6) applies an EWC regularizer to all decoder parameters with Fisher terms from previous tasks. If old parameters are truly frozen, the EWC term on those parameters is vacuous; if they are trainable with soft constraints, calling them 'fixed' is inaccurate. Please clarify which mechanism is actually used, because this determines whether AACL is an architecture-preserving method or a regularization-based method and affects the interpretation of the ablation results.
minor comments (6)
  1. [Algorithm 1 (Appendix B)] The outer loop variable t is not used and the task index i is not incremented; the loop should run over tasks i = 1, ..., N. Also, 'See Task' should read 'Train on Task'.
  2. [Appendix A.2] Items 1 and 2 in the application example are identical: 'The underlying dynamics of the task do not fundamentally change.' appears twice and should be corrected.
  3. [Table 4 (Appendix D.3)] The policy output head is listed as 'Linear 5 512+9+1 7'; for Bigfish with up to nine actions the output dimension should be 9, and for Atlantis with up to four actions the input dimension 9 appears inconsistent. Please correct the table.
  4. [Appendix D.4] The statement that AACL hyperparameters were not tuned is difficult to reconcile with the specific value λ = 2 × 10^4 in Table 5; please state how λ was selected, for example from a prior study or a coarse validation check.
  5. [Eq. (12), Appendix D.5] The forward-transfer definition uses p_{i−1,j} for tasks j > i that have not yet been trained; please state explicitly that these are evaluation returns obtained by zero-shot evaluation on future tasks, as described in Section 5.2.
  6. [Appendix F] Appendix F is a heading with no content; either fill it or remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AACL's reported action-space generalization is an independent empirical outcome, not a fitted parameter or a self-citation chain.

full rationale

Walking the paper's derivation chain: the action representation is learned from a reward-free inverse-dynamics cross-entropy objective (Eq. 3), the policy is trained afterward on the representation space (Eq. 5), and the decoder is adapted by structural updates plus EWC regularization (Eq. 6). The reported continual-return, forgetting, and forward-transfer metrics (Eqs. 10-12) are computed from evaluation returns, not from any quantity fitted during training, and Appendix D.4 explicitly states that no hyperparameter search was performed for AACL. I found no equation in which a predicted return reduces algebraically to a fitted parameter, a regularization term, or the SSL loss. The paper cites its own prior work (e.g., Pan et al. 2025) only as background for general CRL settings, never as the source of the AACL construction or as a uniqueness theorem, so no load-bearing claim rests on a same-author citation. One caveat, flagged and weighed, is a reproducibility gap rather than circularity: Algorithm 1 returns only the final policy and decoder, while Tables 1 and 8 report forgetting and forward transfer for earlier, larger action spaces in the contraction setting; Section 4.3 says contracted outputs are masked but does not explicitly state whether the mask is removed during evaluation. This makes the contraction numbers incompletely specified as an empirical matter, but it does not make the claimed generalization equivalent to the method's inputs. Overall, the central claim is self-contained and empirically testable, so no significant circularity is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three hand-set hyperparameters (lambda, representation size, exploration steps) and on the problem-definition assumptions in Appendix A.1, especially the optimal-policy consistency constraint (Eq. 9). No new physical entities are postulated.

free parameters (3)
  • EWC regularization coefficient lambda = 2e4
    Regularization weight in Eq. 6; set by hand, sensitivity analyzed in Appendix E.4.
  • Action representation size = 256
    Dimension of the latent action space e; set by hand, sensitivity analyzed in Appendix E.4.
  • Exploration steps per new task = 1e4
    Number of transitions collected by a random policy before fine-tuning; the paper states this is not tuned (Appendix D.4).
assumptions (5)
  • standard math MDP formulation and standard RL background
    Used in Section 3.1 to define the CL-DC problem.
  • domain assumption Discrete and finite action spaces
    Section 3.2 restricts the problem to discrete finite action sets; continuous action spaces are listed as future work in Appendix G.2.
  • domain assumption Shared actions have identical transition and reward functions across tasks
    Appendix A.1, Eqs. 7 and 8; this defines the 'same task logic' condition.
  • domain assumption Optimal policy consistency for shared actions
    Appendix A.1, Eq. 9; assumes adding or removing other actions does not change the optimal choice of a shared action. This is the weakest premise of the problem definition.
  • domain assumption Inverse dynamics (predicting action from state and next state) yields an action representation useful for policy generalization
    Section 4.2 and Appendix C; the method's effectiveness depends on this transferable representation, which is empirically validated only on the three benchmark environments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces." pith.science (2026). https://pith.science/paper/VPIIIN4P

@misc{pith2026250605702,
  author       = {Pith},
  title        = {Pith review of: Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPIIIN4P}},
  note         = {Machine review of arXiv:2506.05702}
}
read the original abstract

Continual Learning (CL) is a powerful tool that enables agents to learn a sequence of tasks, accumulating knowledge learned in the past and using it for problem-solving or future task learning. However, existing CL methods often assume that the agent's capabilities remain static within dynamic environments, which doesn't reflect real-world scenarios where capabilities dynamically change. This paper introduces a new and realistic problem: Continual Learning with Dynamic Capabilities (CL-DC), posing a significant challenge for CL agents: How can policy generalization across different action spaces be achieved? Inspired by the cortical functions, we propose an Action-Adaptive Continual Learning framework (AACL) to address this challenge. Our framework decouples the agent's policy from the specific action space by building an action representation space. For a new action space, the encoder-decoder of action representations is adaptively fine-tuned to maintain a balance between stability and plasticity. Furthermore, we release a benchmark based on three environments to validate the effectiveness of methods for CL-DC. Experimental results demonstrate that our framework outperforms popular methods by generalizing the policy across action spaces.

Figures

Figures reproduced from arXiv: 2506.05702 by the authors.

Figure 1
Figure 1. An example of two CL problems. Exist￾ing CL: A robot uses two fingers to grasp objects while the objects or grasping way changes. CL￾DC: A robot initially trained with two fingers is upgraded to four fingers or loses a finger but must continue grasping objects. Continual Learning (CL, a.k.a. lifelong learn￾ing) is an emerging research field that aims to emulate the human capacity for lifelong learn￾ing and tackles t… view at source ↗
Figure 2
Figure 2. Different challenges of two problems. Existing CL: After the environment changes, the number of actions remains constant, while the prob￾ability distribution shifts (trend of the red line). CL-DC: After the action space changes, the num￾ber of actions changes, while the probability distri￾bution is relatively stable. While existing works in RL (Chandak et al., 2020; Ding et al., 2023) have made initial explo￾rations… view at source ↗
Figure 3
Figure 3. The overview of AACL. The tasks with different action spaces are learned sequentially. Each task consists of two stages: the exploration stage (green) and the learning stage (yellow). The former aims to build an action representation space, and the latter aims to learn a policy based on the learned space. This framework separates the policy from the action space, allowing the agent’s gener￾alization across different… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Performance of eight methods on three MiniGrid tasks in the expansion situation. 0 1M 2M 3M 4M 5M 6M 7M 8M 9M 0 0.2 0.4 0.6 0.8 1 ALL AACL Mask CLEAR Online-EWC EWC FT IND Task1 Step Expected Return (a) Task1: Seven actions 0 1M 2M 3M 4M 5M 6M 7M 8M 9M 0 0.2 0.4 0.6 0.…
Figure 5
Figure 5. Figure 5: Performance of eight methods on three MiniGrid tasks in the contraction situation. (Kirkpatrick et al., 2017) and online-EWC (Schwarz et al., 2018); and one architecture-based method, Mask (Ben-Iwhiwhu et al., 2023). Additionally, we take the DRL methods trained with f…
Figure 6
Figure 6. Figure 6: Performance of eight methods on three Bigfish tasks in the contraction situation [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The screenshots of actions in MiniGrid. The transparent white area represents the agent’s [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The screenshots of Bigfish and Atlantis. The texture and objects of Bigfish are procedurally [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Average runtime of seven methods on three MiniGrid tasks and three Bigfish tasks. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Performance of seven methods on three MiniGrid tasks in the expansion & contraction situation. 0 1M 2M 3M 4M 5M 6M 7M 8M 9M 0 0.2 0.4 0.6 0.8 1 AACL Mask CLEAR Online-EWC EWC FT IND Task1 Step Expected Return (a) Task 1: Five actions 0 1M 2M 3M 4M 5M 6M 7M 8M 9M 0 0.2…
Figure 11
Figure 11. Figure 11: Performance of seven methods on three MiniGrid tasks in the contraction & expansion situation. Figures 10 and 11, along with [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Performance of seven methods in a longer sequence (five MiniGrid tasks). [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Hyperparameter sensitivity analysis for AACL across three MiniGrid tasks in the situations [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: 2D t-SNE visualizations of learned action representations on MiniGrid tasks, colored by [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Performance of eight methods on three Bigfish tasks in the expansion situation [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Performance of eight methods on three Atlantis tasks in the contraction situation. among these methods during these tasks differs from that observed in Bigfish. This variability highlights the challenges inherent in generalizing policies across diverse actions. Accord…
Figure 17
Figure 17. Figure 17: Performance of eight methods on three [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 72 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abbas, Z.; Zhao, R.; Modayil, J.; White, A.; and Machado, M. C. 2023. Loss of Plasticity in Continual Deep Reinforcement Learning. In CoLLAs , volume 232, 620--636

    work page 2023

  4. [4]

    P.; and Singh, S

    Abel, D.; Barreto, A.; Van Roy, B.; Precup, D.; van Hasselt, H. P.; and Singh, S. 2023. A Definition of Continual Reinforcement Learning. In NeurIPS , volume 36, 50377--50407

    work page 2023

  5. [5]

    Y.; Konidaris, G.; and Littman, M

    Abel, D.; Jinnai, Y.; Guo, S. Y.; Konidaris, G.; and Littman, M. 2018. Policy and Value Transfer in Lifelong Reinforcement Learning. In ICML , volume 80, 20--29

    work page 2018

  6. [6]

    G.; Naddaf , Y.; Veness , J.; and Bowling , M

    Bellemare , M. G.; Naddaf , Y.; Veness , J.; and Bowling , M. 2013. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47: 253--279

    work page 2013

  7. [7]

    K.; Kolouri, S.; and Soltoggio, A

    Ben-Iwhiwhu, E.; Nath, S.; Pilly, P. K.; Kolouri, S.; and Soltoggio, A. 2023. Lifelong Reinforcement Learning with Modulating Masks. Transactions on Machine Learning Research

    work page 2023

  8. [8]

    Berseth, G.; Zhang, Z.; Zhang, G.; Finn, C.; and Levine, S. 2022. Co MPS : Continual Meta Policy Search. In ICLR

    work page 2022

Show all 79 references
  1. [9]

    J.; Shomrat, T.; and Levin, M

    Blackiston, D. J.; Shomrat, T.; and Levin, M. 2015. The Stability of Memories During Brain Remodeling: A Perspective. Communicative & Integrative Biology, 8(5): e1073424

    2015

  2. [10]

    Caccia, M.; Mueller, J.; Kim, T.; Charlin, L.; and Fakoor, R. 2023. Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges. In CoLLAs , volume 232, 89--119

    2023

  3. [11]

    Chandak, Y.; Theocharous, G.; Kostas, J.; Jordan, S.; and Thomas, P. 2019. Learning Action Representations for Reinforcement Learning. In ICML , volume 97, 941--950

    2019

  4. [12]

    Chandak, Y.; Theocharous, G.; Nota, C.; and Thomas, P. 2020. Lifelong Learning with a Changing Action Set. In AAAI, volume 34, 3373--3380

    2020

  5. [13]

    S.; and Terry, J

    Chevalier-Boisvert, M.; Dai, B.; Towers, M.; Perez-Vicente, R.; Willems, L.; Lahlou, S.; Pal, S.; Castro, P. S.; and Terry, J. 2023. Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks. In NeurIPS , volume 36, 73383--73394

    2023

  6. [14]

    M.; Cunningham, J

    Churchland, M. M.; Cunningham, J. P.; Kaufman, M. T.; Foster, J. D.; Nuyujukian, P.; Ryu, S. I.; and Shenoy, K. V. 2012. Neural population dynamics during reaching. Nature, 487(7405): 51--56

    2012

  7. [15]

    Cobbe, K.; Hesse, C.; Hilton, J.; and Schulman, J. 2020. Leveraging Procedural Generation to Benchmark Reinforcement Learning. In ICML , volume 119, 2048--2056

    2020

  8. [16]

    De Lange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2022. A Continual Learning Survey: Defying Forgetting in Classification Tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7): 3366--3385

    2022

  9. [17]

    D \' az-Rodr \' guez, N.; Lomonaco, V.; Filliat, D.; and Maltoni, D. 2018. Don't Forget, There Is More Than Forgetting: New Metrics for Continual Learning. arXiv preprint arXiv:1810.13166

    2018 arXiv

  10. [18]

    Dick, J.; Nath, S.; Peridis, C.; Benjamin, E.; Kolouri, S.; and Soltoggio, A. 2024. Statistical Context Detection for Deep Lifelong Reinforcement Learning. In CoLLAs

    2024

  11. [19]

    Ding, W.; Jiang, S.; Chen, H.-W.; and Chen, M.-S. 2023. Incremental Reinforcement Learning with Dual-Adaptive -Greedy Exploration. In AAAI , volume 37, 7387--7395

    2023

  12. [20]

    M.; Gabrieli, J

    D'Mello, A. M.; Gabrieli, J. D.; and Nee, D. E. 2020. Evidence for Hierarchical Cognitive Control in the Human Cerebellum. Current Biology, 30(10): 1881--1892.e3

    2020

  13. [21]

    J.; Davidian, D.; Morokuma, J.; and Levin, M

    Emmons-Bell, M.; Durant, F.; Tung, A.; Pietak, A.; Miller, K.; Kane, A.; Martyniuk, C. J.; Davidian, D.; Morokuma, J.; and Levin, M. 2019. Regenerative Adaptation to Electrochemical Perturbation in Planaria: A Molecular Analysis of Physiological Plasticity. iScience , 22: 147--165

    2019

  14. [22]

    Espeholt, L.; Soyer, H.; Munos, R.; Simonyan, K.; Mnih, V.; Ward, T.; Doron, Y.; Firoiu, V.; Harley, T.; Dunning, I.; Legg, S.; and Kavukcuoglu, K. 2018. IMPALA : Scalable Distributed Deep- RL with Importance Weighted Actor-Learner Architectures. In ICML , volume 80, 1407--1416

    2018

  15. [23]

    Eysenbach, B.; Zhang, T.; Levine, S.; and Salakhutdinov, R. R. 2022. Contrastive Learning as Goal-Conditioned Reinforcement Learning. In NeurIPS , volume 35, 35603--35620

    2022

  16. [24]

    Fang, C.; and Stachenfeld, K. 2024. Predictive Auxiliary Objectives in Deep RL Mimic Learning in the Brain. In ICLR

    2024

  17. [25]

    P.; and Robbins, T

    Friedman, N. P.; and Robbins, T. W. 2022. The Role of Prefrontal Cortex in Cognitive Control and Executive Function. Neuropsychopharmacology, 47(1): 72--89

    2022

  18. [26]

    A.; Perich, M

    Gallego, J. A.; Perich, M. G.; Chowdhury, R. H.; Solla, S. A.; and Miller, L. E. 2020. Long-term stability of cortical population dynamics underlying consistent behavior. Nature Neuroscience, 23(2): 260--270

    2020

  19. [27]

    Gaya, J.-B.; Doan, T.; Caccia, L.; Soulier, L.; Denoyer, L.; and Raileanu, R. 2022. Building a Subspace of Policies for Scalable Continual Learning. In NeurIPS DRL Workshop

    2022

  20. [28]

    S.; Ivry, R

    Gazzaniga, M. S.; Ivry, R. B.; and Mangun, G. 2019. Cognitive Neuroscience. The Biology of the Mind. New York: W.W. Norton & Company. ISBN 978-7-5184-4043-6

    2019

  21. [29]

    P.; and Pellizzer, G

    Georgopoulos, A. P.; and Pellizzer, G. 1995. The mental and the neural: Psychological and neural studies of mental rotation and memory scanning. Neuropsychologia, 33(11): 1531--1547

    1995

  22. [30]

    Haar, S.; Donchin, O.; and Dinstein, I. 2015. Dissociating Visual and Motor Directional Selectivity using Visuomotor Adaptation. Journal of Neuroscience, 35(17): 6813--6821

    2015

  23. [31]

    Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2020. Dream to Control: Learning Behaviors by Latent Imagination. In ICLR

    2020

  24. [32]

    He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR

    2020

  25. [33]

    S.; and Strick, P

    Kakei, S.; Hoffman, D. S.; and Strick, P. L. 1999. Muscle and Movement Representations in the Primary Motor Cortex. Science, 285(5436): 2136--2139

    1999

  26. [34]

    Kaplanis, C.; Shanahan, M.; and Clopath, C. 2019. Policy Consolidation for Continual Reinforcement Learning. In ICML , volume 97, 3242--3251

    2019

  27. [35]

    Kessler, S.; Parker-Holder, J.; Ball, P.; Zohren, S.; and Roberts, S. J. 2022. Same State, Different Task: Continual Reinforcement Learning without Interference. In AAAI , volume 36, 7143--7151

    2022

  28. [36]

    Khetarpal, K.; Riemer, M.; Rish, I.; and Precup, D. 2022. Towards Continual Reinforcement Learning: A Review and Perspectives. Journal of Academia and Industrial Research, 75: 1401--1476

    2022

  29. [37]

    A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R

    Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2017. Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of t...

    2017

  30. [38]

    Kriegman, S.; Walker, S.; Shah, D.; Levin, M.; Kramer-Bottiglio, R.; and Bongard, J. 2019. Automated Shapeshifting for Function Recovery in Damaged Robots. In RSS

    2019

  31. [39]

    P.; Chakravarthi Raja, S.; Cheney, N.; Clune, J.; Daram, A.; Fusi, S.; Helfer, P.; Kay, L.; Ketz, N.; Kira, Z.; Kolouri, S.; Krichmar, J

    Kudithipudi, D.; Aguilar-Simon, M.; Babb, J.; Bazhenov, M.; Blackiston, D.; Bongard, J.; Brna, A. P.; Chakravarthi Raja, S.; Cheney, N.; Clune, J.; Daram, A.; Fusi, S.; Helfer, P.; Kay, L.; Ketz, N.; Kira, Z.; Kolouri, S.; Krichmar, J. L.; Kriegman, S.; Levin, M.; Madireddy, S...

    2022

  32. [40]

    Kwiatkowski, R.; and Lipson, H. 2019. Task-agnostic self-modeling machines. Science Robotics, 4(26): eaau9354

    2019

  33. [41]

    Laskin, M.; Lee, K.; Stooke, A.; Pinto, L.; Abbeel, P.; and Srinivas, A. 2020. Reinforcement Learning with Augmented Data. In NeurIPS , volume 33, 19884--19895

    2020

  34. [42]

    Laskin, M.; Srinivas, A.; and Abbeel, P. 2020. CURL : Contrastive Unsupervised Representations for Reinforcement Learning. In ICML , volume 119, 5639--5650

    2020

  35. [43]

    Li, X.; Shang, J.; Das, S.; and Ryoo, M. 2022. Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels? In NeurIPS , volume 35, 30865--30881

    2022

  36. [44]

    Li, Y.; Liu, J.; Yang, L.; Pan, C.; Wang, X.; and Yang, X. 2024 a . Three-way open intent classification with nearest centroid-based representation. Information Sciences, 681: 121251

    2024

  37. [45]

    Li, Y.; Yang, X.; Wang, H.; Wang, X.; and Li, T. 2024 b . Learning to Prompt Knowledge Transfer for Open-World Continual Learning. In AAAI , volume 38, 13700--13708

    2024

  38. [46]

    Liu, J.; HAO, J.; Ma, Y.; and Xia, S. 2024. Unlock the Cognitive Generalization of Deep Reinforcement Learning via Granular Ball Representation. In ICML

    2024

  39. [47]

    Lomonaco, V.; Desai, K.; Culurciello, E.; and Maltoni, D. 2020. Continual Reinforcement Learning in 3D Non-Stationary Environments. In CVPR Workshops

    2020

  40. [48]

    Maaten, L. V. D.; and Hinton, G. 2008. Visualizing Data using T-SNE . Journal of Machine Learning Research, 9(86): 2579--2605

    2008

  41. [49]

    Mallya, A.; Davis, D.; and Lazebnik, S. 2018. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights. In ECCV , 67--82

    2018

  42. [50]

    Mallya, A.; and Lazebnik, S. 2018. PackNet : Adding Multiple Tasks to a Single Network by Iterative Pruning. In CVPR , 7765--7773

    2018

  43. [51]

    D.; and van de Weijer, J

    Masana, M.; Liu, X.; Twardowski, B.; Menta, M.; Bagdanov, A. D.; and van de Weijer, J. 2022. Class-Incremental Learning: Survey and Performance Evaluation on Image Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5513–5533

    2022

  44. [52]

    Mazzaglia, P.; Catal, O.; Verbelen, T.; and Dhoedt, B. 2022. Curiosity-Driven Exploration via Latent Bayesian Surprise. In AAAI, volume 36, 7752--7760

    2022

  45. [53]

    Pan, C.; Ren, L.; Feng, Y.; Xiong, L.; Wei, W.; Li, Y.; and Xin, Y. 2025. Multi-granularity Knowledge Transfer for Continual Reinforcement Learning. In IJCAI

    2025

  46. [54]

    A.; and Darrell, T

    Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017. Curiosity-driven exploration by self-supervised prediction. In ICML, 2778–2787

    2017

  47. [55]

    Pong, V.; Dalal, M.; Lin, S.; Nair, A.; Bahl, S.; and Levine, S. 2020. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. In ICML , volume 119, 7783--7792

    2020

  48. [56]

    Powers, S.; Xing, E.; Kolve, E.; Mottaghi, R.; and Gupta, A. 2022. CORA : Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents. In CoLLAs , volume 199, 705--743

    2022

  49. [57]

    Rolnick, D.; Ahuja, A.; Schwarz, J.; Lillicrap, T.; and Wayne, G. 2019. Experience Replay for Continual Learning. In NeurIPS , volume 32

    2019

  50. [58]

    A.; Rabinowitz, N

    Rusu, A. A.; Rabinowitz, N. C.; Desjardins, G.; Soyer, H.; Kirkpatrick, J.; Kavukcuoglu, K.; Pascanu, R.; and Hadsell, R. 2016. Progressive Neural Networks. arXiv preprint arXiv:1606.04671

    2016 arXiv

  51. [59]

    Sahisnu Mazumder, B. L. 2024. Lifelong and Continual Learning Dialogue Systems. ISBN 978-3-031-48188-8

    2024

  52. [60]

    Sch \"o pf, P.; Auddy, S.; Hollenstein, J.; and Rodriguez-sanchez, A. 2022. Hypernetwork- PPO for Continual Reinforcement Learning. In NeurIPS DRL Workshop

    2022

  53. [61]

    Schrittwieser, J.; Antonoglou, I.; Hubert, T.; Simonyan, K.; Sifre, L.; Schmitt, S.; Guez, A.; Lockhart, E.; Hassabis, D.; Graepel, T.; et al. 2020. Mastering Atari, Go, Chess and Shogi by Planning With a Learned Model. Nature, 588(7839): 604--609

    2020

  54. [62]

    W.; Pascanu, R.; and Hadsell, R

    Schwarz, J.; Czarnecki, W.; Luketina, J.; Grabska-Barwinska, A.; Teh, Y. W.; Pascanu, R.; and Hadsell, R. 2018. Progress & Compress: A Scalable Framework for Continual Learning. In ICML , volume 80, 4528--4537

    2018

  55. [63]

    M.; Pascanu, R.; Latham, P

    Schwarz, J.; Jayakumar, S. M.; Pascanu, R.; Latham, P. E.; and Teh, Y. W. 2021. Powerpropagation: A Sparsity Inducing Weight Reparameterisation. In NeurIPS , 28889--28903

    2021

  56. [64]

    K.; Kim, J.; and Kim, J

    Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017. Continual Learning with Deep Generative Replay. In NeurIPS , volume 30

    2017

  57. [65]

    Stooke, A.; Lee, K.; Abbeel, P.; and Laskin, M. 2021. Decoupling Representation Learning from Reinforcement Learning. In ICML , volume 139, 9870--9879

    2021

  58. [66]

    Wang, L.; Zhang, X.; Su, H.; and Zhu, J. 2024 a . A Comprehensive Survey of Continual Learning: Theory, Method and Application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8): 5362--5383

    2024

  59. [67]

    Wang, N.; Zhang, D.; and Wang, Y. 2020. Learning to Navigate for Mobile Robot with Continual Reinforcement Learning. In CCC , 3701--3706

    2020

  60. [68]

    Wang, X.; Chen, Y.; and Zhu, W. 2022. A Survey on Curriculum Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9): 4555--4576

    2022

  61. [69]

    Wang, X.; Wang, S.; Liang, X.; Zhao, D.; Huang, J.; Xu, X.; Dai, B.; and Miao, Q. 2024 b . Deep Reinforcement Learning: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35(4): 5064--5078

    2024

  62. [70]

    Wang, Z.; Li, H.-X.; and Chen, C. 2019. Incremental Reinforcement Learning in Continuous Spaces via Policy Relaxation and Importance Weighting. IEEE Transactions on Neural Networks and Learning Systems, 31(6): 1870--1883

    2019

  63. [71]

    S.; Galea, J

    Weightman, M.; Lalji, N.; Lin, C.-H. S.; Galea, J. M.; Jenkinson, N.; and Miall, R. C. 2023. Short Duration Event Related Cerebellar TDCS Enhances Visuomotor Adaptation. Brain Stimulation, 16(2): 431--441

    2023

  64. [72]

    Wolczyk, M.; Zajac, M.; Pascanu, R.; Kucinski, L.; and Milos, P. 2021. Continual World : A Robotic Benchmark For Continual Reinforcement Learning. In NeurIPS , 28496--28510

    2021

  65. [73]

    Yarats, D.; Kostrikov, I.; and Fergus, R. 2021. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. In ICLR

    2021

  66. [74]

    Ye, D.; Liu, Z.; Sun, M.; Shi, B.; Zhao, P.; Wu, H.; Yu, H.; Yang, S.; Wu, X.; Guo, Q.; et al. 2020. Mastering Complex Control in Moba Games With Deep Reinforcement Learning. In AAAI , volume 34, 6672--6679

    2020

  67. [75]

    Yue, W.; Liu, B.; and Stone, P. 2023. t- DGR : A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Making. In NeurIPS ALOE Workshop

    2023

  68. [76]

    Zhang, H.; Lei, Y.; Gui, L.; Yang, M.; He, Y.; Wang, H.; and Xu, R. 2024. CPPO : Continual Learning for Reinforcement Learning with Human Feedback. In ICLR

    2024

  69. [77]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  70. [78]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  71. [79]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.