Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation

Kerim Erekmen; Muhammad Burhan Hafez

REVIEW 5 major objections 5 minor 48 references

Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces Task-Agnostic Policy Distillation (TAPD), a framework that adds a curiosity-driven, task-agnostic pre-training phase to Progress & Compress so that a continual reinforcement learning agent transfers exploratory…

desk verdict Useful incremental idea, but the central claim needs a distillation ablation before it is supported. read the letter →

arxiv 2411.16532 v1 pith:OH6NNPQ5 submitted 2024-11-25 cs.LG

Muhammad Burhan Hafez , Kerim Erekmen This is my paper · ORCID

classification cs.LG

keywords continuallearningreinforcementtask-agnosticpolicydistillationintrinsicmotivationcuriosity-drivenexplorationAtari2600self-supervisedprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a continual reinforcement learning agent can learn a sequence of tasks much more efficiently if it first spends a phase exploring without any task-specific goal, driven only by curiosity, and periodically distills what it discovers into a knowledge base. It introduces Task-Agnostic Policy Distillation (TAPD), which inserts this task-agnostic phase into the Progress & Compress framework: during the phase, an A2C policy maximizes an intrinsic reward equal to the log prediction error of a forward dynamics model, and the exploratory policy is repeatedly distilled into the knowledge base with online elastic weight consolidation. The agent then enters the standard progress and compress phases with a knowledge base that already contains general exploration skills, so downstream tasks are learned faster and with better sample efficiency. The authors evaluate TAPD on five Atari 2600 games—Pong, SpaceInvaders, BeamRider, DemonAttack, and AirRaid—and report that it outperforms Online EWC, Progressive Nets, and the Progress & Compress baseline across visits. A sympathetic reader would care because this addresses four obstacles to universal agents: catastrophic forgetting, positive forward transfer, scalability, and learning without task labels or clear task boundaries.

What carries the argument

The load-bearing mechanism is the task-agnostic phase, an alternating loop of curiosity-driven exploration and knowledge-base distillation that runs before any task-specific learning. The intrinsic reward is the log forward-model prediction error in feature space, so the agent systematically seeks states it cannot yet predict. Distillation transfers the exploratory action distribution of the active column into the knowledge base using KL divergence, and online EWC prevents the knowledge base from forgetting earlier distilled exploration policies; lateral connections from the knowledge base then feed the active column in both the task-agnostic and progress phases. This loop is what converts raw curiosity into reusable, task-independent knowledge.

What would settle it

Run TAPD's task-agnostic phase on SpaceInvaders and BeamRider, then measure downstream learning curves on Pong, DemonAttack, and AirRaid against a control that receives the same number of environment steps of uniformly random exploration (or starts directly in the progress phase with the same total budget). If the control matches or beats TAPD's scores, the claim that distilled curiosity knowledge transfers to unseen games is falsified. A second check removes distillation: train the same intrinsic-reward policy for the same steps but never distill into the knowledge base; if performance is unchanged, periodic policy distillation is not the operative mechanism.

Watch

Extended reading notes

Core claim

The central discovery is that exploration itself can be treated as a distillable asset. TAPD adds an alternating task-agnostic phase to Progress & Compress. In each cycle the active column is trained for x steps with A2C to maximize the intrinsic return $\mathbb{E}_\pi[\sum_n \gamma^n r^i_{t+n}]$, where $r^i_t = \log(L_{\mathrm{forward}} + \epsilon)$ and $L_{\mathrm{forward}} = \|\phi(s_{t+1}) - \hat\phi(s_{t+1})\|_2^2$ is the prediction error of a forward model $F(\phi(s_t), a_t)$; the inverse dynamics model of the Intrinsic Curiosity Module is deliberately omitted. The exploratory policy is then distilled into the knowledge base by minimizing a KL divergence between the active and knowledge-base policies plus an online-EWC penalty that protects older distilled knowledge. After this phase, downstream tasks are learned through the usual progress phase (A2C with lateral connections from the knowledge base) and compress phase (distillation into the knowledge base with online EWC). The authors claim this makes the agent solve downstream tasks more efficiently and report higher scores than all three baselines on each visit to all five Atari games, with lower policy entropy indicating more directed exploration.

Load-bearing premise

The central claim depends on the assumption that an exploration policy trained on two Atari games transfers useful exploratory knowledge to three unseen games, so that the measured performance gains come from that transferred knowledge rather than from extra training time or hyperparameter choices.

Editorial extensions

If this is right

With a curiosity-pretrained knowledge base, TAPD reaches high scores on Pong by roughly 0.6 million timesteps, where Online EWC, Progressive Nets, and Progress & Compress show little improvement after 1.5 million; the paper interprets this as improved sample efficiency.
Because the task-agnostic phase samples tasks uniformly and never uses extrinsic rewards or task IDs, the framework works when task boundaries are unknown, so a single policy can move between games without being told which game it is playing.
Distilling the exploratory policy into the knowledge base and protecting it with online EWC accumulates exploration behavior across games, which the authors link to lower policy entropy and a better exploration-exploitation balance during later visits.
TAPD keeps only two networks and does not grow a new column per task, so it avoids the quadratic parameter growth of Progressive Nets and remains applicable in memory-limited settings.
Across three visits to five tasks, TAPD reports the highest average normalized performance and lower variance across tasks than Progressive Nets and Progress & Compress, indicating positive forward transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the task-agnostic phase uses only two games (SpaceInvaders and BeamRider), the paper leaves open whether the transfer is general curiosity or familiarity with shooter-like dynamics; a direct test would run the phase on a disjoint set of games and compare downstream transfer.
One testable extension suggested by the mechanism is to mix intrinsic and extrinsic rewards during the progress phase, which could further help long-horizon tasks where rewards are sparse; the paper mentions this as future work rather than testing it.
The framework predicts that the knowledge base's forward-model coverage of pretraining environments determines downstream speedup; measuring prediction-error reduction on unseen games would give a quantitative proxy for transfer.
The reported gains could partly reflect the extra environment interaction of the task-agnostic phase rather than distillation itself, so an ablation that matches total timesteps across all methods would isolate the operative contribution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a plausible, clearly written empirical paper, but the main claim is not yet supported because the key ablation is missing. The task-agnostic phase adds two things at once—7.5M timesteps of curiosity-driven pretraining and periodic distillation into the knowledge base—and no experiment separates them. The gains over Progress & Compress could therefore come from the extra exploration alone, not from distilled knowledge.

What is genuinely new: the specific integration—alternating intrinsic-reward exploration with KL distillation into a knowledge base, then reusing the knowledge base via lateral connections in the Progress & Compress framework—is not in the cited literature. The components are all prior work, but the task-agnostic phase as an explicit pretraining step is a real combination, and the paper is the first to evaluate it in continual RL on Atari. The writing is clear, the related work is accurate, and the code is released.

The soft spots are real and proportionate. The missing ablation is load-bearing; without a run that does intrinsic-reward pretraining without distillation, the paper cannot identify what makes TAPD work, and the Pong narrative is speculation. The hyperparameters were tuned by Bayesian optimization on the same downstream tasks used for evaluation, which biases the comparison in TAPD's favor. There are no error bars or significance tests; the learning curves in Fig. 4 are averaged over 4 seeds, and Table 2 over 8, but no variance is shown in the score curves. The Meta-Environment description is inconsistent—Section 5.2 says BeamRider and SpaceInvaders, Appendix B.3 says Pong, SpaceInvaders, and BeamRider. Minor but should be fixed. Also, the Progress & Compress baseline is called 'reproduced' but is not validated against the original results, which matters for a fair comparison.

The citation pattern looks fine. The self-citations are to relevant work and not inflated.

Who is this for: people working in continual RL, especially on task-agnostic pretraining. It is not a field-reorganizing paper, but it is a reasonable incremental contribution. It deserves a serious referee: the question is well-posed and the result, if it holds up after the ablation, would be useful. My recommendation is to send it to peer review and require the missing ablation, variance reporting, and a corrected Meta-Environment description before acceptance.

Referee Report

5 major / 5 minor

Summary. The paper proposes Task-Agnostic Policy Distillation (TAPD), which augments the Progress & Compress continual RL framework of Schwarz et al. with a task-agnostic phase. In this phase, an A2C agent maximizes an intrinsic curiosity reward derived from forward-model prediction error, periodically distills its exploratory policy into a knowledge base using KL divergence and online EWC, and samples games uniformly from a 'Meta-Environment' without task boundaries. The authors claim that the resulting distilled task-agnostic knowledge improves sample efficiency and positive forward transfer on five Atari games, and they report comparisons against Progress & Compress, Online EWC, and Progressive Nets. The paper includes code and extensive implementation details in the appendices.

Significance. If the central claim is supported, the paper makes a useful contribution to continual deep RL by showing that self-supervised task-agnostic exploration can be distilled and reused for faster downstream learning. The proposed framework directly targets learning without task boundaries, which is a recognized gap in the Progress & Compress line of work. The release of code and the inclusion of three baselines are also strengths. However, the current experimental evidence does not yet isolate the mechanism responsible for the reported gains, and several experimental inconsistencies undermine the strength of the empirical conclusions.

major comments (5)

[§4.2, §5.3, Fig. 4, Table 2] No ablation isolates the effect of distillation from the effect of the extra intrinsic-reward pretraining. In TAPD (Section 4.2, Equation 4), the task-agnostic phase introduces two changes relative to Progress & Compress: (i) 7.5M timesteps of curiosity-driven exploration on selected games, and (ii) periodic distillation of the exploratory policy into the knowledge base via KL plus EWC (Appendix A, Eq. A1). The reported gains in Fig. 4 and Table 2 could therefore stem from the additional pretraining alone, particularly on SpaceInvaders and BeamRider, which appear in the task-agnostic phase. A control that runs the same intrinsic-reward pretraining for the same number of steps without compressing into the knowledge base is missing. This ablation is necessary to support the paper's central claim that distilled task-agnostic knowledge drives the improvement.
[§5.3, §5.4, Fig. 4, Table 2] The evaluation lacks error bars and significance tests. Fig. 4 reports averages over 4 seeds without variance; Table 2 reports averages over 8 seeds but no standard deviations; the two seed counts are inconsistent. On several tasks (e.g., DemonAttack and AirRaid in Table 2) the differences between TAPD and Progress & Compress are small relative to the between-visit fluctuations, so without confidence intervals or a paired test the claim that TAPD 'surpasses all three continual learning baselines' is not statistically supported.
[§5.1, Appendix B.3] Hyperparameters are tuned on the same downstream tasks used for evaluation. Section 5.1 states that Bayesian Hyperparameter Optimization was run 'to maximize normalized scores across tasks,' and Appendix B.3 lists multiple TAPD-specific hyperparameters (num-env-steps-agnostic, num-samples-drawn-in-task-agnostic-phase, ewc-lambda, etc.). Because the same five tasks are used for both tuning and reporting, the comparison to baselines is not a purely prospective test; the authors should either use a held-out validation task for tuning or report the sensitivity of the ranking to the tuned hyperparameters.
[§5.2, Appendix B.3] There is a direct contradiction about which games appear in the Meta-Environment during the task-agnostic phase. Section 5.2 states 'the Meta-Environment, which includes BeamRider and SpaceInvaders,' while Appendix B.3 states 'The Meta-Environment includes Pong (P), SpaceInvaders (S), and BeamRider (B).' Table B3 lists only S and B. If Pong was in fact part of the task-agnostic phase, then the claim in Section 5.3 that TAPD achieved faster Pong learning through transfer to an unseen game is unsupported, because the agent would have had direct pretraining on Pong.
[§5.3, Fig. 4] The reproduced Progress & Compress baseline is not validated against the original results of Schwarz et al. (2018). Fig. 4 and Table 2 compare against a 'reproduced' baseline, but the paper provides no evidence that this reproduction matches the original method's behavior or scores on the same games. If the baseline underperforms its published implementation, the relative improvement of TAPD is overstated. The authors should report the original published Progress & Compress numbers alongside their reproduction, or explain any discrepancies.

minor comments (5)

[§5.1] There is a typo: 'In this paper, we focuses on five different Atari games' should read 'we focus'.
[Appendix A] The symbol gamma is overloaded: it denotes both the reward discount factor and the forgetting constant in Eq. A1. Please use distinct notation for these two quantities.
[Table 3] The training-phase order for TAPD is listed as 'DemonAttack, Pong, BeamRider, SpaceInvaders, AirRaid,' but the experiments report the order Pong, SpaceInvaders, BeamRider, DemonAttack, AirRaid. This appears to be a transcription error and should be corrected.
[References] Progressive Nets are cited as [48] (Xu et al., 'Adaptive progressive continual learning'), but the method described is the original Progressive Networks of Rusu et al. [12]. Please align the citation with the actual method being used.
[Fig. 3, Fig. 4, Table 2] The number of seeds is reported inconsistently: Fig. 3 says 8 seeds, Fig. 4 says 4 seeds, and Table 2 says 8 seeds. Please state the exact seed count used for each result and explain any discrepancy.

Circularity Check

1 steps flagged · score 4.0 of 10

Task-agnostic pretraining overlaps two of the five evaluation games, so part of the reported improvement is forced by the training distribution; the central claim retains independent support on unseen games.

fitted input called prediction [Section 5.2 (Task-Agnostic Phase), Section 5.3 (Evaluation), and Table B3]
"The task-agnostic phase receives tasks from the Meta-Environment, which includes BeamRider and SpaceInvaders. ... The tasks Pong, SpaceInvaders, BeamRider, DemonAttack, and AirRaid are sequentially learned. ... TAPD appears to remember this environment, which potentially contributes to its superior performance compared to all other algorithms in the initial phase."

The headline 'prediction' is improved downstream sample efficiency from distilled task-agnostic knowledge. The task-agnostic phase is trained on SpaceInvaders and BeamRider, and the downstream evaluation in Section 5.3 scores exactly those games among the five. Superior TAPD scores on SpaceInvaders and BeamRider are therefore partly forced by direct training on the same games, not by transfer of distilled knowledge; the paper itself concedes that TAPD 'remembers' SpaceInvaders. Appendix B.3 additionally lists Pong as part of the Meta-Environment, contradicting the Section 5.3 claim that TAPD had not previously learned Pong. Since no ablation removes the distillation step from the pretraining, the gains on overlapping games cannot be attributed to the proposed distillation mechanism.

full rationale

This is an empirical continual-RL paper, so there is no equation-level derivation chain to be circular: Eq. (3), Eq. (4), and Eq. (A1) are standard RL, EWC, and distillation losses imported from Pathak et al. and Schwarz et al., and they do not encode the measured result. The self-citations (Hafez et al. [33], Zhao et al. [15], Hafez and Wermter [23]) appear only in related-work discussions and are not load-bearing; no uniqueness theorem is invoked. The Progress and Compress baseline and the ICM curiosity signal are external, cited components, so the comparison has independent content. The main circularity concern is at the benchmark-construction level: the task-agnostic phase is pretrained on two of the five evaluation games, and Section 5.1 reports Bayesian hyperparameter optimization that maximizes normalized scores across those same tasks, so part of the reported advantage is fitted rather than predicted. Unseen games (DemonAttack, AirRaid, and possibly Pong depending on the conflicting Meta-Environment descriptions) provide independent evidence, which keeps the paper from being wholly circular; the missing distillation-only ablation is a validity concern rather than circularity per se, but it leaves the causal component unidentified. Overall, the paper's central claim has real independent content, yet a nontrivial portion of its 'prediction' reduces to the pretraining/evaluation overlap.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The central claim is empirical, not derived. The main assumptions are domain assumptions inherited from prior work: curiosity via forward-model error is a useful intrinsic reward, distillation preserves exploration, Online EWC prevents forgetting, and the Meta-Environment with action-space shaping preserves task identity. The free parameters are mostly hyperparameters tuned on the evaluation tasks, which introduces a benchmark-fitting burden.

free parameters (7)

ewc-lambda = 2
Regularization strength for Online EWC in the distillation loss; tuned by Bayesian hyperparameter optimization on downstream scores.
ewc-gamma = 0.3
Forgetting factor in online Fisher information updates; tuned.
learning rate (lr) = 7e-4
A2C optimizer learning rate; tuned.
entropy-coef = 0.01
Entropy regularization coefficient in the policy loss; tuned.
num-env-steps-agnostic = 3e5
Number of environment steps in each task-agnostic exploration segment before compression; hand-selected or tuned.
num-env-steps-compress-agnostic = 3e5
Number of environment steps for each task-agnostic compression; hand-selected or tuned.
num-samples-drawn-in-task-agnostic-phase = 25
Number of task samples drawn from the Meta-Environment during the task-agnostic phase; hand-selected or tuned, controls pretraining budget.

assumptions (6)

standard math The A2C policy gradient estimator (Eq. 2) provides a valid optimization signal for both intrinsic and extrinsic rewards.
Background Section 3.2; assumed without proof, standard in RL.
domain assumption The Meta-Environment abstraction and action-space shaping preserve the identities of the Atari tasks.
Section 4.1 and Table 1; if remapping actions or reward normalization changes the task distribution, transfer results may not apply to standard Atari.
domain assumption Forward-model prediction error is a sufficient intrinsic reward for learning reusable exploration.
Section 4.3, ri_t = log(L_forward + epsilon), adopted from Pathak et al. [7] without standalone validation in this setting.
domain assumption Distilling the exploratory policy into the knowledge base preserves and augments exploratory behavior.
Section 4.2, Eq. 4; relies on prior distillation results [30] and is not ablated.
domain assumption Online EWC protects task-agnostic knowledge from catastrophic forgetting.
Appendix A, Eq. A1 and Algorithm 1; inherited from Schwarz et al. [13].
ad hoc to paper Uniform sampling of two games during the task-agnostic phase is a valid operationalization of learning without task boundaries.
Section 5.2; specific to this paper's experimental protocol, not derived from a general theory.

invented entities (1)

Meta-Environment
purpose: A conceptual container that treats all Atari games as tasks within one environment, so task switches are just shifts within the container.
Introduced in Section 4.1 as a framing abstraction; it makes no falsifiable prediction outside the paper's experimental protocol.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation." pith.science (2026). https://pith.science/paper/OH6NNPQ5

@misc{pith2026241116532,
  author       = {Pith},
  title        = {Pith review of: Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OH6NNPQ5}},
  note         = {Machine review of arXiv:2411.16532}
}

read the original abstract

Central to the development of universal learning systems is the ability to solve multiple tasks without retraining from scratch when new data arrives. This is crucial because each task requires significant training time. Addressing the problem of continual learning necessitates various methods due to the complexity of the problem space. This problem space includes: (1) addressing catastrophic forgetting to retain previously learned tasks, (2) demonstrating positive forward transfer for faster learning, (3) ensuring scalability across numerous tasks, and (4) facilitating learning without requiring task labels, even in the absence of clear task boundaries. In this paper, the Task-Agnostic Policy Distillation (TAPD) framework is introduced. This framework alleviates problems (1)-(4) by incorporating a task-agnostic phase, where an agent explores its environment without any external goal and maximizes only its intrinsic motivation. The knowledge gained during this phase is later distilled for further exploration. Therefore, the agent acts in a self-supervised manner by systematically seeking novel states. By utilizing task-agnostic distilled knowledge, the agent can solve downstream tasks more efficiently, leading to improved sample efficiency. Our code is available at the repository: https://github.com/wabbajack1/TAPD.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 30 canonical work pages

[1]

Cells 10(4), 735 (2021)

Speranza, L., Di Porzio, U., Viggiano, D., Donato, A., Volpicelli, F.: Dopamine: The neuromodulator of long-term synaptic plasticity, reward and movement control. Cells 10(4), 735 (2021)

work page 2021
[2]

Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction (2018)

work page 2018
[3]

Science 245(4918), 605–615 (1989) 30

Miller, K.D., Keller, J.B., Stryker, M.P.: Ocular dominance column development: analysis and simulation. Science 245(4918), 605–615 (1989) 30

work page 1989
[4]

Neural networks 113, 54–71 (2019)

Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural networks 113, 54–71 (2019)

work page 2019
[5]

Journal of Artificial Intelligence Research 61, 523–562 (2018)

Machado, M.C., Bellemare, M.G., Talvitie, E., Veness, J., Hausknecht, M., Bowl- ing, M.: Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research 61, 523–562 (2018)

work page 2018
[6]

arXiv preprint arXiv:2304.09355 (2023)

Shwartz-Ziv, R., LeCun, Y.: To compress or not to compress–self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 (2023)

arXiv 2023
[7]

arXiv preprint arXiv:1705.05363 (2017)

Pathak, D., Agrawal, P., Efros, A.A., Darrell, T.: Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv:1705.05363 (2017)

arXiv 2017
[8]

Advances in Neural Information Processing Systems 34, 20516–20530 (2021)

Parisi, S., Dean, V., Pathak, D., Gupta, A.: Interesting object, curious agent: Learning task-agnostic exploration. Advances in Neural Information Processing Systems 34, 20516–20530 (2021)

work page 2021

Show all 48 references

[9]

Advances in neural information processing systems 12 (1999)

Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems 12 (1999)

1999
[10]

Advances in Neural Information Processing Systems 33, 11734–11743 (2020)

Zhang, X., Ma, Y., Singla, A.: Task-agnostic exploration in reinforcement learning. Advances in Neural Information Processing Systems 33, 11734–11743 (2020)

2020
[11]

Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114(13), 3521–3526 (2017)

2017
[12]

arXiv preprint arXiv:1606.04671 (2016)

Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)

2016 arXiv
[13]

In: International Conference on Machine Learning, pp

Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., Hadsell, R.: Progress & compress: A scalable framework for contin- ual learning. In: International Conference on Machine Learning, pp. 4528–4537 (2018). PMLR

2018
[14]

arXiv preprint arXiv:1803.10123 (2018)

Zeno, C., Golan, I., Hoffer, E., Soudry, D.: Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123 (2018)

2018 arXiv
[15]

In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 31 pp

Zhao, X., Weber, C., Hafez, M.B., Wermter, S.: Impact makes a sound and sound makes an impact: Sound guides representations and explorations. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 31 pp. 2512–2518 (2022). IEEE

2022
[16]

Journal of Artifi- cial Intelligence Research 47, 253–279 (2013) https://doi.org/10.1613/jair.3912

Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artifi- cial Intelligence Research 47, 253–279 (2013) https://doi.org/10.1613/jair.3912 . arXiv:1207.4708 [cs]

2013 arXiv
[17]

In: ICLR (2016)

Rusu, A.A., Colmenarejo, S.G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., Hadsell, R.: Policy distillation. In: ICLR (2016)

2016
[18]

In: The 22nd International Conference on Artificial Intelligence and Statistics, pp

Czarnecki, W.M., Pascanu, R., Osindero, S., Jayakumar, S., Swirszcz, G., Jader- berg, M.: Distilling policy distillation. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1331–1340 (2019). PMLR

2019
[19]

Advances in Neural Information Processing Systems 34, 6920–6933 (2021)

Watkins, O., Gupta, A., Darrell, T., Abbeel, P., Andreas, J.: Teachable reinforce- ment learning via advice distillation. Advances in Neural Information Processing Systems 34, 6920–6933 (2021)

2021
[20]

arXiv preprint arXiv:1912.12630 (2019)

Sun, Y., Fazli, P.: Real-time policy distillation in deep reinforcement learning. arXiv preprint arXiv:1912.12630 (2019)

2019 arXiv
[21]

In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp

Thorne, J., Vlachos, A.: Elastic weight consolidation for better bias inocula- tion. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 957–964 (2021)

2021
[22]

In: 2017 International Joint Conference on Neural Networks (IJCNN), pp

Draelos, T.J., Miner, N.E., Lamb, C.C., Cox, J.A., Vineyard, C.M., Carlson, K.D., Severa, W.M., James, C.D., Aimone, J.B.: Neurogenesis deep learning: Extending deep networks to accommodate new classes. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 52...

2017
[23]

IEEE Transactions on Cognitive and Developmental Systems (2023)

Hafez, M.B., Wermter, S.: Continual robot learning using self-supervised task inference. IEEE Transactions on Cognitive and Developmental Systems (2023)

2023
[24]

In: International Conference on Learning Representations (2018)

Kemker, R., Kanan, C.: Fearnet: Brain-inspired model for incremental learning. In: International Conference on Learning Representations (2018)

2018
[25]

Nature communications 11(1), 4069 (2020)

Ven, G.M., Siegelmann, H.T., Tolias, A.S.: Brain-inspired replay for contin- ual learning with artificial neural networks. Nature communications 11(1), 4069 (2020)

2020
[26]

IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

Wang, L., Zhang, X., Su, H., Zhu, J.: A comprehensive survey of continual learn- ing: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

2024
[27]

Advances in neural information processing 32 systems 32 (2019)

Rao, D., Visin, F., Rusu, A., Pascanu, R., Teh, Y.W., Hadsell, R.: Continual unsupervised representation learning. Advances in neural information processing 32 systems 32 (2019)

2019
[28]

arXiv preprint arXiv:2110.06976 (2021)

Madaan, D., Yoon, J., Li, Y., Liu, Y., Hwang, S.J.: Representational continuity for unsupervised continual learning. arXiv preprint arXiv:2110.06976 (2021)

2021 arXiv
[29]

Journal of Neuroscience 35(3), 1319–1334 (2015)

Ziegler, L., Zenke, F., Kastner, D.B., Gerstner, W.: Synaptic consolidation: from synapses to behavioral modeling. Journal of Neuroscience 35(3), 1319–1334 (2015)

2015
[30]

arXiv preprint arXiv:1503.02531 (2015)

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

2015 arXiv
[31]

Advances in neural information processing systems 17 (2004)

Chentanez, N., Barto, A., Singh, S.: Intrinsically motivated reinforcement learn- ing. Advances in neural information processing systems 17 (2004)

2004
[32]

Neuron 36(2), 285–298 (2002)

Dayan, P., Balleine, B.W.: Reward, motivation, and reinforcement learning. Neuron 36(2), 285–298 (2002)

2002
[33]

In: 2017 Joint IEEE Inter- national Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp

Hafez, M.B., Weber, C., Wermter, S.: Curiosity-driven exploration enhances motor skills of continuous actor-critic learner. In: 2017 Joint IEEE Inter- national Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp. 39–46 (2017). IEEE

2017
[34]

In: Proc

Schmidhuber, J.: A possibility for implementing curiosity and boredom in model-building neural controllers. In: Proc. of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, pp. 222–227 (1991)

1991
[35]

arXiv (2013)

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with Deep Reinforcement Learning. arXiv (2013). https://doi.org/10.48550/arXiv.1312.5602 . http://arxiv.org/abs/1312.5602

2013 doi arXiv
[36]

Li, Y.: Deep Reinforcement Learning: An Overview. arXiv. arXiv:1701.07274 [cs] (2018). http://arxiv.org/abs/1701.07274

2018 arXiv
[37]

In: International Conference on Machine Learning, pp

Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937 (2016). PMLR

2016
[38]

Advances in neural information processing systems 30 (2017)

Wu, Y., Mansimov, E., Grosse, R.B., Liao, S., Ba, J.: Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. Advances in neural information processing systems 30 (2017)

2017
[39]

arXiv preprint arXiv:1506.02438 (2015)

Schulman, J., Moritz, P., Levine, S., Jordan, M.I., Abbeel, P.: High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015)

2015 arXiv
[40]

arXiv preprint arXiv:2302.00487 33 (2023)

Wang, L., Zhang, X., Su, H., Zhu, J.: A comprehensive survey of contin- ual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 33 (2023)

2023 arXiv
[41]

The Royal Society (2017)

Fox, K., Stryker, M.: Integrating Hebbian and homeostatic plasticity: introduc- tion. The Royal Society (2017)

2017
[42]

arXiv preprint arXiv:1904.07734 (2019)

Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 (2019)

2019 arXiv
[43]

In: 2020 IEEE Conference on Games (CoG), pp

Kanervisto, A., Scheller, C., Hautam¨ aki, V.: Action space shaping in deep rein- forcement learning. In: 2020 IEEE Conference on Games (CoG), pp. 479–486 (2020). IEEE

2020
[44]

Advances in neural information processing systems 32 (2019)

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems 32 (2019)

2019
[45]

GitHub (2018)

Kostrikov, I.: PyTorch Implementations of Reinforcement Learning Algorithms. GitHub (2018)

2018
[46]

Journal of Machine Learning Research 22(268), 1–8 (2021)

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22(268), 1–8 (2021)

2021
[47]

arXiv (2022)

Lucchesi, N., Carta, A., Lomonaco, V., Bacciu, D.: Avalanche RL: a Contin- ual Reinforcement Learning Library. arXiv (2022). https://doi.org/10.48550/ ARXIV.2202.13657 . https://arxiv.org/abs/2202.13657

2022 doi arXiv
[48]

IEEE transactions on pattern analysis and machine intelligence 44(10), 6715–6728 (2021) 34

Xu, J., Ma, J., Gao, X., Zhu, Z.: Adaptive progressive continual learning. IEEE transactions on pattern analysis and machine intelligence 44(10), 6715–6728 (2021) 34

2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Cells 10(4), 735 (2021)

Speranza, L., Di Porzio, U., Viggiano, D., Donato, A., Volpicelli, F.: Dopamine: The neuromodulator of long-term synaptic plasticity, reward and movement control. Cells 10(4), 735 (2021)

work page 2021

[2] [2]

Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction (2018)

work page 2018

[3] [3]

Science 245(4918), 605–615 (1989) 30

Miller, K.D., Keller, J.B., Stryker, M.P.: Ocular dominance column development: analysis and simulation. Science 245(4918), 605–615 (1989) 30

work page 1989

[4] [4]

Neural networks 113, 54–71 (2019)

Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural networks 113, 54–71 (2019)

work page 2019

[5] [5]

Journal of Artificial Intelligence Research 61, 523–562 (2018)

Machado, M.C., Bellemare, M.G., Talvitie, E., Veness, J., Hausknecht, M., Bowl- ing, M.: Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research 61, 523–562 (2018)

work page 2018

[6] [6]

arXiv preprint arXiv:2304.09355 (2023)

Shwartz-Ziv, R., LeCun, Y.: To compress or not to compress–self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 (2023)

arXiv 2023

[7] [7]

arXiv preprint arXiv:1705.05363 (2017)

Pathak, D., Agrawal, P., Efros, A.A., Darrell, T.: Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv:1705.05363 (2017)

arXiv 2017

[8] [8]

Advances in Neural Information Processing Systems 34, 20516–20530 (2021)

Parisi, S., Dean, V., Pathak, D., Gupta, A.: Interesting object, curious agent: Learning task-agnostic exploration. Advances in Neural Information Processing Systems 34, 20516–20530 (2021)

work page 2021