Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Self-Composing Policies for Scalable Continual Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A growable neural network that composes frozen past policies learns continual RL tasks without forgetting and with linear parameter growth.

desk verdict CompoNet is a genuinely new growable CRL architecture with convincing mechanistic experiments, but its headline comparative claim is undercut by an ALE success-score calibration that uses the evaluated methods themselves. read the letter →

arxiv 2506.14811 v1 pith:BQTHOEFG submitted 2025-06-04 cs.LG

Mikel Malagón , Josu Ceberio , Jose A. Lozano This is my paper
classification cs.LG
keywords continualreinforcementlearninggrowableneuralnetworkscatastrophicforgettingknowledgetransferself-composingpoliciesattentionmechanismCompoNetforward
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a continually learning reinforcement-learning agent can avoid catastrophic forgetting and interference by freezing old policy modules and letting each new module compose their outputs through attention. The proposed architecture, CompoNet, adds one module per task and grows linearly in parameters, unlike previous growable networks such as progressive networks that grow quadratically. In benchmark sequences from Meta-World, SpaceInvaders, and Freeway, CompoNet reports the highest final performance and forward transfer, and it is the only method with positive forward transfer in the Meta-World sequence. A reader should care because continual RL agents that accumulate and reuse skills across many tasks would need exactly this combination of no forgetting, knowledge transfer, and scalable growth.

What carries the argument

The self-composing policy module is the load-bearing unit. For a new task, it receives the current state representation $h_s$ and the matrix $\Phi_{k;s}$ of outputs from all previous frozen policies. An output attention head computes scaled dot-product attention over those outputs, conditioned on $h_s$, to produce a tentative output vector; an input attention head then attends to the concatenation of that tentative vector with the previous outputs, again conditioned on $h_s$, to feed contextual information to an internal feed-forward policy whose output is added to the tentative vector to form the module's final action distribution. Because each module's parameter count depends only on fixed hyperparameters ($d_{\text{enc}}$, $d_{\text{model}}$, $|A|$) and not on the number of tasks, the architecture grows linearly in parameters with the number of tasks.

What would settle it

Recompute the SpaceInvaders and Freeway success scores using only the baseline and FT-1 methods (or fixed thresholds) and re-run Table 1; if CompoNet no longer achieves the top PERF and forward transfer, the headline claim depends on the self-referential threshold.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a modular network in which every new policy module is built from two attention heads over the outputs of all frozen previous modules, plus an internal feed-forward policy, can solve new tasks in three regimes: directly reusing a matching previous policy, composing a function of previous policies, and learning from scratch with no interference. CompoNet is reported to achieve superior performance and forward transfer in all three evaluated task sequences, consistently exceeding the reference forward transfer and matching or beating ProgressiveNet, PackNet, FT-1, and FT-N, while adding a constant number of parameters per task and scaling empirically to hundreds of modules.

Load-bearing premise

The Atari success thresholds are computed as 90% of the average final episodic return across the eight evaluated methods, including CompoNet itself, so the headline PERF metric measures relative standing within the evaluated set rather than an external task-mastery standard.

Editorial extensions

If this is right

  • Continual RL agents built this way can accumulate an unbounded number of task skills without retraining old modules, so forgetting and interference are structurally prevented rather than managed.
  • Memory cost per task is constant, so sequences of hundreds or thousands of tasks remain feasible on a single GPU, where progressive-network-style growth would become prohibitive.
  • Positive forward transfer in the Meta-World sequence, where the reference transfer is negative, suggests CompoNet can extract usable knowledge even when most pairwise task transfers are harmful.
  • Because the module outputs are probability distributions over a shared action space, the same architecture applies to both continuous-control (SAC) and discrete visual (PPO) domains with only encoder changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two attention heads can be read as a differentiable, task-conditional routing mechanism, suggesting that CompoNet could be extended to hierarchical or options-style RL where high-level modules compose low-level skills, though the paper does not test this.
  • The Atari success scores are defined as 90% of the average final return across all eight evaluated methods, so the PERF numbers in Table 1 are relative to the method set itself; an external or baseline-only threshold could reorder the rankings.
  • Theoretical inference cost is $O(n^2)$ in the number of modules, so even with linear memory, never-ending learning would eventually hit an inference bottleneck unless the attention computation is pruned or approximated; the paper reports empirical scaling only up to 300 tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces CompoNet, a growable and modular neural architecture for continual reinforcement learning. When a new task arrives, the previous task's policy module is frozen and a new module is added; each new module can attend to the outputs of all previous modules through an output attention head and an input attention head, and its internal policy produces a residual adjustment. The authors argue that this design avoids catastrophic forgetting and interference, grows linearly in parameters with the number of tasks, and achieves positive forward transfer by composing frozen previous policies. The method is evaluated on a 20-task Meta-World sequence (SAC) and on the playing modes of SpaceInvaders and Freeway (PPO), with comparisons against a from-scratch baseline, FT-1, FT-N, ProgressiveNet, and PackNet. The paper also includes architectural validation experiments studying attention weights, a shifted-action-label ablation, and scalability up to hundreds of modules.

Significance. The architectural idea is well motivated and the paper gives concrete evidence for the proposed mechanism: the attention-weight analyses in Figure 4 and the shifted-action-label experiment in Appendix G.2 convincingly show that the module can select and compose a relevant previous policy, and can fall back to learning from scratch when no previous module is informative. The linear parameter-growth claim is backed by a simple counting argument in Appendix B and by direct measurements. The code is linked. If the empirical superiority claim could be placed on a firmer footing, this would be a useful contribution to continual RL, as it offers a rare combination of strict forgetting avoidance (frozen modules), positive transfer, and modest memory growth. As it stands, however, the headline comparative claim is supported by only one of the three benchmarks with a clear margin, and the success thresholds used in the other two benchmarks are computed from the very methods being ranked, which weakens the 'superior in all three sequences' statement.

major comments (3)
  1. [Appendix D.4 and Table D.1] The success scores for the SpaceInvaders and Freeway sequences are defined as 90% of the average final episodic return over all evaluated methods, which includes CompoNet itself. Since the final performance P(T) and the forward-transfer metric in Eq. (2) are nonlinear functions of these thresholds, a change in any one method's final return changes every method's PERF and forward-transfer values. Consequently, the PERF and FWD columns in Table 1 for these two sequences measure standing within the evaluated pool rather than task mastery, and the caption's claim that CompoNet is 'superior' in all three sequences is not anchored to an external standard. The appendix also states that thresholds are computed over 'all 8 methods', while Table 1 lists only six methods; the calibration set is therefore incompletely specified. Please either anchor the thresholds to a method-independent standard (e.g., a fixed absolute episodic return or a human baseline) or, failing that, provide a leave-one-out sensitivity analysis showing that the Table 1 ranking is stable when each method is excluded from the threshold computation.
  2. [Table 1 and caption] The caption states that CompoNet 'achieves superior performance and forward transfer in all three sequences', but the table itself shows ties within one standard deviation in two of the three sequences. In Meta-World, CompoNet's PERF is 0.42±0.49 versus ProgressiveNet's 0.41±0.49; in SpaceInvaders, CompoNet's PERF is 0.99±0.01, identical to FT-N's 0.99±0.01, and the forward-transfer values 0.74±0.22 versus 0.73±0.25 are well within noise. Only the Freeway sequence shows a clear margin (PERF 0.94±0.06 versus 0.81±0.01). Please either restrict the claim to the Freeway sequence and to forward-transfer-oriented conclusions elsewhere, or add statistical tests (paired bootstrap or Wilcoxon over the 10 seeds) that justify the word 'superior' for the remaining comparisons.
  3. [Section 5.3 and Appendix F.1] The Meta-World results are presented as supporting 'superior performance', but the PERF values for CompoNet and ProgressiveNet are statistically indistinguishable (0.42±0.49 versus 0.41±0.49), and in tasks 4, 7, 14, and 17 no method obtains a consistent positive success rate, as acknowledged in Appendix F.1. Since these four tasks contribute to the average P(T) and to the forward-transfer computation, the Meta-World comparison should be discussed more cautiously; the current text in Section 5.3 overstates the evidence for 'outperforming' ProgressiveNet in that sequence.
minor comments (4)
  1. [Table D.1(b) and Section 5.2] The main text says the Freeway sequence consists of the '7 playing modes' of ALE/Freeway-v5, and Appendix D.3 lists Mode 0 through Mode 6, but Table D.1(b) reports success scores for TASK 0 through TASK 7 (eight columns). Please reconcile this discrepancy.
  2. [Appendix D.4] The sentence 'we compute the success score for each task as 90% of the average final episodic return of all 8 methods' should be corrected to specify exactly which methods and how many seeds were used, since Table 1 lists six methods and the transfer matrices in Appendix D.5 use three seeds per method.
  3. [Figure 3] The left panel of Figure 3 combines two y-axes (inference time and memory) without clear separation of the curves; consider splitting into two panels or using distinct line styles with an explicit legend for each axis.
  4. [Equation (2)] The denominator 1−AUC^b_i in the forward-transfer formula can be zero if the baseline solves the task immediately; a short note on how this case is handled (or that it does not occur in the reported experiments) would improve precision.

Circularity Check

1 steps flagged · score 3.0 of 10

The ALE success thresholds are computed from the same evaluated methods, including CompoNet, so the headline 'superior in all three sequences' is partly self-referential for SpaceInvaders and Freeway.

  1. self definitional [Appendix D.4 (Success Scores for the SpaceInvaders and Freeway Sequences), Eq. (2) in Section 5.1, and Table 1 caption]
    "we define success as 1 if the episodic return is greater or equal to the success score, otherwise zero. ... we compute the success score for each task as 90% of the average final episodic return (i.e., the performance once trained) of all 8 methods (10 random seeds per method)."

    The success score that defines p_i(t) is computed from the final episodic returns of all evaluated methods, including CompoNet. Eq. (2) then turns p_i(t) into AUC_i and FTr_i, and P(T) averages p_i(t); hence CompoNet's own final returns enter the threshold against which its success is judged. For SpaceInvaders and Freeway, PERF and forward transfer therefore partly measure standing within the evaluated pool rather than task mastery, so the Table 1 claim that CompoNet 'achieves superior performance and forward transfer in all three sequences' is self-referential for two of those sequences. The calibration set is also incompletely specified: the appendix says 'all 8 methods' while Section 5.2 and Table 1 list six methods.

full rationale

The core CompoNet derivation is not circular. The architecture is defined by self-composing policy modules that attend to frozen previous policies; no parameter is fitted to the target outcome, and the linear parameter-growth claim is derived from the constant per-module parameter count in Appendix B. The forward-transfer metric in Eq. (2) is computed relative to an independently trained from-scratch baseline, which is external to CompoNet, so the Meta-World evaluation and the relative ranking mechanism are meaningful. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in through author citation. The one definitional issue is the ALE success threshold in Appendix D.4: success is declared when episodic return reaches 90% of the average final return of all evaluated methods, including CompoNet itself. Because P(T), AUC, and FTr all inherit this threshold, the SpaceInvaders and Freeway performance claims are not anchored to a method-independent standard. This is a genuine, but partial, self-reference in the evaluation chain; it does not force a particular winner and does not invalidate the architecture's internal behavior or scalability analysis. Score 3 reflects one metric-level circularity while the central derivation remains independent.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The architecture introduces no fitted physical constants or invented entities. The main ledger item is the self-referential ALE success threshold, which is a fitted calibration of the metric. The listed axioms are the task-level assumptions that define the problem setting; they are standard for the CRL literature and are explicitly acknowledged in the paper.

free parameters (1)
  • Per-task success thresholds for SpaceInvaders and Freeway = 90% of average final episodic return across all 8 methods; e.g., SpaceInvaders task 0: 340.94, Freeway task 0: 16.65
    Used to binarize episodic returns into the success measure underlying the PERF metric (Table 1). Computed from the final returns of the evaluated methods, including CompoNet itself, so the metric is calibrated on the evaluated set rather than an external standard.
assumptions (5)
  • domain assumption Task boundaries and task identifiers are known to the agent.
    Stated in Section 3, Assumptions; the paper acknowledges in Conclusions that removing this is future work.
  • domain assumption The action space remains constant across tasks (soft assumption).
    Section 3, Assumptions; needed so that outputs of previous modules share the same dimension and can be composed by attention.
  • domain assumption State spaces across tasks are similar, S(i) ≈ S(j).
    Section 3, Assumptions; if state spaces differ substantially, the same encoders and policy inputs would not transfer.
  • domain assumption The output distributions of previous policies provide a sufficient interface for composing knowledge.
    Central design assumption in Section 4.2. CompoNet does not access previous modules' hidden representations, only their action probability outputs; this is what keeps parameter growth linear. If useful knowledge lives only in hidden states, the architecture cannot exploit it.
  • domain assumption Restarting the critic at each task does not distort the comparisons.
    Appendix E.2: 'the CRL methods are only applied to the network of the actor while the critic is restarted at the beginning of each task.' This follows common practice but means value-function transfer is not measured or exploited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Composing Policies for Scalable Continual Reinforcement Learning." pith.science (2026). https://pith.science/paper/BQTHOEFG

@misc{pith2026250614811,
  author       = {Pith},
  title        = {Pith review of: Self-Composing Policies for Scalable Continual Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQTHOEFG}},
  note         = {Machine review of arXiv:2506.14811}
}
read the original abstract

This work introduces a growable and modular neural network architecture that naturally avoids catastrophic forgetting and interference in continual reinforcement learning. The structure of each module allows the selective combination of previous policies along with its internal policy, accelerating the learning process on the current task. Unlike previous growing neural network approaches, we show that the number of parameters of the proposed approach grows linearly with respect to the number of tasks, and does not sacrifice plasticity to scale. Experiments conducted in benchmark continuous control and visual problems reveal that the proposed approach achieves greater knowledge transfer and performance than alternative methods.

Figures

Figures reproduced from arXiv: 2506.14811 by the authors.

Figure 1
Figure 1. Evolution of the Self-Composing Policies Network architecture (CompoNet) across multiple tasks. Trainable self-composing policy modules are represented by light yellow blocks, and frozen modules are denoted by dark blue blocks. The initial task involves training a single policy module from scratch. Following the completion of each task, the trained module is frozen, and a new trainable module is introduced for the s… view at source ↗
Figure 2
Figure 2. Diagram of the self-composing policy module. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Empirical computational cost of inference (left) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Empirical results on the fulfillment of objectives (i) and (iii) from Section [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

    cs.LG 2025-06 reject novelty 5.0 of 10

    A curriculum that aligns temporal-difference error distributions across increasing adversarial perturbations is claimed to make UAV policies robust to unseen GNSS spoofing attacks, with a generalization certificate.

Reference graph

Works this paper leans on

44 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z

    Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the Atari human benchmark. In Proceed- ings of the 2020 International Conference on Machine Learning (ICML), pp. 507–517,

    work page 2020

  2. [2]

    Therefore, the computational cost of the internal policy is constant and independent of the number of modules (i.e., number of tasks), Tint(n) = O(1).9 Total Complexity

    · d2 model| {z } Intermediate layers + dmodel · a| {z } Last layer (6) Note that none of the terms of Tint(n) depends on n (the number of policy modules). Therefore, the computational cost of the internal policy is constant and independent of the number of modules (i.e., number of tasks), Tint(n) = O(1).9 Total Complexity. Considering the computational co...

    work page 2016

  3. [4]

    9For the sake of simplicity, this definition of the internal policy ignores possible activation and normalization layers

    Results were measured in a machine with an AMD EPIC 7252 CPU and an NVIDIA A5000 GPU. 9For the sake of simplicity, this definition of the internal policy ignores possible activation and normalization layers. 15 Self-Composing Policies for Scalable Continual Reinforcement Learning D. Environments and Tasks The experimentation is conducted across three task...

    work page 2020

  4. [5]

    For this purpose, we have designed an experimental setting where CompoNet should employ the input attention head to gather information to solve the current task, while this information can not be directly used to solve the task via imitation using the output attention head. Specifically, we have trained a CompoNet agent in the fifth task of the SpaceInvad...

    work page 2021

  5. [6]

    Same as the previous mode but with heavier traffic. D.4. Success Scores for the SpaceInvaders and Freeway Sequences As described in Section 5.1, the CRL metrics we use in this work are defined over the concept of performance, which is in turn defined in terms of the success rate. The success rate is based on the binary metric that indicates whether the ta...

    work page 2020

  6. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 2021 International Conference on Learning Representations (ICLR),

    work page 2021

  7. [9]

    Finally, note that the diagonal of the matrix has no especially positive transfer values

    Contrarily, looking at the columns of the matrix, we see that when the second task is 2, 5, or 8, the transfer to these tasks from practically any other task is negative, implying that other tasks provide almost no information to solve the mentioned ones. Finally, note that the diagonal of the matrix has no especially positive transfer values. Even though...

    work page 2021

  8. [11]

    DARLA: Improving zero-shot transfer in reinforcement learning

    Higgins, I., Pal, A., Rusu, A., Matthey, L., Burgess, C., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A. DARLA: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 2017 International Con- ference on Machine Learning (ICML) , pp. 1480–1490,

    work page 2017

Show all 44 references
  1. [13]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the 2017 National Academy of Sciences, pp. 3521–352...

    2017

  2. [15]

    and Lazebnik, S

    Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the 2018 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 7765–7773,

    2018

  3. [16]

    and Cohen, N

    McCloskey, M. and Cohen, N. J. Catastrophic interference in connectionist networks: The sequential learning prob- lem. In Proceedings of 1989 Psychology of Learning and Motivation, volume 24, pp. 109–165. Elsevier,

    1989

  4. [17]

    A., van Seijen, H., and EATON, E

    Mendez, J. A., van Seijen, H., and EATON, E. Modular lifelong reinforcement learning via neural composition. In Proceedings of the 2022 International Conference on Learning Representations (ICLR),

    2022

  5. [18]

    DINOv2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. DINOv2: Learning robust visual features without supervision. ArXiv preprint arXiv:2304.07193,

    arXiv

  6. [19]

    Routing net- works: Adaptive selection of non-linear functions for multi-task learning

    Rosenbaum, C., Klinger, T., and Riemer, M. Routing net- works: Adaptive selection of non-linear functions for multi-task learning. In Proceedings of the 2018 Interna- tional Conference on Learning Representations (ICLR),

    2018

  7. [20]

    Rout- ing networks and the challenges of modular and compo- sitional computation

    Rosenbaum, C., Cases, I., Riemer, M., and Klinger, T. Rout- ing networks and the challenges of modular and compo- sitional computation. ArXiv preprint arXiv:1904.12774,

    1904 arXiv

  8. [21]

    A., Colmenarejo, S

    Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V ., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:1511.06295,

    arXiv

  9. [24]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv preprint arXiv:1707.06347,

    arXiv

  10. [25]

    V ., Montone, G., and O’Regan, J

    Terekhov, A. V ., Montone, G., and O’Regan, J. K. Knowl- edge transfer in deep block-modular neural networks. In Proceedings of the 2015 International Conference on Biomimetic and Biohybrid Systems , pp. 268–279. Springer,

    2015

  11. [27]

    Wang, R., Lehman, J., Clune, J., and Stanley, K. O. Paired open-ended trailblazer (POET): Endlessly generating in- creasingly complex and diverse learning environments and their solutions. ArXiv preprint arXiv:1901.01753,

    1901 arXiv

  12. [28]

    Continual World: A robotic benchmark for continual reinforcement learning

    Wolczyk, M., Zajac, M., Pascanu, R., Kucinski, L., and Milo´s, P. Continual World: A robotic benchmark for continual reinforcement learning. In Proceedings of the 2021 Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 28496–28510,

    2021

  13. [29]

    Disentangling transfer in continual reinforce- ment learning

    Wolczyk, M., Zajkac, M., Pascanu, R., Kucinski, L., and Milo´s, P. Disentangling transfer in continual reinforce- ment learning. In Proceedings of the 2022 Advances in Neural Information Processing Systems (NeurIPS), vol- ume 35, pp. 6304–6317,

    2022

  14. [30]

    Supermasks in superposition

    Wortsman, M., Ramanujan, V ., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., and Farhadi, A. Supermasks in superposition. In Proceedings of the 2020 Advances in Neural Information Processing Systems (NeurIPS), vol- ume 33, pp. 15173–15184,

    2020

  15. [31]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. InProceedings of the 2023 International Conference on Machine Learning (ICML), pp. 38087–38099,

    2023

  16. [32]

    Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learn- ing with dynamically expandable networks. In Proceed- ings of the 2018 International Conference on Learning Representations (ICLR),

    2018

  17. [33]

    Gradient surgery for multi-task learning

    Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In Proceedings of the 2020 Advances in Neural Infor- mation Processing Systems (NeurIPS) , volume 33, pp. 5824–5836, 2020a. Yu, T., Quillen, D., He, Z., Julian, R., Ha...

    2020

  18. [34]

    11 Self-Composing Policies for Scalable Continual Reinforcement Learning A. Vision Foundation Models for Visual Control Tasks in RL In the context of Section 4.1, where different state encoding strategies for CompoNet are discussed, in this appendix we show preliminary results...

    2023

  19. [37]

    (a) Meta-World (b) SpaceInvaders (c) Freeway Figure D.1: An example of a frame for each of the considered task sequences. D.1. Meta-World This sequence is composed of tasks from Meta-world, an open-source suite that includes a variety of robotic arm tasks proposed in Yu et al....

    2020

  20. [38]

    Trucks are longer vehicles, and thus, more difficult to avoid

    Traffic is heavier and the speed of the vehicles increases, the upper lane closest to the center has trucks. Trucks are longer vehicles, and thus, more difficult to avoid. 10See the following issue for more details: https://github.com/Farama-Foundation/Metaworld/issues/392. 11...

    2023

  21. [41]

    Figure D.2c provides the FTr matrix of the last sequence, Freeway

    We believe that this might be caused by the stochasticity introduced when re-initializing the last layer of the actor and the critic network every time the task changes. Figure D.2c provides the FTr matrix of the last sequence, Freeway. Likewise SpaceInvaders, we see that, in ...

    2022

  22. [42]

    All methods share the same common hyperparameters in every task sequence

    the CRL methods are only applied to the network of the actor while the critic is restarted at the beginning of each task. All methods share the same common hyperparameters in every task sequence. In the Meta-World sequence, where SAC is used to optimize all methods, the comple...

    2020

  23. [512]

    The rest of the hyperparameters of the encoder network are the ones provided in Table E.2. 20 Self-Composing Policies for Scalable Continual Reinforcement Learning Table E.2: Hyperparameters shared by all methods in the SpaceInvaders and Freeway task sequences under the PPO al...

    2021

  24. [1952]

    Conflict- averse gradient descent for multi-task learning

    Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. In Pro- ceedings of the 2021 Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 18878– 18890,

    2021

  25. [1999]

    Building a subspace of policies for scalable continual learning

    Gaya, J.-B., Doan, T., Caccia, L., Soulier, L., Denoyer, L., and Raileanu, R. Building a subspace of policies for scalable continual learning. In Proceedings of the 2023 International Conference on Learning Representations (ICLR),

    2023

  26. [2009]

    D., Juraf- sky, D., et al

    Cases, I., Rosenbaum, C., Riemer, M., Geiger, A., Klinger, T., Tamkin, A., Li, O., Agarwal, S., Greene, J. D., Juraf- sky, D., et al. Recursive routing networks: Learning to compose modules for language understanding. In Pro- ceedings of the 2019 Conference of the North Americ...

    2019

  27. [2013]

    Curriculum learning

    Bengio, Y ., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 2009 Inter- national Conference on Machine Learning (ICML), pp. 41–48,

    2009

  28. [2015]

    A., Rabinowitz, N

    Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. ArXiv preprint arXiv:1606.04671,

    arXiv

  29. [2016]

    A., Veˇcer´ık, M., Roth¨orl, T., Heess, N., Pascanu, R., and Hadsell, R

    Rusu, A. A., Veˇcer´ık, M., Roth¨orl, T., Heess, N., Pascanu, R., and Hadsell, R. Sim-to-real robot learning from pixels with progressive nets. In Proceedings of the 2017 Confer- ence on Robot Learning (CoRL), pp. 262–270,

    2017

  30. [2017]

    Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor

    9 Self-Composing Policies for Scalable Continual Reinforcement Learning Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In Proceedings of the 2018 International Conference o...

    2018

  31. [2018]

    Don’t forget, there is more than forgetting: new metrics for continual learning

    D´ıaz-Rodr´ıguez, N., Lomonaco, V ., Filliat, D., and Mal- toni, D. Don’t forget, there is more than forgetting: new metrics for continual learning. ArXiv preprint arXiv:1810.13166,

    arXiv

  32. [2019]

    W., Heess, N., Osindero, S., and Pascanu, R

    Czarnecki, W., Jayakumar, S., Jaderberg, M., Hasenclever, L., Teh, Y . W., Heess, N., Osindero, S., and Pascanu, R. Mix & match agent curricula for reinforcement learning. In Proceedings of the 2018 International Conference on Machine Learning (ICML), pp. 1087–1095,

    2018

  33. [2020]

    Bauer, J., Baumli, K., Behbahani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., Dasagi, V ., Gonzalez, L., Gregor, K., Hughes, E., Kashem, S., Loks-Thompson, M., Openshaw, H., Parker- Holder, J., Pathak, S., Perez-Nieves, N., Rakicevic, N., Rockt...

    2023

  34. [2021]

    N., Kaiser,Ł., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser,Ł., and Polosukhin, I. Attention is all you need. Proceedings of the 2017 Advances in Neural Information Processing Systems (NeurIPS) , 30,

    2017

  35. [2022]

    Compacting, picking and growing for unforgetting continual learning

    Hung, C.-Y ., Tu, C.-H., Wu, C.-E., Chen, C.-H., Chan, Y .- M., and Chen, C.-S. Compacting, picking and growing for unforgetting continual learning. In Proceedings of the 2019 Advances in Neural Information Processing Systems (NeurIPS), volume 32,

    2019

  36. [2023]

    G., Menick, J., Munos, R., and Kavukcuoglu, K

    Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K. Automated curriculum learning for neural networks. In Proceedings of the 2017 Inter- national Conference on Machine Learning (ICML), pp. 1311–1320,

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.