REVIEW 4 major objections 5 minor 17 references
Continual Task Learning through Adaptive Policy Self-Composition
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read By composing previously learned policies through semantic attention, CompoFormer shows that continual offline reinforcement learning can avoid catastrophic forgetting and reuse old skills when tasks repeat.
desk verdict CompoFormer is a useful benchmark plus architecture for continual offline RL, but the long-sequence advantage is repeat retrieval rather than compositional transfer, and the headline overstates the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the self-composing policy module: a scaled dot-product attention block (Eq. 5) whose query is the sentence embedding of the new task's text description, whose keys are the embeddings of all earlier task descriptions, and whose values are the output features of the earlier task-specific policy sub-networks. A brief warm-up on the current task's offline data trains the query and key projections, and the resulting attention scores decide the grow/no-grow switch: if the composed policy clears a performance threshold (eta = 0.8) it is used as is, otherwise new parameters are added, LoRA low-rank matrices in the Grow variant and binary neuron masks in the Prune variant, and the new sub-network's output is concatenated with the attended prior outputs before a final MLP produces actions. This module is where semantic text and learned behavior meet, and it is what turns the model into a cascading stack of policies that grows only when needed.
What would settle it
Compare the learned attention weights to an independent transfer measurement: train a policy on each task, fine-tune it briefly on every other task, build a pairwise transfer matrix from the resulting scores, and correlate it with the attention weights CompoFormer assigns between the same task pairs. If semantically similar pairs with poor measured transfer receive high attention, or if the two matrices do not correlate, then the semantic composition step is not the mechanism producing the reported gains.
Extended reading notes
Core claim
On this paper's terms, the discovery is that in continual offline reinforcement learning the effective unit of transfer is the composed policy, selected by semantic attention, rather than shared hidden representations or replayed data. Given a new task, CompoFormer encodes the task's textual description with a frozen Sentence-BERT model, treats that embedding as the query to an attention module whose keys are the embeddings of all previous task descriptions and whose values are the output features of the previous task policies (Eq. 5), and after a short warm-up on the new task's offline data either accepts the composed policy as sufficient or grows the model. Growth comes in two variants: CompoFormer-Grow adds LoRA low-rank adaptation matrices, while CompoFormer-Prune carves out binary masks of previously idle parameters; in both cases the new sub-network's output is concatenated with the attended prior outputs and projected to actions. Because old parameters are frozen, reported forgetting is essentially zero, and because the attention is trained on data from the current task, related skills such as two tasks that both push a puck receive high weight while unrelated tasks are down-weighted. On the new Offline Continual World benchmark the Prune variant achieves average performance 0.69 on the ten-task sequence and 0.73 on the repeated twenty-task sequence, ahead of the structure-based baselines it is compared with and remaining below the multi-task upper bound.
Load-bearing premise
Everything rests on one premise: the text descriptions of the tasks, once embedded and combined by attention, actually point to the older policies whose behavior helps the new task, and the short warm-up on the new task's data reliably reveals when the composed policy is already good enough.
Editorial extensions
If this is right
- Forgetting stays near zero without storing any past experience: freezing each task's sub-network and composing old policies removes the need for rehearsal buffers, which the paper shows hurt in offline RL.
- Revisited tasks become a strength: on OCW20, where the same ten tasks appear twice, CompoFormer attends back to the first occurrence and posts its best average performance (Prune: 0.73), whereas fixed-capacity structure baselines decline.
- The advantage widens as sequences lengthen, which is precisely the regime where regularization and replay methods forget most and where parameter-isolation kits run out of free capacity.
- Plasticity does not have to be traded away for stability: the attended prior knowledge makes new tasks learn faster, as the paper's learning curves show faster adaptation than all baselines.
- The reported gains are specific: forward transfer to tasks never seen remains small for every method including CompoFormer, so the mechanism improves stability and adaptation speed, not the ability to anticipate genuinely new tasks.
Reading between the lines
- Because the attention weights are trained on the current task's data and visibly concentrate on related skills, they double as a learned measure of task relatedness; a natural test the paper leaves open is using them to order the task curriculum, since the paper only evaluates shuffled orders, not attention-guided ones.
- The paper reports inference cost that grows as the square of the number of composed policies and names computational cost as the remaining bottleneck; a direct extension is sparsifying the attended set of prior policies so composition stays roughly constant-cost as the task count grows.
- The design separates the question of which skills to reuse (semantic attention plus threshold) from how to add new capacity (LoRA or masks), so the same gate could be carried onto other base sequence models or other descriptors of tasks, such as demonstrations or reward functions, without changing the composition logic.
- The Offline Continual World benchmark itself is a reusable testbed: it could be used to check whether any generative-replay or rehearsal method can close the gap to CompoFormer, or whether the distribution-shift handicap the paper describes is intrinsic to replay in offline RL.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies continual offline reinforcement learning (CORL) with a sequence of Meta-World manipulation tasks. It introduces the Offline Continual World (OCW) benchmark, consisting of OCW10 and OCW20, where OCW20 is defined as the OCW10 sequence repeated twice (Appendix A, Table 3). The proposed method, CompoFormer, is a Decision Transformer-based modular architecture with two variants: CompoFormer-Grow, which adds LoRA parameters for each new task, and CompoFormer-Prune, which uses binary masks. For each new task, a frozen Sentence-BERT encoder embeds the task description; a trainable attention module scores previous task embeddings and composes the output features of prior policies. If the composed policy exceeds a predefined threshold after a warmup phase, no new sub-network is added; otherwise, new parameters are introduced. The paper compares CompoFormer with regularization-, structure-, and rehearsal-based continual learning baselines on OCW10 and OCW20 and reports average performance, forgetting, and forward transfer, together with ablations and task-order robustness experiments. The central claim is that CompoFormer consistently outperforms all baselines, particularly on longer task sequences, achieving a better stability-plasticity trade-off.
Significance. If the claimed results held, the paper would make a useful contribution: the OCW benchmark provides a common evaluation suite for continual offline RL, and the idea of using task-description embeddings to select and compose prior policies is a reasonable and testable alternative to fixed parameter sharing. The paper is also commendable for releasing code and for reporting detailed hyperparameters and ablations. However, the evidence as presented does not establish the headline claim. The strongest quantitative result is on OCW20, but OCW20 consists of exact task repeats, so the attention mechanism can retrieve an already stored identical policy by matching the task description; this demonstrates task identification and reuse rather than compositional transfer to genuinely new tasks. On the only non-repeated benchmark, OCW10, CompoFormer-Prune's margin over PackNet is within one standard deviation, CompoFormer-Grow is numerically worse than PackNet, and no significance tests are reported.
major comments (4)
- [Section 5.2, Table 1] The statement that "CompoFormer consistently outperforms all other methods" is contradicted by Table 1: CompoFormer-Grow reaches 0.60 plus or minus 0.06 average performance on OCW10, while PackNet reaches 0.64 plus or minus 0.06. CompoFormer-Prune's margin (0.69 plus or minus 0.01 versus 0.64 plus or minus 0.06) is within one standard deviation of the baseline. Since only three seeds are reported and no significance tests are provided, the consistency claim is not supported for OCW10.
- [Section 5.1, Appendix A, Table 3; Section 5.2, Table 1; Appendix F] OCW20 is defined as the OCW10 sequence repeated twice, and Appendix F confirms that for the second half of OCW20 the model assigns highest attention to the corresponding task from the first half. The large OCW20 advantage (CompoFormer-Prune 0.73 plus or minus 0.04 versus PackNet 0.57 plus or minus 0.04) is therefore better interpreted as retrieval of an identical previously stored policy than as compositional transfer to new tasks. On OCW10, the only benchmark without exact task repeats, the advantage over PackNet is not significant, and CompoFormer-Grow trails PackNet. The paper's emphasis on "particularly in longer task sequences" accordingly needs support from a longer sequence of distinct tasks, or at minimum an analysis that separates reuse from transfer.
- [Algorithm 1 versus Section 4.2] Algorithm 1 assigns a new head(k) for every task at line 4 and updates the parameters of head(k) together with the attention parameters {W^Q, W^K} during the warmup phase at lines 10-12, even when flag is set to False and no LoRA or pruning parameters are added. This contradicts the statement in Section 4.2 that scenario (i) "avoids the need for additional learnable parameters." The parameter accounting and the description of the no-growth case should be corrected.
- [Section 5.3, Table 2] The text claims that CompoFormer "consistently outperforms LoRA and PackNet" across task orders, but the table shows CompoFormer-Grow's average performance (0.52) is below PackNet's (0.65), and CompoFormer-Grow is below PackNet in every individual order (0.60 versus 0.64, 0.54 versus 0.67, 0.43 versus 0.65, 0.51 versus 0.65). Only CompoFormer-Prune supports the claim. The claim should either be restricted to the Prune variant or the Grow results need additional evidence and explanation.
minor comments (5)
- [Figure 4b] The ablation figure reports "averaged over three random seeds" but shows no error bars or standard deviations; adding them or a corresponding table would make the comparison more informative.
- [Algorithm 1, line 14] The condition "performance is greater than or equal to eta" is not accompanied by a precise evaluation protocol; the manuscript should state whether this is success rate on the offline validation data, how many evaluation episodes are used, and whether the evaluation is performed with the composed policy alone.
- [Equation (5), Section 4.2] The dimensions of the value matrix V, which is the concatenation of policy output features Phi^(i), and their alignment with the key embeddings are not specified; a sentence clarifying the shapes would remove ambiguity.
- [Appendix D] The notation is confusing: Equation (17) is written as T(k), but the text then says this is the complexity of a single module and that the total complexity is k times O(k); please clarify whether T(k) denotes single-module or total inference time.
- [Abstract and Introduction] The strong wording "consistently outperforms all other methods" appears in several places before the detailed results are introduced; the wording should be calibrated to match the actual comparative results in Tables 1 and 2, for example by distinguishing the Grow and Prune variants.
Circularity Check
No significant circularity: the main claim is an empirical benchmark comparison, and no prediction is constructed from its own fitted values.
full rationale
The paper makes no derivation claim that reduces to its inputs. CompoFormer's attention weights WQ and WK are trained during a warmup phase and then evaluated; the final reported numbers are success rates over Meta-World evaluation, not a reconstruction of the warmup objective. The performance threshold eta is a hyperparameter, not a quantity fitted to the reported metrics. The S-BERT embeddings are a frozen external pretrained model; task descriptions enter as inputs, and the paper does not define 'transfer' as equivalent to description similarity. The OCW20 benchmark repeats the OCW10 sequence, so the large OCW20 advantage may partly reflect exact task retrieval rather than compositional generalization; this is a benchmark-validity concern, not circularity, because no equation or fitted constant forces the reported gain. The LoRA baseline is attributed to Huang et al. (2024), which shares authors with the present paper, but that citation supports a baseline implementation and is not load-bearing for the central stability-plasticity claim. One internal inconsistency is noted but is not circular: Section 4.2 says scenario (i) avoids additional learnable parameters, while Algorithm 1 always assigns and warms up head(k) before the grow/no-grow decision. This affects parameter-count claims, not the logical or statistical reduction of the main result.
Assumptions & free parameters
free parameters (3)
- performance threshold eta =
0.8
- warmup budget Iwp =
unreported
- LoRA rank r =
unreported
assumptions (4)
- domain assumption Decision Transformer's supervised sequence-modeling objective (Eq. 2) is a valid offline RL policy learning method.
- domain assumption Task descriptions embedded by S-BERT encode transferable semantic structure between manipulation tasks.
- domain assumption Action spaces are constant and state spaces similar across tasks (Section 3).
- domain assumption Offline datasets collected from SAC replay buffers during training toward convergence are sufficient for learning each task.
Cite this review
Pith. "Pith review of Continual Task Learning through Adaptive Policy Self-Composition." pith.science (2026). https://pith.science/paper/C5R3VTGX
@misc{pith2026241111364,
author = {Pith},
title = {Pith review of: Continual Task Learning through Adaptive Policy Self-Composition},
year = {2026},
howpublished = {\url{https://pith.science/paper/C5R3VTGX}},
note = {Machine review of arXiv:2411.11364}
}
read the original abstract
Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[6]
Jifeng Hu, Li Shen, Sili Huang, Zhejian Yang, Hechang Chen, Lichao Sun, Yi Chang, and Dacheng Tao. Continual diffuser (cod): Mastering continual offline reinforcement learning with experience rehearsal. arXiv preprint arXiv:2409.02512, 2024a. Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint arXiv:2303.03747, 202...
-
[8]
Merging decision transformers: Weight averaging for form- ing multi-task policies
Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for form- ing multi-task policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12942–12948. IEEE,
work page 2024
-
[9]
Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,
arXiv 2005
-
[11]
Sentence-bert: Sentence embeddings using siamese bert-networks
N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084,
arXiv 1908
-
[12]
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671,
-
[13]
t-dgr: A trajectory-based deep generative replay method for continual learning in decision making
13 William Yue, Bo Liu, and Peter Stone. t-dgr: A trajectory-based deep generative replay method for continual learning in decision making. arXiv preprint arXiv:2401.02576,
-
[14]
Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning
Yuhang Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, and Yanfeng Wang. Balanced destruction- reconstruction dynamics for memory-replay class incremental learning. arXiv preprint arXiv:2308.01698,
-
[15]
policy in isolation for each task from scratch until convergence. Once the policy converges, we collect 1 million transitions from the SAC replay buffer for each task, comprising samples observed during training as the policy approaches optimal performance (Hu et al., 2024c; He et al., 2024). In the ablation study described in Section 5.3, we use differen...
work page 2022
Show all 17 references
-
[16]
builds on Bayesian neural networks by maintaining a factorized Gaussian distribution over network parameters and applying variational inference to approximate the Bayes update. The training objective includes an additional term: λDKL(θ ∥ θ(k−1)), (12) where DKL is the Kullback...
2024
-
[17]
In contrast, regularization-based and rehearsal-based methods introduce additional loss terms to mitigate catastrophic forgetting
As shown in Figure 8, the MT method, which has access to data from all tasks throughout the learning process, achieves high performance on every task, comparable to the single-task method. In contrast, regularization-based and rehearsal-based methods introduce additional loss ...
2022
-
[2017]
Offline reinforcement learning with implicit q- learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169,
-
[2018]
Is mamba compatible with trajectory optimization in offline rein- forcement learning? arXiv preprint arXiv:2405.12094,
Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, and Li Shen. Is mamba compatible with trajectory optimization in offline rein- forcement learning? arXiv preprint arXiv:2405.12094,
-
[2019]
Oer: Offline experience replay for continual offline reinforce- ment learning
Sibo Gai, Donglin Wang, and Li He. Oer: Offline experience replay for continual offline reinforce- ment learning. arXiv preprint arXiv:2305.13804,
-
[2020]
Riemannian walk for incremental learning: Understanding forgetting and intransigence
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. 532–547, 2018a. Arslan Chaudhry, Marc’...
-
[2021]
Off-policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052–2062. PMLR,
-
[2022]
Variational continual learning
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628,
-
[2024]
Lora: Low-rank adaptation of large language models
11 Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.