{"id":"4143a5ef-dd4b-440b-b656-e7d6edd6f571","arxiv_id":"2411.11364","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CompoFormer adaptively composes prior task policies in a Decision Transformer to improve stability and plasticity in continual offline RL, and the paper introduces the OCW benchmark.","lead":"This paper presents CompoFormer, a method for continually learning multiple robotic tasks from offline data by composing previous task policies through learned attention. It also introduces the Offline Continual World benchmark, where the method outperforms many continual learning baselines on longer task sequences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Longer-sequence advantage rests on OCW20's exact task repeats, making it repeat retrieval rather than compositional transfer; the only non-repeated evidence is within noise.","rationale":"The reader's weakest assumption was that S-BERT text embeddings plus a warmed-up attention module select policies that truly transfer to a new task. I agree that this is an important mechanism, but the more load-bearing issue is experimental design: the marquee OCW20 result is obtained by repeating the same task sequence, so CompoFormer's ability to recognize exact text matches turns the longer-sequence benchmark into a retrieval task rather than a test of compositional transfer. The non-repeated evidence is fragile because one variant, CompoFormer-Grow, loses to PackNet on OCW10, and the other variant's gain is within noise with no significance test. Thus, even if the attention mechanism works as intended, the paper's generalized claim that CompoFormer excels 'particularly in longer task sequences' is not established. This does not invalidate the OCW benchmark or the engineering contribution, and a conditional acceptance requiring a non-repeated long-sequence experiment remains the appropriate verdict. I mark partial agreement because my concern overlaps with the reader's concern about textual alignment and the reliability of the grow/no-grow gate, but it is distinct: the problem is not only whether text predicts transfer, but whether the strongest experimental result is artificially easy because of exact task repetition.","tokens_in":19861,"tokens_out":8867,"duration_ms":104512,"concrete_test":"Construct a matched 20-task OCW benchmark with no exact task repeats, for example the 10 OCW tasks plus 10 distinct remaining Meta-World tasks. Run CompoFormer-Grow, CompoFormer-Prune, and PackNet with identical training budgets and seeds, and report per-task success with confidence intervals or significance tests for the differences. If Prune's advantage over PackNet on this non-repeated sequence is within one standard deviation, as it is on OCW10, then the longer-sequence claim reduces to repeat retrieval and the central argument fails; if the advantage persists with non-overlapping intervals, the concern is resolved. A complementary check is to compare the learned attention weights against the precomputed transfer matrix used to order OCW10, to verify that the selected prior policies are actually the ones that transfer for non-identical task pairs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that CompoFormer's adaptive composition of previous policies yields a better stability-plasticity trade-off than standard continual learning methods, especially on longer task sequences. The strongest quantitative support is OCW20: CompoFormer-Prune reaches 0.73±0.04 versus PackNet's 0.57±0.04. However, OCW20 is defined in Appendix A as the same OCW10 sequence repeated twice. Because the grow/no-grow decision and the attention weights are driven by S-BERT embeddings of task descriptions, the second occurrence of each task is an exact textual match to a stored policy; the 'adaptive composition' therefore degenerates to retrieving an existing identical policy. This demonstrates task identification and reuse, not compositional transfer to genuinely new tasks. On the only non-repeated benchmark, OCW10, CompoFormer-Grow (0.60±0.06) is numerically below PackNet (0.64±0.06), and Prune's margin (0.69±0.01 versus 0.64±0.06) is within one standard deviation of the baseline with no significance test reported. The claim that the method is 'particularly' effective on longer sequences remains unsupported unless a longer sequence of distinct tasks shows the same advantage. Relatedly, Algorithm 1 always assigns and warms up a new head(k) even on a no-grow branch, so the 'no additional learnable parameters' scenario is not literally parameter-free; this is secondary to the benchmark issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies continual offline reinforcement learning (CORL) with a sequence of Meta-World manipulation tasks. It introduces the Offline Continual World (OCW) benchmark, consisting of OCW10 and OCW20, where OCW20 is defined as the OCW10 sequence repeated twice (Appendix A, Table 3). The proposed method, CompoFormer, is a Decision Transformer-based modular architecture with two variants: CompoFormer-Grow, which adds LoRA parameters for each new task, and CompoFormer-Prune, which uses binary masks. For each new task, a frozen Sentence-BERT encoder embeds the task description; a trainable attention module scores previous task embeddings and composes the output features of prior policies. If the composed policy exceeds a predefined threshold after a warmup phase, no new sub-network is added; otherwise, new parameters are introduced. The paper compares CompoFormer with regularization-, structure-, and rehearsal-based continual learning baselines on OCW10 and OCW20 and reports average performance, forgetting, and forward transfer, together with ablations and task-order robustness experiments. The central claim is that CompoFormer consistently outperforms all baselines, particularly on longer task sequences, achieving a better stability-plasticity trade-off.","tokens_in":20054,"tokens_out":5473,"duration_ms":53624,"significance":"If the claimed results held, the paper would make a useful contribution: the OCW benchmark provides a common evaluation suite for continual offline RL, and the idea of using task-description embeddings to select and compose prior policies is a reasonable and testable alternative to fixed parameter sharing. The paper is also commendable for releasing code and for reporting detailed hyperparameters and ablations. However, the evidence as presented does not establish the headline claim. The strongest quantitative result is on OCW20, but OCW20 consists of exact task repeats, so the attention mechanism can retrieve an already stored identical policy by matching the task description; this demonstrates task identification and reuse rather than compositional transfer to genuinely new tasks. On the only non-repeated benchmark, OCW10, CompoFormer-Prune's margin over PackNet is within one standard deviation, CompoFormer-Grow is numerically worse than PackNet, and no significance tests are reported.","major_comments":[{"comment":"The statement that \"CompoFormer consistently outperforms all other methods\" is contradicted by Table 1: CompoFormer-Grow reaches 0.60 plus or minus 0.06 average performance on OCW10, while PackNet reaches 0.64 plus or minus 0.06. CompoFormer-Prune's margin (0.69 plus or minus 0.01 versus 0.64 plus or minus 0.06) is within one standard deviation of the baseline. Since only three seeds are reported and no significance tests are provided, the consistency claim is not supported for OCW10.","section":"Section 5.2, Table 1"},{"comment":"OCW20 is defined as the OCW10 sequence repeated twice, and Appendix F confirms that for the second half of OCW20 the model assigns highest attention to the corresponding task from the first half. The large OCW20 advantage (CompoFormer-Prune 0.73 plus or minus 0.04 versus PackNet 0.57 plus or minus 0.04) is therefore better interpreted as retrieval of an identical previously stored policy than as compositional transfer to new tasks. On OCW10, the only benchmark without exact task repeats, the advantage over PackNet is not significant, and CompoFormer-Grow trails PackNet. The paper's emphasis on \"particularly in longer task sequences\" accordingly needs support from a longer sequence of distinct tasks, or at minimum an analysis that separates reuse from transfer.","section":"Section 5.1, Appendix A, Table 3; Section 5.2, Table 1; Appendix F"},{"comment":"Algorithm 1 assigns a new head(k) for every task at line 4 and updates the parameters of head(k) together with the attention parameters {W^Q, W^K} during the warmup phase at lines 10-12, even when flag is set to False and no LoRA or pruning parameters are added. This contradicts the statement in Section 4.2 that scenario (i) \"avoids the need for additional learnable parameters.\" The parameter accounting and the description of the no-growth case should be corrected.","section":"Algorithm 1 versus Section 4.2"},{"comment":"The text claims that CompoFormer \"consistently outperforms LoRA and PackNet\" across task orders, but the table shows CompoFormer-Grow's average performance (0.52) is below PackNet's (0.65), and CompoFormer-Grow is below PackNet in every individual order (0.60 versus 0.64, 0.54 versus 0.67, 0.43 versus 0.65, 0.51 versus 0.65). Only CompoFormer-Prune supports the claim. The claim should either be restricted to the Prune variant or the Grow results need additional evidence and explanation.","section":"Section 5.3, Table 2"}],"minor_comments":[{"comment":"The ablation figure reports \"averaged over three random seeds\" but shows no error bars or standard deviations; adding them or a corresponding table would make the comparison more informative.","section":"Figure 4b"},{"comment":"The condition \"performance is greater than or equal to eta\" is not accompanied by a precise evaluation protocol; the manuscript should state whether this is success rate on the offline validation data, how many evaluation episodes are used, and whether the evaluation is performed with the composed policy alone.","section":"Algorithm 1, line 14"},{"comment":"The dimensions of the value matrix V, which is the concatenation of policy output features Phi^(i), and their alignment with the key embeddings are not specified; a sentence clarifying the shapes would remove ambiguity.","section":"Equation (5), Section 4.2"},{"comment":"The notation is confusing: Equation (17) is written as T(k), but the text then says this is the complexity of a single module and that the total complexity is k times O(k); please clarify whether T(k) denotes single-module or total inference time.","section":"Appendix D"},{"comment":"The strong wording \"consistently outperforms all other methods\" appears in several places before the detailed results are introduced; the wording should be calibrated to match the actual comparative results in Tables 1 and 2, for example by distinguishing the Grow and Prune variants.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a continual-learning or offline-RL venue, and the benchmark plus code release are potentially useful. My main concern is that the central claim of superiority on longer sequences rests on OCW20, which is a repetition of OCW10; this is explicitly acknowledged in Appendix A and the attention heatmaps in Appendix F show retrieval. I would encourage the authors to add a longer sequence of distinct tasks, or to reframe the OCW20 result as evidence of task identification and reuse. The contradictions between the textual claims and the numbers in Tables 1 and 2, as well as the parameter accounting in Algorithm 1, should be fixed in revision. Given the moderate confidence of the reader's report and the fact that the main evidence is empirical, I do not see grounds for rejection, but the load-bearing comparison needs additional support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper for a niche subfield, but the headline claim is bigger than the evidence. The genuinely new parts are the Offline Continual World benchmark and the offline-RL adaptation of self-composing policies with text-based attention. That combination is not in Malagon et al., and the paper does a service by benchmarking eleven CL baselines on a controlled offline version of Continual World.\n\nWhat is good: the benchmark construction is documented, the baselines come from standard implementations with a hyperparameter grid in the appendix, the ablations test sharing versus addition versus selection, and the attention visualizations support that the module picks semantically related tasks. Code is public. Prune on OCW10 (0.69) is the best structure-based result and forgetting is near zero, so the method is plausible.\n\nSoft spots: the central claim about long sequences rests on OCW20, which repeats OCW10 exactly. In the second half, the S-BERT embeddings are near-identical to stored policies, so the attention module is retrieving an exact match rather than composing transferable knowledge for a genuinely new task. That makes the \"particularly in longer sequences\" claim unsupported; it is a task-identification result. On OCW10, Grow (0.60) is numerically below PackNet (0.64), and Prune's margin (0.69 vs 0.64) is within one standard deviation with three seeds and no significance tests. So the paper's sentence that CompoFormer \"consistently outperforms all other methods\" is not accurate as written. Also, Algorithm 1 always trains a new head and attention parameters before deciding whether to grow, so the \"no additional learnable parameters\" scenario is not literally parameter-free; that is minor but should be fixed.\n\nWho it is for: researchers working on continual offline RL or structure-based continual learning. They will get a useful benchmark and a reasonable baseline architecture. The paper deserves a serious referee, but the authors should be asked to either add a longer sequence of distinct tasks or soften the long-sequence claim, and to report significance or more seeds.\n\nRecommendation: engage with it, but treat the benchmark as the main contribution and demand the claims be matched to the actual evidence.","headline":"CompoFormer is a useful benchmark plus architecture for continual offline RL, but the long-sequence advantage is repeat retrieval rather than compositional transfer, and the headline overstates the numbers.","tokens_in":20677,"tokens_out":2212,"would_cite":true,"duration_ms":22496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By composing previously learned policies through semantic attention, CompoFormer shows that continual offline reinforcement learning can avoid catastrophic forgetting and reuse old skills when tasks repeat.","keywords":["continual reinforcement learning","offline reinforcement learning","catastrophic forgetting","stability-plasticity tradeoff","policy composition","semantic attention","decision transformer","offline continual world"],"falsifier":"Compare the learned attention weights to an independent transfer measurement: train a policy on each task, fine-tune it briefly on every other task, build a pairwise transfer matrix from the resulting scores, and correlate it with the attention weights CompoFormer assigns between the same task pairs. If semantically similar pairs with poor measured transfer receive high attention, or if the two matrices do not correlate, then the semantic composition step is not the mechanism producing the reported gains.","tokens_in":19582,"feed_emoji":"🧩","tokens_out":13700,"duration_ms":116428,"temperature":0.7,"pith_summary":"An agent that learns a sequence of manipulation tasks from pre-collected datasets, with no live interaction with the environment, must keep old skills intact while mastering new ones, and this paper claims that the right way to do so is to compose rather than replay or overwrite. For each new task, CompoFormer first tries to solve it by combining the already-trained policies of previous tasks, choosing which to borrow through an attention mechanism that compares the tasks' text descriptions, and only grows new network parameters when that combination falls short of a performance threshold. The authors build a new benchmark, Offline Continual World, from ten manipulation tasks with offline datasets, and find that standard continual learning methods forget badly in this setting, losing more than half of prior performance. CompoFormer keeps forgetting near zero while learning new tasks faster than structure-based baselines, and its best results come on a twenty-task sequence in which the same ten tasks are revisited, the regime where reusing the right old skills matters most.","feed_headline":"Self-composed policies cut forgetting in continual offline RL","feed_subtitle":"A semantic gate borrows only relevant past skills, growing new parameters only when needed — no replay.","key_machinery":"The load-bearing object is the self-composing policy module: a scaled dot-product attention block (Eq. 5) whose query is the sentence embedding of the new task's text description, whose keys are the embeddings of all earlier task descriptions, and whose values are the output features of the earlier task-specific policy sub-networks. A brief warm-up on the current task's offline data trains the query and key projections, and the resulting attention scores decide the grow/no-grow switch: if the composed policy clears a performance threshold (eta = 0.8) it is used as is, otherwise new parameters are added, LoRA low-rank matrices in the Grow variant and binary neuron masks in the Prune variant, and the new sub-network's output is concatenated with the attended prior outputs before a final MLP produces actions. This module is where semantic text and learned behavior meet, and it is what turns the model into a cascading stack of policies that grows only when needed.","core_discovery":"On this paper's terms, the discovery is that in continual offline reinforcement learning the effective unit of transfer is the composed policy, selected by semantic attention, rather than shared hidden representations or replayed data. Given a new task, CompoFormer encodes the task's textual description with a frozen Sentence-BERT model, treats that embedding as the query to an attention module whose keys are the embeddings of all previous task descriptions and whose values are the output features of the previous task policies (Eq. 5), and after a short warm-up on the new task's offline data either accepts the composed policy as sufficient or grows the model. Growth comes in two variants: CompoFormer-Grow adds LoRA low-rank adaptation matrices, while CompoFormer-Prune carves out binary masks of previously idle parameters; in both cases the new sub-network's output is concatenated with the attended prior outputs and projected to actions. Because old parameters are frozen, reported forgetting is essentially zero, and because the attention is trained on data from the current task, related skills such as two tasks that both push a puck receive high weight while unrelated tasks are down-weighted. On the new Offline Continual World benchmark the Prune variant achieves average performance 0.69 on the ten-task sequence and 0.73 on the repeated twenty-task sequence, ahead of the structure-based baselines it is compared with and remaining below the multi-task upper bound.","pith_inferences":["Because the attention weights are trained on the current task's data and visibly concentrate on related skills, they double as a learned measure of task relatedness; a natural test the paper leaves open is using them to order the task curriculum, since the paper only evaluates shuffled orders, not attention-guided ones.","The paper reports inference cost that grows as the square of the number of composed policies and names computational cost as the remaining bottleneck; a direct extension is sparsifying the attended set of prior policies so composition stays roughly constant-cost as the task count grows.","The design separates the question of which skills to reuse (semantic attention plus threshold) from how to add new capacity (LoRA or masks), so the same gate could be carried onto other base sequence models or other descriptors of tasks, such as demonstrations or reward functions, without changing the composition logic.","The Offline Continual World benchmark itself is a reusable testbed: it could be used to check whether any generative-replay or rehearsal method can close the gap to CompoFormer, or whether the distribution-shift handicap the paper describes is intrinsic to replay in offline RL."],"forward_implications":["Forgetting stays near zero without storing any past experience: freezing each task's sub-network and composing old policies removes the need for rehearsal buffers, which the paper shows hurt in offline RL.","Revisited tasks become a strength: on OCW20, where the same ten tasks appear twice, CompoFormer attends back to the first occurrence and posts its best average performance (Prune: 0.73), whereas fixed-capacity structure baselines decline.","The advantage widens as sequences lengthen, which is precisely the regime where regularization and replay methods forget most and where parameter-isolation kits run out of free capacity.","Plasticity does not have to be traded away for stability: the attended prior knowledge makes new tasks learn faster, as the paper's learning curves show faster adaptation than all baselines.","The reported gains are specific: forward transfer to tasks never seen remains small for every method including CompoFormer, so the mechanism improves stability and adaptation speed, not the ability to anticipate genuinely new tasks."],"supporting_citations":[{"why":"Supplies the Decision Transformer base architecture whose sequence-modeling objective (Eq. 2) every method in the paper, including CompoFormer, is trained with.","marker":"(Chen et al., 2021)"},{"why":"Provides the frozen Sentence-BERT encoder that turns each task's text description into the embedding used as the attention query.","marker":"(Reimers, 2019)"},{"why":"Defines Continual World, the robotic benchmark and evaluation protocol the paper adapts into the offline setting as Offline Continual World.","marker":"(Wolczyk et al., 2021)"},{"why":"Introduces self-composing policies for continual RL, the progressive composition idea CompoFormer extends with semantic attention and a grow-or-not gate.","marker":"(Malagon et al., 2024)"},{"why":"Contributes the iterative-pruning technique behind CompoFormer-Prune and serves as one of the structure-based baselines.","marker":"(Mallya & Lazebnik, 2018)"},{"why":"Supplies the low-rank adaptation (LoRA) matrices that CompoFormer-Grow trains as the new parameters for each task.","marker":"(Hu et al., 2021)"},{"why":"Provides the distribution-shift analysis of continual offline RL that the paper leans on to explain why replay- and regularization-based baselines fail.","marker":"(Gai et al., 2023)"},{"why":"Gives the scaled dot-product attention used in Eq. 5 to compose previous policy outputs.","marker":"(Vaswani, 2017)"},{"why":"Provides Meta-World, the task suite from which the ten manipulation tasks of the Offline Continual World benchmark are drawn.","marker":"(Yu et al., 2020)"},{"why":"Supplies the pre-computed transfer matrix used to rank tasks in the benchmark and the meta-policy-network formulation the method builds on.","marker":"(Yang et al., 2023)"}],"fun_headline_variants":["Policy self-composition tames continual offline RL forgetting","Composed policies beat replay in continual offline RL","Semantic attention composes policies for zero-forgetting RL","Offline RL agents grow only when needed, compose past skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on one premise: the text descriptions of the tasks, once embedded and combined by attention, actually point to the older policies whose behavior helps the new task, and the short warm-up on the new task's data reliably reveals when the composed policy is already good enough.","fun_headline_variants_meta":{"raw":{"variants":["Policy self-composition tames continual offline RL forgetting","Composed policies beat replay in continual offline RL","Semantic attention composes policies for zero-forgetting RL","Offline RL agents grow only when needed, compose past skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3693,"prompt_tokens":1023,"completion_tokens":2670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2605}},"tokens_in":639,"tokens_out":2670,"duration_ms":20242,"temperature":1.0,"reasoning_tokens":2605,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:37:00.630292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the learned attention weights to an independent transfer measurement: train a policy on each task, fine-tune it briefly on every other task, build a pairwise transfer matrix from the resulting scores, and correlate it with the attention weights CompoFormer assigns between the same task pairs. If semantically similar pairs with poor measured transfer receive high attention, or if the two matrices do not correlate, then the semantic composition step is not the mechanism producing the reported gains.","supporting_citations":[],"review_version":1}