{"id":"39ca6859-db7f-44f8-b81d-732788bb1494","arxiv_id":"2411.10809","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DISTR combines a task-conditioned diffusion model that generates whole high-return trajectories with behavior-cloning replay, and reports higher average success than several baselines on Continual World.","lead":"This paper introduces DISTR, a continual reinforcement learning method that uses a diffusion model to generate and replay past tasks' high-return trajectories while the agent learns new tasks. The authors report higher average success than several baselines on Continual World, with less forgetting than standard fine-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-cloned diffusion replay may drift on longer task streams; only qualitative UMAP validation is provided, so the stability claim is not yet supported beyond CW5.","rationale":"The reader identified the same weakest assumption that I consider most load-bearing: the fidelity of self-cloned diffusion replay over long task streams. The paper's experiments do not provide a quantitative check of this assumption, and the only visualization is limited to five tasks and a single task's trajectories. My proposed oracle-replay test directly isolates the contribution of the diffusion model by comparing its generated past-task trajectories against real stored ones while keeping the rest of the algorithm fixed. If the oracle variant performs substantially better, the headline result is not due to the generative memory but to the replay mechanism itself, which would weaken the paper's claimed advance over simpler replay methods. If the oracle variant performs comparably, the self-cloning drift concern is mitigated and the central claim becomes more credible. This test is feasible with the existing Continual World setup and would settle whether the conditional acceptance should be upgraded or whether additional safeguards against drift are needed.","tokens_in":12451,"tokens_out":6082,"duration_ms":62072,"concrete_test":"Run DISTR on CW10 with a diagnostic 'oracle replay' variant where, during general-policy BC training (Eq. 4), past-task trajectories are taken from a fixed set of real stored skilled trajectories instead of diffusion-generated ones, with all other components unchanged. If the oracle variant's final average success rate exceeds DISTR's reported 81.2 ± 0.2 by more than the standard error (0.2), the diffusion replay is not faithfully preserving past-task behavior, and the claimed stability-plasticity balance is not attributable to the proposed generative memory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that DISTR beats existing continual RL baselines on CW5 and CW10 by using a diffusion model as a faithful generative replay memory. The load-bearing assumption is that the diffusion model, trained incrementally with self-cloning (Section 3.2), preserves high-fidelity trajectory distributions for all past tasks indefinitely. Self-cloning means that from task k onward, the model is trained on its own generated samples {D(i)|i<k} together with real data D(k). This is a closed loop: any bias or error in generated past-task trajectories is amplified as it becomes training data for later stages, a known failure mode for generative models trained on their own outputs. If drift occurs, the general policy's BC targets for past tasks (Eq. 4) are corrupted, causing exactly the catastrophic forgetting the method aims to prevent. The only evidence against this is the UMAP visualization in Section 4.3 (Figure 3a), which is qualitative, limited to task 0, and only covers the first 5 tasks; no quantitative distribution-distance metric is reported, and CW10 is not analyzed. Because the reported Forgetting values (4.7±2.0 on CW5, 3.9±0.8 on CW10) are central to the stability claim, and because the diffusion model is the only memory of past tasks, the method's empirical success is not yet established as a property of the generative replay mechanism rather than of the specific benchmark sequence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DISTR, a continual reinforcement learning method that uses a diffusion model to memorize and replay whole trajectory distributions of previously encountered tasks. Policy learning is decoupled into an immediate policy trained with SAC on the current task and a general policy trained purely with behavior-cloning losses on replayed and current trajectories. The diffusion model is trained continually on real current-task trajectories plus its own generated past-task trajectories (self-cloning), and a prioritization mechanism selects which tasks to replay based on vulnerability and specificity. Experiments on Continual World CW5 and CW10 report the highest final average success rate among the five baselines considered.","tokens_in":12756,"tokens_out":3642,"duration_ms":37592,"significance":"If the claims are substantiated, DISTR would be a meaningful step toward storage-efficient and privacy-preserving continual RL, advancing generative replay from individual observations to full trajectory distributions. The decoupled immediate/general policy training scheme is a sensible design, and the empirical results are suggestive. However, the paper's central empirical claim is not yet fully supported: key components are not ablated, the two closest diffusion-based replay methods are not compared, and the generative memory's fidelity is validated only qualitatively. These gaps are addressable and would strengthen the work considerably.","major_comments":[{"comment":"The stability claim rests on the diffusion model preserving past-task trajectory distributions under self-cloning, but Section 4.3 validates this only with a UMAP visualization for task 0 on CW5. No quantitative distribution-fidelity metric is reported, and CW10 is not analyzed. Because Eq. (4) trains the general policy on generated trajectories, drift in the diffusion model would directly corrupt the behavior-cloning targets and cause the forgetting the method aims to prevent. Please add quantitative fidelity evaluation (e.g., MMD, trajectory return, or success rate of generated trajectories) for all tasks and both task sequences.","section":"Section 3.2, Section 4.3"},{"comment":"No ablation isolates the contribution of the diffusion model, the self-cloning mechanism, or the prioritization mechanism. For example, replacing generated trajectories with real stored trajectories, disabling self-cloning, or replacing prioritization with uniform replay would clarify which component drives the reported gains. Without such ablations, Table 1 supports only the full system against the chosen baselines, not the claimed mechanism.","section":"Sections 3.1-3.4, Table 1"},{"comment":"The two closest diffusion-based replay methods, t-DGR and CuGRO, are acknowledged in Section 5.1 but are not compared. Since the paper's central claim is that diffusion-based trajectory-level replay outperforms existing generative replay, a comparison or a clearly justified reason for its absence (e.g., different problem setting) is needed. If the settings differ, provide a comparison on a common benchmark or at least a state-level diffusion replay baseline.","section":"Section 5.1, Table 1"},{"comment":"The reported average performance for DISTR on CW10 (81.2±0.2) overlaps with PackNet (80.8±0.9). The paper does not report the number of seeds or statistical significance tests. The claim of the highest average success rate is therefore not established against PackNet. Please report per-seed results, number of seeds, and significance tests (e.g., paired t-test or bootstrap) for all metrics.","section":"Section 4.1, Table 1"}],"minor_comments":[{"comment":"There are several typos and grammatical issues: \"wakeups\" should be \"wakes up\" in the abstract, \"provalent\" should be \"prevalent\", \"disguish\" should be \"distinguish\" in Section 3.2, and \"The most close to ours\" should be \"The closest to ours\" in Section 5.1.","section":"Abstract, Sections 3.2 and 5.1"},{"comment":"The label \"Distillation\" in Figure 1 is not defined or explained in the text; please clarify what is distilled and how it relates to the two-policy training scheme in Section 3.1.","section":"Figure 1"},{"comment":"The priority formula (s_v + 1 - s_s)/2 and its normalization are not fully specified. Please clarify how the top trajectories are selected, how N_traj is allocated across tasks, and what happens when the number of tasks exceeds the replay budget.","section":"Section 3.3"},{"comment":"The diffusion loss is written as L(θ) = E[∥ϵ − ϵθ(xt, t)∥], but the text says the L1 loss is used; the norm notation should specify ∥·∥_1 to avoid ambiguity.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The missing comparison with t-DGR and CuGRO is the most consequential gap, as the paper explicitly identifies them as concurrent methods with the closest setting. I do not see a logical circularity in the self-cloning training loop, but the empirical drift risk is real and needs quantitative evaluation. The related-work citation to Zhang et al. (2024) involves authors from the same group; this is not a problem per se, but the paper should ensure the citation is justified by actual content overlap. The manuscript would benefit from reporting seeds and significance tests, especially given the small margin over PackNet on CW10."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: DISTR is a coherent empirical paper whose headline result is plausible but not fully nailed down. It reports higher average success than its chosen baselines on Continual World, and the method design is cleaner than most generative replay work. The soft spots are the usual workshop ones: no ablations of the prioritization mechanism, no comparison with the two closest diffusion replay methods, and only a UMAP plot as evidence that the self-cloned diffusion memory stays faithful.\n\nWhat's actually new is the combination: a task-id-conditioned diffusion model trained on whole trajectories, with self-cloning (training on its own generated past-task data), and a simple heuristic to prioritize which tasks to replay. Whole-trajectory generation is a real difference from t-DGR and CuGRO, which work on states. The two-policy training scheme—an immediate policy for the new task and a general policy distilled by BC from real plus generated data—is sensible and avoids the messy multi-loss tuning that plagues many replay methods.\n\nThe empirical results are genuinely good for the benchmark: 84.8 ± 2.3 on CW5 and 81.2 ± 0.2 on CW10, above PackNet and far above RePR. The paper also does the right thing by acknowledging the closest diffusion-based works and stating that comparison is future work.\n\nNow the soft spots. The load-bearing assumption is that the self-cloned diffusion model keeps generating faithful high-return trajectories for all past tasks. That is exactly the kind of closed loop that can drift: generated data becomes training data, so errors can compound. The only evidence is a qualitative UMAP visualization for task 0 on CW5. No quantitative distribution distance, no CW10 analysis, no longer task streams. That doesn't sink the paper, but it means the stability claim is supported by the forgetting numbers, not by a direct test of the memory. Second, the prioritization mechanism is not ablated, so we don't know whether it helps or hurts. Third, no code or hyperparameters, and the evaluation is a single benchmark. These are real but proportionate limitations, not fatal flaws.\n\nVerdict: this deserves a serious referee. I would send it to review, and ask for ablations of prioritization, a quantitative generation-fidelity metric, and analysis on CW10. I would not treat DISTR as a settled best practice yet, but it is a useful and honest contribution.","headline":"DISTR is a clean, plausible empirical result on Continual World; the lack of ablations, closest-baseline comparisons, and quantitative generation-fidelity evidence makes it a conditional accept rather than a settled one.","tokens_in":13269,"tokens_out":3247,"would_cite":true,"duration_ms":33642,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion-generated trajectory replay lets a continual reinforcement learning agent learn task sequences without storing raw experience, and reaches the top average success rates on the Continual World benchmark.","keywords":["continual reinforcement learning","catastrophic forgetting","generative replay","diffusion models","trajectory replay","stability-plasticity trade-off","prioritized replay","Continual World"],"falsifier":"Run DISTR on a noticeably longer task stream, say twenty Continual World tasks instead of ten, and after each training stage generate trajectories for an early task and compare them against held-out real trajectories of that same task with a distributional distance such as FID or coverage. Then compare the final policy's success on that early task when trained with generated replays versus with an identical setup where the same high-return trajectories are stored and replayed exactly; if exact replay increasingly outperforms generated replay as the stream grows, the generative memory is the bottleneck and the central claim fails.","tokens_in":12245,"feed_emoji":"🤖","tokens_out":8100,"duration_ms":72061,"temperature":0.7,"pith_summary":"This paper argues that the reason generative replay has underperformed stored replay in continual reinforcement learning is model capacity: GANs and VAEs cannot faithfully reproduce whole state-action trajectories, so the policy trains on distorted memories. It proposes DISTR, a method that trains a diffusion model to memorize the distribution of high-return trajectories for each encountered task and, while learning a new task, replays generated trajectories from past tasks. On Continual World, the method reports the highest average success rate among the compared baselines on both CW5 (84.8±2.3) and CW10 (81.2±0.2), with positive forward transfer. A sympathetic reader would care because, if the generative memory holds, continual agents can keep lifelong skills without storing raw experience, saving storage and respecting data-privacy constraints.","feed_headline":"Diffusion replay beats forgetting in continual RL","feed_subtitle":"Generated past-task trajectories let an agent keep old skills and top Continual World scores without storing raw data.","key_machinery":"The load-bearing object is a task-conditioned Denoising Diffusion Probabilistic Model that generates whole state-action trajectories rather than individual states, which avoids the per-timestep state imbalance and compounding errors of earlier generative replay. During each training stage the model is fit on real skilled trajectories of the newest task together with its own generated trajectories from older tasks, a self-cloning procedure intended to keep older distributions alive inside the generative model itself. Replay is not uniform: a prioritization mechanism scores each past task by vulnerability (the drop in success rate when policy outputs are perturbed) and specificity (how little a task can benefit from other tasks' data), and samples past-task trajectories with probability proportional to that score. The general policy is then distilled from the mixture of real and generated trajectories through behavior cloning, so replay gradients never interfere with the learning of the new task.","core_discovery":"The central claim is that a diffusion model can serve as a faithful generative memory for whole trajectories in online continual RL, and that this is enough to make the agent's long-term policy stable without sacrificing plasticity. DISTR decouples policy training into two stages: an immediate policy learns only the current task with SAC, and a general policy is then trained purely by behavior cloning on real current-task trajectories plus diffusion-generated past-task trajectories. The diffusion model is updated after every task on real high-return trajectories of the current task together with its own generated past-task trajectories, so that each task's trajectory distribution remains available under a task-id condition. The reported outcome is a final average success rate of 84.8±2.3 on CW5 and 81.2±0.2 on CW10, with forgetting close to parameter-isolation methods and higher forward transfer, plus UMAP visualizations in which generated trajectories overlap the real data distribution while GAN-generated ones do not.","pith_inferences":["If the self-cloning diffusion memory scales to longer streams, the same trajectory-replay design could transfer to continual imitation learning and offline continual RL, where expert or logged trajectories are already available; the paper evaluates only online RL on Continual World.","The prioritization score is a heuristic combining vulnerability and specificity, and the paper does not compare it against uniform random replay; an ablation would show whether the mechanism is necessary at moderate task counts.","The UMAP overlap between generated and real trajectories is qualitative; adding a quantitative distribution-distance audit after each task would give practitioners a concrete threshold for when the generative memory needs refreshing.","The same diffusion memory could double as a data synthesizer in multi-task RL, generating trajectories for related tasks from a shared task-conditioned distribution rather than training a separate memory per task."],"forward_implications":["If diffusion replay stays faithful over long streams, continual RL agents can drop raw experience buffers entirely, removing growing storage costs and enabling learning from streams where data cannot be retained.","Because the immediate policy sees only the current task's RL loss, and replay is injected afterward through behavior cloning, the stability-plasticity trade-off is governed by the quality of the generative memory rather than by hand-tuned loss weights.","The prioritization mechanism gives the method a route to infinitely long task streams by spending the replay budget on tasks that are most fragile and least replaceable.","On the tested benchmark, DISTR outperforms the GAN-based generative replay baseline by a wide margin and matches or exceeds parameter-isolation methods while keeping higher forward transfer.","Because trajectories are generated directly, the method needs no separate world model and avoids the compounding-error issue of model-based generative replay."],"supporting_citations":[{"why":"supplies the Continual World benchmark and the CW5/CW10 task sequences used for all experiments.","marker":"Wolczyk et al., 2021"},{"why":"supplies SAC, the RL algorithm used to train the immediate policy on each new task.","marker":"Haarnoja et al., 2018"},{"why":"supplies the DDPM diffusion backbone used to generate and replay whole trajectories.","marker":"Ho et al., 2020"},{"why":"defines RePR, the GAN-based generative replay baseline that DISTR is compared against and visualized alongside.","marker":"Atkinson et al., 2021"},{"why":"defines EWC, the regularization-based baseline used in the main comparisons.","marker":"Kirkpatrick et al., 2017"},{"why":"defines PackNet, the parameter-isolation baseline whose performance DISTR surpasses.","marker":"Mallya & Lazebnik, 2018"},{"why":"supplies the UMAP projection used to visualize that generated trajectories cover the real data distribution.","marker":"McInnes et al., 2018"}],"fun_headline_variants":["Diffusion model replays old tasks to stop RL forgetting","Diffusion-based replay keeps RL agents stable across tasks","Forget-proof RL via diffusion-generated trajectory replay","Diffusion memory lets RL agents keep old skills without raw data","Diffusion-based trajectory replay achieves stable continual RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on the diffusion model, retrained on its own past outputs, to keep an accurate memory of what good behavior looked like on every earlier task; if that memory drifts, everything replayed from it is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion model replays old tasks to stop RL forgetting","Diffusion-based replay keeps RL agents stable across tasks","Forget-proof RL via diffusion-generated trajectory replay","Diffusion memory lets RL agents keep old skills without raw data","Diffusion-based trajectory replay achieves stable continual RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2902,"prompt_tokens":1032,"completion_tokens":1870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1793}},"tokens_in":648,"tokens_out":1870,"duration_ms":13319,"temperature":1.0,"reasoning_tokens":1793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:17:09.905868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DISTR on a noticeably longer task stream, say twenty Continual World tasks instead of ten, and after each training stage generate trajectories for an early task and compare them against held-out real trajectories of that same task with a distributional distance such as FID or coverage. Then compare the final policy's success on that early task when trained with generated replays versus with an identical setup where the same high-return trajectories are stored and replayed exactly; if exact replay increasingly outperforms generated replay as the stream grows, the generative memory is the bottleneck and the central claim fails.","supporting_citations":[],"review_version":1}