REVIEW 3 major objections 5 minor 14 references
In replay-protected world-model agents, forgetting is a channel problem, not a memory problem.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Under replay, DreamerV3's world model remembers old tasks but its actor forgets; supervised self-imitation on graded dreams retains behavior where RL-in-imagination fails.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection A transparent, useful localization of forgetting in replay-protected MBRL, but the paper's signature channel-isolation comparison is confounded by non-identical training data. the 3 major comments →
The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
Under never-clear replay, the world model of a Dreamer-family agent is a sufficient behavioral memory at the tested scale: reward discrimination, value estimates, and termination structure for old tasks survive (retention ratio near 1.0), and the actor loses old-task behavior anyway. With the world model frozen and identical imagined rollouts, policy-gradient RL in imagination fails to recover a lost skill (0/3 seeds); supervised self-imitation on the world model's own graded dreams recovers it (3/3 seeds, zero environment steps). So forgetting in this regime is a channel problem, not a memory problem. Interleaved graded dream rehearsal—behavior cloning the top 25% of imagined trajectories s
What carries the argument
Graded dream rehearsal: after each task phase, the agent keeps that phase's buffered episodes as rehearsal starts; during later-task training it imagines rollouts from those states with the current sampling actor, scores each imagined trajectory with the live reward/continuation/value heads, and behavior-clones the top 25%. The scoring rule (Equation 1) is load-bearing: a realized-first gate that weights each step by reach probability under the continuation head, prioritizes trajectories that actually achieved reward in imagination, and bootstraps only through surviving steps. This makes grading safe for imagined data, where dreams do not stop at their own endings and the critic's optimism i
Load-bearing premise
The load-bearing premise is that an imagined rollout from the current actor is a faithful stand-in for real old-task experience, and that the two recovery arms truly train on identical imagined data; if the RL arm's data drifts further from old-task states than the imitation arm's does, the channel-isolation conclusion weakens.
What would settle it
Re-run the recovery race with both arms consuming the same pre-generated, fixed set of imagined trajectories (no on-policy regeneration), so the data distribution cannot diverge between arms. If RL-in-imagination then recovers the lost skill on a substantial fraction of seeds, the claim that the policy-gradient channel itself is the failure would be falsified.
If this is right
- If the localization is right, replay-based continual MBRL should stop treating the world model as the component needing protection; it is already protected by replay, so effort should go into a stable behavioral readout.
- A single actor, with no task labels and no parameter growth, can retain long task chains when periodically imitating graded dreams of prior tasks.
- The grading rule, not the rehearsal schedule, is where the difficulty lies: a naive scorer selects near-randomly on short lethal episodes (AUC 0.49–0.56), and the realized-first rule closes both failure modes.
- Dream rehearsal outperforms matched real-episode cloning on the hardest task by a paired difference of +0.13 with complete seed separation, so imagination contributes beyond the simplest supervised channel.
- At doubled chain length (eight tasks), the mechanism still passes all seeds, with no observed rehearsal dilution; retention settles at each task's own competence ceiling.
Where Pith is reading between the lines
- Our inference: the paper's candidate explanation—that the actor–critic bootstrap chases unconstrained critic optimism in imagination—is directly testable by penalizing or calibrating the critic during imagination-only recovery; a successful RL recovery under pessimistic bootstrapping would confirm the mechanism.
- Our inference: the result suggests that any world model able to generate realistic rollouts could serve as a data source for supervised policy distillation, potentially transferring to settings where reward or continuation heads are unreliable.
- Our inference: the reported recovery race may be confounded if the RL arm's imagined data distribution drifts away from demonstrated states after the first update; a fixed pre-generated dream buffer held identical across both arms would settle whether the channel, not the data, is the cause.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies catastrophic forgetting in DreamerV3-style model-based RL with a never-cleared replay buffer. Using MiniGrid task chains and n=3 seeds, it reports that the world model retains reward discrimination, value estimates, and termination structure for earlier tasks, while the actor's behavior collapses; freezing the actor or the world model alone fails. It then reports a recovery race from a frozen world model: RL-in-imagination fails (0/3) whereas supervised self-imitation on graded imagined rollouts succeeds (3/3), with zero environment interaction. Interleaving this 'dream rehearsal' during training yields 3/3 passes on four- and eight-task chains, outperforming matched real-episode cloning, and the authors identify two failure modes of dream grading and propose a realized-first scoring rule with an offline selection gauge. All experiments are pre-registered and per-seed data are provided. The paper's central framing is that forgetting in this regime is a channel problem, not a memory problem, and that behavior can be read out of a replay-maintained world model via a supervised channel.
Significance. If correct, the component-level localization is an important negative result for the assumption behind replay-based continual MBRL: protecting the world model is not enough. The recovery and rehearsal results are a concrete, falsifiable intervention with practical potential. The manuscript's strengths are real: pre-registered protocols, full per-seed reporting, honest discussion of refuted hypotheses and bugs, direct probes rather than inference from returns, and an offline selection gauge. However, the headline channel-isolation claim rests on a comparison that does not actually hold data fixed; the recovery arms differ in data-generation distribution and in selection/filtering, so the 0/3 vs 3/3 split cannot yet be attributed to the learning channel. The paper's own limitations (§9) honestly delimit scale, n=3, and fixed order, but the channel claim is a central contribution and needs a controlled replication.
major comments (3)
- [§5 and Fig. 1 caption; Abstract] The claim that the two recovery arms receive 'identical imagined rollouts' is not supported by the protocol. The RL arm applies the standard DreamerV3 actor-critic update to rollouts generated by the current (drifting) actor, while the dream self-imitation arm behavior-clones the top 25% of trajectories scored by the realized-first rule, also generated by the current actor. After the first update the two actors differ, so the imagined data distributions diverge. Moreover, the BC arm trains on a filtered subset (top-quartile) while RL trains on all rollouts. Thus the 0/3 vs 3/3 split varies the learning rule, the data distribution, and the selection filter simultaneously. This is load-bearing for the 'channel problem' framing and for contribution #2. A data-locked control (e.g., freezing the rollout policy for both arms, or feeding the RL arm the same top-25% graded batch, or an unfiltere
- [§5 with §C] The paper's own Appendix C lists candidate mechanisms for the RL channel's failure, including critic-optimism contamination through the bootstrapped return and update-magnitude interference. These are not merely alternative explanations; they are entangled with the recovery-race design itself, because the RL arm uses exactly the ungraded, bootstrapped objective while the BC arm uses realized actions from graded trajectories. The current protocol therefore cannot distinguish 'RL channel is unstable' from 'RL without the grading/filtering prior is unstable.' This weakens the strong conclusion stated at the end of §5 ('the instability is the RL channel') and in the abstract.
- [§6.2 and Table 2] The comparison between dream rehearsal and real-episode cloning is well designed in spirit, but the headline paired difference (+0.131, bootstrap CI [0.073, 0.238]) is computed on n=3 pairs. With complete seed separation the qualitative claim is credible, yet the bootstrap CI is not a strong inferential statement at this sample size. The authors already state this limitation in §9; I recommend reporting the three raw paired differences explicitly in the main text so readers can assess stability, rather than only the bootstrap aggregate.
minor comments (5)
- [§4.3] The abstract and §4.3 say the world model retains 'essentially everything we can measure,' but the same section reports a frozen-action-margin probe degrading to −0.25, indicating latent-space drift. The caveat that co-trained heads track the drift is fair, but the wording overstates the localization. Since the title and central claim hinge on 'the world model remembers,' the measurement basis should be stated more precisely: what is retained is replay-maintained head knowledge, not stable latent geometry.
- [§5 and Fig. 1] The caption says 'identical imagined data'; given the protocol, this should be revised to something like 'matched starts and same frozen world model' to avoid the implication of exactly equal trajectories.
- [§7, Eq. (1)] The definition of p_H is implicit (p_t = product_{k<t} c_k, so p_H uses the first H continuation probabilities). Consider writing p_0=1 explicitly, since the terminal bootstrap term γ^H p_H V(s_H) depends on it.
- [Fig. 2] The legend text 'per-seed points over bar means' is unclear; the figure seems to show per-seed points with means as bars, but this should be stated more directly.
- [§9 and Table 4] The 'audition-gated' task selection in §8 is described only informally. Please state the gate criterion (e.g., whether all candidate tasks had to reach 0.6 within the phase budget in a pilot run) so readers can gauge the selection pressure.
Circularity Check
Realized-first grading is validated against a label it is defined to maximize; the reported AUC 1.0 is by construction.
specific steps
-
self definitional
[§7, Eq. (1) and the offline-gauge paragraph]
"generate sibling dreams from banked starts, label each by its own imagined outcome, and measure each candidate scorer’s selection quality directly (AUC, top-quartile purity) [...] On the gauge, realized-first scores AUC 1.0 and top-quartile purity 1.0 on both task profiles"
The gauge labels each dream by its own imagined outcome, i.e. the realized-reward event. Eq. (1) gives a 10-point bonus iff realized > 0.3, so the score's primary ordering axis is exactly the gauge's label. Reporting AUC 1.0 / top-quartile purity 1.0 for realized-first on this gauge is therefore true by construction, not an independent measurement of selection quality. The same gauge was also used to develop the corrected rule ('we now consider part of the method'), so this validation is self-referential. The paper's other evidence (recovery referee, four-task re-run) is external, so the circularity is partial.
full rationale
Most of the paper is not circular. The §4 component probes are direct measurements on held checkpoints; the §5 recovery race compares two learning updates on a frozen world model with real-environment evaluation (though the 'identical imagined data' claim is undermined by online rollouts and the BC top-25% filter, a validity confound rather than a definitional reduction); the §6 chain results are external real-env pass/fail; and the realized-first rule has independent support in the recovery-referee and four-task re-run. The one clear circular step is the AUC/purity validation of Eq. (1): the gauge labels trajectories by realized imagined outcome and the scoring rule ranks by a 10-point indicator on exactly that quantity, so AUC 1.0 is baked into the construction. No load-bearing self-citations, imported uniqueness theorems, or smuggled ansätze are present; prior art for the self-imitation gate is explicitly attributed to Oh et al. The score reflects one partial, definitional validation embedded in an otherwise externally evaluated study.
Axiom & Free-Parameter Ledger
free parameters (5)
- realized-first reward threshold τ =
0.3
- clone fraction q =
0.25
- rehearsal updates per prior task K =
50 per 2,000-step chunk
- competence filter for real-episode cloning =
return > 0.05
- pass/recovery bars =
0.6 (chains), 0.75 (recovery)
axioms (4)
- domain assumption DreamerV3 at MiniGrid with a 17M-parameter world model is a faithful representative of replay-protected continual model-based RL.
- domain assumption The chosen probes (reward-head discrimination, critic means, continuation-head discrimination, frozen-action margin) are sufficient to conclude the world model 'remembers' old tasks.
- domain assumption Imagined rollouts from the world model are a faithful proxy for real old-task experience after the realized-first grading fix.
- domain assumption The audited MiniGrid task chains (including exclusion of tasks that cannot bootstrap) capture the relevant continual-learning failure mode.
Cite this review
Pith. "Pith review of The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL." pith.science (2026). https://pith.science/paper/ROW6UXUX
@misc{pith2026260719749,
author = {Pith},
title = {Pith review of: The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL},
year = {2026},
howpublished = {\url{https://pith.science/paper/ROW6UXUX}},
note = {Machine review of arXiv:2607.19749}
}
read the original abstract
Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.
Figures
Reference graph
Works this paper leans on
-
[1]
S. Kessler, M. Ostaszewski, M. Bortkiewicz, M. Żarski, M. Wołczyk, J. Parker-Holder, S. J. Roberts, and P. Miłoś. The effectiveness of world models for continual reinforcement learning.Conference on Lifelong Learning Agents (CoLLAs), 2023. arXiv:2211.15944
Pith/arXiv arXiv 2023
-
[2]
L. Yang, L. Kuhlmann, and G. Kowadlo. Augmenting replay in world models for continual reinforcement learning. arXiv:2401.16650, 2024
Pith/arXiv arXiv 2024
-
[3]
A. Alyahya, A. Al Siyabi, M. R. Ernst, L. Yang, L. Kuhlmann, and G. Kowadlo. ARROW: Augmented replay for robust world models. arXiv:2603.11395, 2026
Pith/arXiv arXiv 2026
-
[4]
M. K. Govind, D. Reilly, S. Patel, H. Le, and S. Das. World action models enable continual imitation learning with recurrent generative replays. arXiv:2606.27374, 2026
Pith/arXiv arXiv 2026
-
[5]
J. Oh, Y. Guo, S. Singh, and H. Lee. Self-imitation learning.International Conference on Machine Learning (ICML), 2018
2018
-
[6]
Wołczyk, B
M. Wołczyk, B. Cupiał, M. Ostaszewski, M. Bortkiewicz, M. Zając, R. Pascanu, Ł. Kuciński, and P. Miłoś. Fine-tuningreinforcementlearningmodelsissecretlyaforgettingmitigationproblem.InternationalConference on Machine Learning (ICML), 2024
2024
-
[7]
Rolnick, A
D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems (NeurIPS), 2019. 9
2019
-
[8]
Kirkpatrick et al
J. Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks.PNAS, 114(13), 2017
2017
-
[9]
A. A. Rusu et al. Progressive neural networks. arXiv:1606.04671, 2016
Pith/arXiv arXiv 2016
-
[10]
H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay.Advances in Neural Information Processing Systems (NeurIPS), 2017
2017
-
[11]
D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv:2301.04104, 2023
Pith/arXiv arXiv 2023
-
[12]
C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforcement learning. International Conference on Learning Representations (ICLR), 2022
2022
-
[13]
Nikishin, M
E. Nikishin, M. Schwarzer, P. D’Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning.International Conference on Machine Learning (ICML), 2022
2022
-
[14]
Chevalier-Boisvert et al
M. Chevalier-Boisvert et al. Minigrid & Miniworld: modular & customizable reinforcement learning environments for goal-oriented tasks.NeurIPS Datasets and Benchmarks, 2023. A Protocol Details and Hyperparameters Observation64×64×3RGB, partial egocentric view World model 17M parameters (CNN encoder, RSSM, reward/continuation/decoder heads) Actor 1.8M param...
2023
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.