pith. sign in

arxiv: 2605.22454 · v1 · pith:HRQIYAXBnew · submitted 2026-05-21 · 💻 cs.LG · cs.AI

Don't Forget the Critic: Value-Based Data Rehearsal for Multi-Cyclic Continual Reinforcement Learning

Benjamin Poole , Andrew Quinn , Li Yang , Minwoo Lee This is my paper

Pith reviewed 2026-05-22 07:10 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords continual reinforcement learningcatastrophic forgettingdata rehearsaldeep Q-networksvalue function approximationmulti-cyclic environmentsQ-value regularization
0
0 comments X

The pith

Value-based data rehearsal with continuous updates and immediate regularization improves continual reinforcement learning for critics.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing continual reinforcement learning methods that use data rehearsal have focused on policy gradients and avoided regularizing critics because it previously hurt performance. This paper tests rehearsal directly on Deep Q-Networks inside multi-cyclic task sequences that repeat and therefore stress forgetting more than one-pass evaluations. The authors add two changes: continuous collection and updating of stored Q-values throughout training, plus immediate application of the regularization instead of waiting until after the first task. These changes produce better learning speed, less forgetting, and stronger transfer of knowledge across task repetitions.

Core claim

In multi-cyclic continual reinforcement learning, Q-value regularization applied to Deep Q-Networks through data rehearsal can be made effective by collecting and refreshing the replay buffer continuously and by starting the regularization without delay after the first task; the resulting method, Qreg+NWLU, improves learning efficiency, reduces forgetting, and increases knowledge transfer relative to plain Qreg and standard CRL baselines.

What carries the argument

Qreg+NWLU, which performs continuous data rehearsal to keep stored Q-values current and applies No-Wait regularization immediately rather than after the first task.

If this is right

  • Value-function methods can now use rehearsal without the degradation that previously confined rehearsal to actor-only methods.
  • Multi-cyclic evaluations reveal forgetting patterns that single-cycle tests miss, so future CRL benchmarks should include repeated task sequences.
  • Knowledge accumulated in early cycles can be retained and reused more effectively when rehearsal is kept active and applied at once.
  • The approach stays inside standard DQN training loops, so it can be added to existing value-based agents with modest storage overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same continuous-rehearsal idea could be tested on other value-based algorithms such as distributional RL or actor-critic hybrids that also maintain critics.
  • In robotics or game environments where the same skills recur after long gaps, the multi-cyclic regime may better predict long-term agent stability than current one-shot continual-learning suites.
  • Because the method keeps the replay buffer size fixed while refreshing values, it may scale to longer task sequences without linear growth in memory.
  • An open question left by the work is whether the same immediate and continuous rehearsal rules would help when tasks are not exact repetitions but share partial structure.

Load-bearing premise

That regularizing the critic with rehearsal will avoid the performance drops seen in earlier critic-regularization attempts and that the multi-cyclic test setting reflects the forgetting pressures that appear in real repeated-task environments.

What would settle it

A direct comparison experiment in which Qreg+NWLU shows no gain in average return or no reduction in forgetting rate over plain Qreg across several multi-cyclic task sequences would refute the benefit of the two modifications.

Figures

Figures reproduced from arXiv: 2605.22454 by Andrew Quinn, Benjamin Poole, Li Yang, Minwoo Lee.

Figure 1
Figure 1. Figure 1: Conventional value-based CRL methods struggle to adapt or suffer catastrophic forgetting in multi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Learning curves for the total return G averaged over 10 seeds and smoothed with a moving average. The color shaded regions for each line report the standard error. The rows correspond to the task sequence, while the columns correspond to the individual tasks. The gray-shaded areas indicate the training periods for a particular task. Tomilin et al. (2023). Specifically, PackNet, EWC, and L2 are included on … view at source ↗
Figure 3
Figure 3. Figure 3: Ablation learning curves for the total return [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Q-value norm comparisons between Qreg (left) and Qreg+NWLU (right) across all three tasks [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example of environments form Minihack Room task sequence. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of PLE environments, Flappy and Catcher. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Learning curves of Qreg+NWLU total return [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Learning curves of Qreg+NWLU total return [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Learning curves of Qreg+NWLU total return [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Room comparison of Q-value norms between Qreg variants. Color indicates each unique run. [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Flappy comparison of Q-value norms between Qreg variants. Color indicates each unique run. [PITH_FULL_IMAGE:figures/full_fig_p028_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Catcher comparison of Q-value norms between Qreg variants. Color indicates each unique run. [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
read the original abstract

Data rehearsal has emerged as a leading approach for mitigating catastrophic forgetting in Continual Reinforcement Learning (CRL). However, existing work remains confined to policy gradient frameworks, regularizing only actors due to the performance degradation incurred by critic regularization. This actor-centric approach overlooks the potential of data rehearsal for value function approximation. Moreover, existing evaluations in CRL rarely consider multi-cyclic environments where task sequences repeat, a critical real-world scenario that exacerbates forgetting and plasticity. We investigate data rehearsal for Deep Q-Networks using Q-value regularization in multi-cyclic settings and propose Qreg+NWLU which introduces two simple modifications: (1) continuous data rehearsal that dynamically collects and updates stored Q-values throughout training, and (2) "No-Wait" regularization that applies immediately rather than after the first task. Together, these modifications yield improvements in learning efficiency, forgetting mitigation, and knowledge transfer over Qreg and conventional CRL methods within value function approximation settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes extending data rehearsal to value-based continual reinforcement learning by applying Q-value regularization to Deep Q-Networks in multi-cyclic task sequences. It identifies that prior work avoided critic regularization due to performance degradation and introduces Qreg+NWLU, which adds continuous dynamic collection and updating of stored Q-values plus immediate 'No-Wait' regularization rather than delayed application. The authors claim these two modifications produce gains in learning efficiency, forgetting mitigation, and knowledge transfer relative to standard Qreg and other CRL baselines within value-function approximation settings.

Significance. If the empirical claims are substantiated, the work would meaningfully address a documented gap in CRL by showing that data rehearsal can be made viable for critics rather than being restricted to actors. The multi-cyclic evaluation setting is a clear strength, as it models repeated exposure to tasks that standard single-sequence benchmarks overlook. The proposed modifications are simple enough that, if they prove robust, they could be adopted as a lightweight baseline for value-based continual RL.

major comments (2)
  1. [Abstract] Abstract: the headline claim that dynamic Q-value collection and No-Wait regularization together eliminate the performance degradation previously observed with critic regularization is load-bearing, yet the abstract provides neither a mechanistic explanation nor an ablation isolating why outdated stored Q-values or delayed application were the root cause. Without this, it remains unclear whether the modifications address value-function approximation error accumulation or Bellman-target interference under repeated cycling.
  2. [Experiments] The multi-cyclic experimental protocol is presented as more realistic, but the manuscript does not report whether the observed gains persist when the number of cycles increases or when task boundaries are not explicitly signaled; this directly affects the forgetting-mitigation claim.
minor comments (2)
  1. [Abstract] The acronym NWLU is introduced in the title and abstract without expansion on first use.
  2. [Method] Notation for the stored Q-value buffer and its update rule should be introduced with a clear equation or pseudocode to avoid ambiguity when comparing to standard experience replay.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We are grateful to the referee for the careful review and valuable suggestions. We respond to each major comment in turn and outline the revisions we intend to make to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline claim that dynamic Q-value collection and No-Wait regularization together eliminate the performance degradation previously observed with critic regularization is load-bearing, yet the abstract provides neither a mechanistic explanation nor an ablation isolating why outdated stored Q-values or delayed application were the root cause. Without this, it remains unclear whether the modifications address value-function approximation error accumulation or Bellman-target interference under repeated cycling.

    Authors: We agree that a concise mechanistic explanation in the abstract would improve clarity. The paper details that continuously updating stored Q-values prevents the use of stale targets that accumulate errors in the value function approximator, while immediate No-Wait regularization allows for ongoing knowledge transfer across cycles rather than postponing it. We will revise the abstract to incorporate a short statement on these mechanisms. Furthermore, we will include an additional ablation in the experiments section of the revised version to isolate the impact of each modification on mitigating approximation errors and interference. revision: yes

  2. Referee: [Experiments] The multi-cyclic experimental protocol is presented as more realistic, but the manuscript does not report whether the observed gains persist when the number of cycles increases or when task boundaries are not explicitly signaled; this directly affects the forgetting-mitigation claim.

    Authors: We recognize the importance of evaluating robustness under extended cycling and implicit task detection. Our current results are based on a multi-cyclic setup with explicit boundaries and a moderate number of cycles, which already reveals clear benefits in forgetting mitigation compared to baselines. While we have not exhaustively tested higher cycle counts or boundary-free scenarios, we will add a paragraph in the discussion section addressing these as limitations and outlining how the proposed approach could be extended with online boundary detection methods. This will help contextualize the forgetting-mitigation claims. revision: partial

Circularity Check

0 steps flagged

No significant circularity in empirical proposal

full rationale

The paper is an empirical proposal for value-based data rehearsal in multi-cyclic continual RL, introducing two practical modifications (continuous Q-value updates and immediate No-Wait regularization) to an existing Qreg baseline and reporting experimental gains in efficiency, forgetting mitigation, and transfer. No mathematical derivation chain, equations, or first-principles results are presented that could reduce claimed outcomes to fitted parameters or self-citations by construction. The central claims rest on experimental comparisons rather than any closed logical loop, and the work remains self-contained against external benchmarks without invoking load-bearing self-citations for uniqueness or ansatz justification.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard RL assumptions and the empirical claim that the two modifications improve outcomes; no new entities or free parameters are introduced in the abstract.

axioms (1)
  • domain assumption Markov decision process formulation and standard DQN training dynamics hold in the multi-cyclic setting.
    Implicit in the use of Deep Q-Networks and data rehearsal for value approximation.

pith-pipeline@v0.9.0 · 5698 in / 1169 out tokens · 36282 ms · 2026-05-22T07:10:54.096650+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    International Conference on Machine Learning , year =

    Dueling Network Architectures for Deep Reinforcement Learning , author =. International Conference on Machine Learning , year =

    work page

  2. [2]

    Rainbow: Combining Improvements in Deep Reinforcement Learning , journal=

    Hessel, Matteo and Modayil, Joseph and van Hasselt, Hado and Schaul, Tom and Ostrovski, Georg and Dabney, Will and Horgan, Dan and Piot, Bilal and Azar, Mohammad and Silver, David , year=. Rainbow: Combining Improvements in Deep Reinforcement Learning , journal=

    work page

  3. [3]

    Deep Reinforcement Learning with Double Q-Learning , journal=

    van Hasselt, Hado and Guez, Arthur and Silver, David , year=. Deep Reinforcement Learning with Double Q-Learning , journal=

    work page

  4. [4]

    Nature , year =

    Loss of plasticity in deep continual learning , author =. Nature , year =

    work page

  5. [5]

    2025 , author =

    Continual learning in the presence of repetition , journal =. 2025 , author =

    work page 2025

  6. [6]

    International Conference on Machine Learning , year=

    Reinforcement Learning with Deep Energy-Based Policies , author=. International Conference on Machine Learning , year=

    work page

  7. [7]

    Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline , year =

    Ahn, Hongjoon and Hyeon, Jinu and Oh, Youngmin and Hwang, Bosun and Moon, Taesup , booktitle =. Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline , year =

    work page

  8. [8]

    International Conference on Machine Learning , pages =

    The Primacy Bias in Deep Reinforcement Learning , author =. International Conference on Machine Learning , pages =. 2022 , editor =

    work page 2022

  9. [9]

    arXiv:1910.07207 [cs, stat] , author =

    Soft. arXiv:1910.07207 [cs, stat] , author =

    work page arXiv 1910

  10. [10]

    Conference on Lifelong Learning Agents , year =

    CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents , author =. Conference on Lifelong Learning Agents , year =

    work page

  11. [11]

    International Conference on Learning Representations , year=

    Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference , author=. International Conference on Learning Representations , year=

    work page

  12. [12]

    Advances in Neural Information Processing Systems , author =

    work page

  13. [13]

    2021 , eprint=

    Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning , author=. 2021 , eprint=

    work page 2021

  14. [14]

    Advances in Neural Information Processing Systems , title =

    Wo. Advances in Neural Information Processing Systems , title =

    work page

  15. [15]

    , year =

    Kessler, Samuel and Parker-Holder, Jack and Ball, Philip and Zohren, Stefan and Roberts, Stephen J. , year =. Same

    work page

  16. [16]

    Conference on Computer Vision and Pattern Recognition , year =

    Mallya, Arun and Lazebnik, Svetlana , title =. Conference on Computer Vision and Pattern Recognition , year =

    work page

  17. [17]

    A Definition of Continual Reinforcement Learning , year =

    Abel, David and Barreto, Andre and Van Roy, Benjamin and Precup, Doina and van Hasselt, Hado P and Singh, Satinder , booktitle =. A Definition of Continual Reinforcement Learning , year =

    work page

  18. [18]

    , year =

    Abbas, Zaheer and Zhao, Rosie and Modayil, Joseph and White, Adam and Machado, Marlos C. , year =. Loss of

    work page

  19. [19]

    Advances in Neural Information Processing Systems , author =

    Prediction and. Advances in Neural Information Processing Systems , author =

    work page

  20. [20]

    Nature , author =

    Human-level control through deep reinforcement learning , copyright =. Nature , author =

    work page

  21. [21]

    Proceedings of the National Academy of Sciences , author =

    Overcoming catastrophic forgetting in neural networks , urldate =. Proceedings of the National Academy of Sciences , author =

    work page

  22. [22]

    Arslan Chaudhry and Marcus Rohrbach and Mohamed Elhoseiny and Thalaiyasingam Ajanthan and Puneet Kumar Dokania and Philip H. S. Torr and Marc'Aurelio Ranzato , title =. CoRR , year =. 1902.10486 , biburl =

    work page internal anchor Pith review Pith/arXiv arXiv 1902

  23. [23]

    Experience

    Rolnick, David and Ahuja, Arun and Schwarz, Jonathan and Lillicrap, Timothy and Wayne, Gregory , year =. Experience. Advances in

    work page

  24. [24]

    2016 , journal =

    Tasfi, Norman , title =. 2016 , journal =

    work page 2016

  25. [25]

    Advances in Neural Information Processing Systems , year=

    MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research , author=. Advances in Neural Information Processing Systems , year=

    work page

  26. [26]

    Selective Experience Replay for Lifelong Learning , booktitle=

    Isele, David and Cosgun, Akansel , year=. Selective Experience Replay for Lifelong Learning , booktitle=

    work page

  27. [27]

    Memory Aware Synapses: Learning What (not) to Forget

    Aljundi, Rahaf and Babiloni, Francesca and Elhoseiny, Mohamed and Rohrbach, Marcus and Tuytelaars, Tinne. Memory Aware Synapses: Learning What (not) to Forget. Computer Vision. 2018

    work page 2018

  28. [28]

    International Conference on Learning Representations , year=

    Variational Continual Learning , author=. International Conference on Learning Representations , year=

    work page

  29. [29]

    Online Distillation With Continual Learning for Cyclic Domain Shifts , booktitle =

    Houyon, Joachim and Cioppa, Anthony and Ghunaim, Yasir and Alfarra, Motasem and Halin, Ana. Online Distillation With Continual Learning for Cyclic Domain Shifts , booktitle =

    work page