Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Continual Reinforcement Learning by Planning with Online World Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that learning a sparse Follow-The-Leader world model online — with ridge regularization and closed-form updates — is no-regret, so planning with the latest model solves each new task while keeping old skills…

desk verdict The benchmark and empirical results are solid, but the advertised no-forgetting guarantee does not cover the actual experimental setting; the paper needs major revision before it earns its abstract. read the letter →

arxiv 2507.09177 v1 pith:KT65RND5 submitted 2025-07-12 cs.LG cs.AIstat.ML

Zichen Liu , Guoji Fu , Chao Du , Wee Sun Lee , Min Lin This is my paper · ORCID
classification cs.LGcs.AIstat.ML MSC 68T0568W27
keywords continualreinforcementlearningcatastrophicforgettingonlineworldmodelsFollow-The-LeadermodelpredictivecontrolregretboundssparsefeatureencodingBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that catastrophic forgetting in continual reinforcement learning can be avoided without replay buffers, task IDs, or per-task networks, by learning one shared world model online and planning with it. The world model is a sparse random-feature network updated in closed form by a Follow-The-Leader rule, so each new experience enters a running least-squares solution instead of perturbing a trained deep network. The paper proves this updater is no-regret, with regret bounded by $\mathcal{O}(\sqrt{K^2D\log(T)})$ under its assumptions, which is what makes the model 'immune to forgetting by construction.' Actions come from a model predictive controller (CEM planner) that re-plans with the current model for whatever reward is presented, so a new task is just a new reward function. On the paper's new Continual Bench environment, this Online Agent matches a perfect-memory deep agent on average performance and achieves lower cumulative regret than deep-world-model baselines with regularization, replay, or architecture-based continual learning techniques.

What carries the argument

The load-bearing object is the sparse Follow-The-Leader world model: $y = W\sigma(Px)$ with a fixed random projection $P$, a localized sparse feature map $\phi$, and weights updated in closed form by Eq. (4), $W_s^{(t)} = (A_{ss}^{(t-1)} + \tfrac{1}{\lambda}I)^{-1}(B_s^{(t-1)} - A_{ss}^{(t-1)} W_s^{(t-1)})$. The ridge term $\tfrac{1}{\lambda}I$ is what gives each local update a unique minimizer and lets the proof bound the gap between the sparse update and the dense least-squares solution, while the sparse activation set $s$ keeps the per-step update cost constant in $K$. A CEM planner with shift initialization, colored noise, and memory converts the model's predictions into actions, so the same learned dynamics serve every task.

What would settle it

Train OA on Continual Bench with the task order reversed and record the world model's prediction loss on held-out data from earlier tasks; if the loss on old-task data rises after a task switch beyond the $\mathcal{O}(\log T)$ regret drift, the 'immune to forgetting by construction' claim would be contradicted. Alternatively, compute the Assumption 1 quantity at each switch and check whether the sup norm frequently exceeds $1/(\lambda t)$, which would mean Theorem 1's bound is not in force.

Watch

Extended reading notes

Core claim

The central claim is that a single agent can continually solve sequentially presented tasks by maintaining an online Follow-The-Leader world model and never retraining on past data. The paper formalizes this as Theorem 1: under Assumptions 1--3, the sparse updater of Eq. (4) satisfies $\mathrm{Regret}(T) \le c_W c_y \sqrt{20(K^2 r_M^2 + 1)D(\log T + 1)}$, meaning the cumulative prediction loss of the sparse model stays within $\mathcal{O}(\sqrt{K^2D\log T})$ of the best linear predictor in hindsight. The paper also argues that the planner, which only sees the latest model, does not need task boundaries, because the task dependence lives in the externally provided reward function, not in the learned dynamics. Empirically, OA keeps high success on all previously seen tasks in Continual Bench while learning new ones, matching Perfect Memory and surpassing deep model-based agents with SI, Coreset, and PackNet-style continual learning techniques.

Load-bearing premise

The proof's load-bearing assumption is that every new state-action pair is, in feature space, close to the average of all previously seen pairs (within $1/(\lambda t)$); switching tasks is precisely when that is most likely to fail, and the paper concedes the assumption holds only when new exploration stays near old data.

Editorial extensions

If this is right

  • A continual agent can be built with a single shared component — the world model — that updates every step at constant cost, instead of replaying old data to convergence.
  • With a no-regret world model, each new task is solvable by planning with the current model and the new reward function, so the agent never needs task IDs or task-specific heads.
  • On Continual Bench, OA keeps success on all six previously seen tasks at the level of a perfect-memory agent while deep baselines degrade, and its average performance grows as tasks accumulate.
  • The sparse model's utilization rises as tasks accumulate but its update cost stays fixed, so capacity does not exhaust through the iterative-learning blind spots that subspace-based continual learning methods hit.
  • The benchmark design shows that spatial arrangement, not just temporal concatenation, determines whether a unified dynamics exists and whether forgetting and transfer can be studied at all.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Assumption 1 holds broadly enough, the closed-form update strategy transfers to any sequential prediction problem where dynamics are shared and only goals change, including online system identification outside reinforcement learning.
  • The deterministic FTL model carries no uncertainty estimate; coupling it with calibrated uncertainty and explicit exploration would be a direct extension that the paper itself identifies as future work.
  • Continual Bench is episodic with explicit task switches; a reset-free version with irreversible states would stress-test whether the no-forgetting property persists when tasks blend together without boundaries.
  • A simple empirical check would be to log $\sup_x \|\phi(x)\phi(x)^\top - \tfrac{1}{t}\sum_i \phi(x_i)\phi(x_i)^\top\|$ at each task boundary; if it frequently exceeds $1/(\lambda t)$, the conditions for Theorem 1 are violated even if the practical algorithm still performs well.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an Online Agent (OA) for continual reinforcement learning. OA learns a sparse, randomly featured, ridge-regularized linear world model with a closed-form Follow-The-Leader update, and acts by CEM/MPC planning using this model. The authors claim that the sparse online update is no-regret, with a bound of O(sqrt(K^2 D log T)) under assumptions stated as mild, and hence that the world model is immune to forgetting by construction. To evaluate OA, they introduce Continual Bench, a lightweight Mujoco-based benchmark with six Meta-World-style tasks arranged so that a unified dynamics exists across tasks. On this benchmark, OA is reported to maintain performance on all previously seen tasks, matching a Perfect Memory deep-model baseline and outperforming deep-model agents with EWC/SI, Coreset, and fine-tuning.

Significance. If the theoretical claim held as stated, the paper would make a significant contribution: an online, replay-free world model with a logarithmic regret guarantee would be a strong building block for continual model-based RL, and the proposed benchmark would fill a practical gap in CRL evaluation. The empirical design has real strengths: the benchmark is open-sourced, experiments use 7 seeds, baselines share the same planner and hyperparameters, and OA's non-forgetting behavior on Continual Bench is visually and numerically consistent. However, the theoretical guarantee is currently the central advertised contribution, and both the assumptions and the proof have load-bearing problems; the empirical results alone, while promising, do not establish the paper's headline claim of a proven no-regret world model under mild assumptions.

major comments (3)
  1. [4.2, Eq. (5), Remark 1] Assumption 1 is not mild and is likely violated exactly in the CRL setting. For a nonlinear feature map on a continuous input space, the supremum over all x of ||phi(x)phi(x)^T - (1/t) sum_i phi(x_i)phi(x_i)^T||_2 does not generally decay as 1/(lambda t); if phi has sparse, roughly disjoint supports for distant inputs, the norm of a new rank-one term can remain bounded away from zero. For large t the right-hand side goes to zero, which would force every possible feature outer product to coincide with the empirical mean. Remark 1 itself concedes that the assumption holds only if new inputs are explored within a 1/(lambda t)-neighborhood of observed data, a ball that shrinks to radius zero. Task switches in continual RL are precisely the moments when newly visited state-action pairs are far from previous data, so Theorem 1 does not apply to the Continual Bench experiments or to the general CRL setting claimed in the abstract.
  2. [Appendix E.2, Lemma 2; Appendix E.3, Proposition 3] The proof of Theorem 1 uses the Loewner semidefinite order (⪯) on matrices that are not symmetric and not square. For example, Lemma 2 claims (1 - 1/t) W(t) + Delta_t ⪯ W(t+1) ⪯ W(t) + Delta_t, where W(t), W(t+1), and Delta_t are D x S matrices; the positive-semidefinite order is undefined for nonsymmetric rectangular matrices. The same issue appears in Lemma 3, Proposition 2, and Proposition 3, where expressions such as W(t+1) - fW(t+1) ⪯ K r_M Delta_t are written as if they were PSD inequalities. Since these inequalities are then used to bound the per-step losses and the final regret, the formal proof as written does not establish Theorem 1.
  3. [4.2, Assumption 3, Eq. (8)] Assumption 3 is a data-dependent lower bound on K that involves A^{(t)}_{ss}, the feature values at x_t, and y_t, yet K is a fixed architectural hyperparameter in the experiments (Appendix A.3 fixes Lambda and hence the activation ratio). The paper says 'we choose K such that Assumption 3 holds,' but the quantities in the bound are not known before the data are observed, and no verification of Assumption 3 is provided for Continual Bench. Consequently, the proof that the sparse update in Eq. (4) approximates the exact FTL solution, which is essential to the 'no-forgetting by construction' claim, is not connected to the configurations actually evaluated.
minor comments (5)
  1. [Appendix E.1, Lemma 1] The notation for A^{(t)} is inconsistent: it is introduced as Phi_{t-1}^T Phi_{t-1}, but Eq. (4) and the surrounding text index A by the current time step as if the new point has already been incorporated; please clarify whether A^{(t)} includes phi(x_t).
  2. [Appendix E.3.1, Proposition 2] The derivation of the constant K' contains several unclear steps, including the placement of phi(x_t)^T phi(x_t) and the manipulation of y_t^T y_t as scalar factors; these steps should be rewritten carefully because the resulting bound feeds into Proposition 3.
  3. [6.3 and Appendix A.5] The claim that OA achieves Perfect Memory's performance 'by a much more efficient online update' would be stronger with a direct computation-time or per-step-cost comparison; the current text compares update mechanisms conceptually but does not measure wall-clock time or total optimization cost.
  4. [Figure 6(c)] The x-axis labels for the buffer-size ablation are difficult to read; please reformat the ticks so that the buffer sizes (1, 5000, 10000, 20000, 141946) are legible.
  5. [Remark 1] Remark 1 should explicitly acknowledge that the stated sufficient condition for Assumption 1 (exploration within a shrinking 1/(lambda t) ball) is incompatible with the distributional shift induced by switching tasks; leaving this as a passing remark obscures the strength of the assumption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the regret bound is a genuine FTL-style derivation under stated assumptions, and the benchmark design is an explicit matching of assumptions rather than a fitted prediction.

full rationale

The paper's core derivation is self-contained as a mathematical argument. The sparse update Eq. (4) is shown to approximate the full FTL ridge-regression solution Eq. (3), and Theorem 1 is a regret bound derived from the stated Assumptions 1-3 via a sparse-versus-dense gap bound (Proposition 3) plus a standard FTL regret bound (Corollary 1). None of these assumptions defines the conclusion: Assumption 1 is a feature-covariance stabilization condition, Assumption 2 is a boundedness condition, and Assumption 3 is a lower bound on the number of active features K. The proof does not assume the no-forgetting property or the desired regret rate. The Continual Bench environment is deliberately constructed so that a unified dynamics exists across tasks; this is an experimental design that matches the paper's stated CRL formulation, not a prediction extracted from fitted values. The only notable self-citation is the sparse feature encoder taken from prior work by overlapping authors; while it is an important component of the agent, the paper does not use that citation to justify Theorem 1, and the theorem's derivation is independent of the encoder's empirical quality. The strongest challenge to the paper is a correctness and validity concern rather than a circularity: Remark 1 concedes that Assumption 1 holds only when new points are explored near the observed data, which may fail exactly at task boundaries, so Theorem 1 may not apply to the Continual Bench experiments. That is a potentially violated assumption about applicability, not an input-output equivalence. No fitted parameter is renamed as a prediction, and no load-bearing claim reduces by construction to its own input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The empirical performance depends on the hand-chosen hyperparameters lambda and Lambda, on the unified-dynamics assumption, and on the technical Assumptions 1-3 for the regret bound. No new physical entities are introduced; the method reuses random features and FTL least squares from the authors' prior work.

free parameters (4)
  • lambda (ridge regularization) = 1/lambda = 0.005
    Regularization strength in Eqs. (3)-(4); chosen as a good regularization strength (Appendix A.3). The theory requires lambda >= 1.
  • Lambda (Losse bins per feature) = 9
    Sparsity level of the random feature encoder; swept over {5,7,9,11} and set to 9 (Appendix A.3).
  • Number of Losse features = 300
    Width of the random feature map, chosen by hand (Appendix A.3).
  • CEM planner hyperparameters = N=150, H=15, K=3 iterations, elite ratio 0.1
    Planner settings (Appendix A.3); kept identical for all baseline agents for fair comparison.
assumptions (5)
  • domain assumption Tasks share a unified world dynamics P^u over a consistent state-action space (Section 3.1)
    The CRL formulation defines M_tau = (S,A,P^u,R_tau,gamma,rho0_tau) with a single P^u; the method and benchmark depend on this assumption.
  • ad hoc to paper Assumption 1: sup_x ||phi(x)phi(x)^T - (1/t)sum_{i<=t} phi(x_i)phi(x_i)^T||_2 <= 1/(lambda t) (Eq. 5)
    Requires new features to stay within 1/(lambda t) of the empirical covariance; likely violated under task-switch distributional shift, as Remark 1 concedes.
  • domain assumption Assumption 2: ||y_t|| <= c_y, y_t,i >= 0, ||W||_F <= c_W
    Boundedness is mild, but nonnegativity of state differences is restrictive for general control.
  • ad hoc to paper Assumption 3: sparsity K satisfies a data-dependent lower bound (Eq. 8)
    Defines K in terms of data-dependent spectral norms; in practice K is a fixed small number, which may not satisfy the assumption.
  • standard math Random features provide universal approximation (Huang et al., 2006; Losse encoding from Liu et al., 2024)
    Background result justified the shallow network architecture; cited to prior work without re-proving it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Reinforcement Learning by Planning with Online World Models." pith.science (2026). https://pith.science/paper/KT65RND5

@misc{pith2026250709177,
  author       = {Pith},
  title        = {Pith review of: Continual Reinforcement Learning by Planning with Online World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KT65RND5}},
  note         = {Machine review of arXiv:2507.09177}
}
abstract

Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of $\mathcal{O}(\sqrt{K^2D\log(T)})$ under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment for CRL, and compare with several strong baselines under the same model-planning algorithmic framework. The empirical results show that OA learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.

Figures

Figures reproduced from arXiv: 2507.09177 by the authors.

Figure 1
Figure 1. A motivating example comparing task-indexed dynam￾ics (P τ ) with unified dynamics (P u ). Under the task-indexed viewpoint, the state space of each task is the box’s location x ∈ R while the action space is {“apply force rightwards", “apply force leftwards"}. Clearly, for the same action “apply force rightwards" (denoted as red arrows) applied at the same state xt, the dynamics P τ is conflicting to each other. Hen… view at source ↗
Figure 2
Figure 2. Potential physical conflicts between two consecutive tasks in Continual-World (Wołczyk et al., 2021). The handles of the door and the drawer follow different trajectory when being opened. named Continual-World, where they select a sequence of tasks from Meta-World (Yu et al., 2019) and run experi￾ments using soft actor-critic (SAC) (Haarnoja et al., 2018) with various continual learning techniques. This bench￾mark h… view at source ↗
Figure 3
Figure 3. The Continual Bench environment consists of 6 tasks with diverse difficulty levels. All tasks share the same unified dynamics. tributional shift in state-action visitation when switching tasks. Different tasks also share meaningful overlapping, allowing us to study transfer, though it is not the focus of this work. We open source the code of Continual Bench1 and hope this realistic but lightweight environment can ac… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Performance comparison of our agent OA with different deep model-based planning agents on Continual Bench environment. Results are aggregated from 7 runs with different seeds. The solid lines show the means and shaded areas are the standard errors. The grey regions den…
Figure 6
Figure 6. Figure 6: (a) The average performance curves of different methods. (b) Ratio of the activated weights of the OA sparse world model. (c) Final agent performance with different buffer budgets. (d) Ablation on the world model sparsity [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: OA learning and acting loop. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Curves of the world model errors. Same as the performance measure in the main text, results are aggregated from 7 runs with different seeds. The solid lines show the means and shaded areas are the standard errors. The grey regions denote the learning period for the cur…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.

Reference graph

Works this paper leans on

61 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, 2023

    work page 2023

  3. [3]

    P., and Singh, S

    Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H. P., and Singh, S. A definition of continual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  4. [4]

    Gradient based sample selection for online continual learning

    Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradient based sample selection for online continual learning. Neural Information Processing Systems, 2019 a

    work page 2019

  5. [5]

    Selfless sequential learning

    Aljundi, R., Rohrbach, M., and Tuytelaars, T. Selfless sequential learning. International Conference on Learning Representations, 2019 b

    work page 2019

  6. [6]

    G., Naddaf, Y., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013

    work page 2013

  7. [7]

    A markovian decision process

    Bellman, R. A markovian decision process. Journal of mathematics and mechanics, 1957

    work page 1957

  8. [8]

    Class-incremental continual learning into the extended der-verse

    Boschini, M., Bonicelli, L., Buzzega, P., Porrello, A., and Calderara, S. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    work page 2022

Show all 61 references
  1. [9]

    Efficient lifelong learning with A-GEM

    Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM . In International Conference on Learning Representations, 2019 a

    2019

  2. [10]

    Continual learning with tiny episodic memories

    Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., and Ranzato, M. Continual learning with tiny episodic memories. In Workshop on Multi-Task and Lifelong Reinforcement Learning at ICML, 2019 b

    2019

  3. [11]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018

    2018

  4. [12]

    Leveraging procedural generation to benchmark reinforcement learning

    Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Leveraging procedural generation to benchmark reinforcement learning. International Conference on Machine Learning, 2020

    2020

  5. [13]

    P., Mannor, S., and Rubinstein, R

    De Boer, P.-T., Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A tutorial on the cross-entropy method. Annals of operations research, 2005

    2005

  6. [14]

    Orthogonal gradient descent for continual learning

    Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogonal gradient descent for continual learning. In Chiappa, S. and Calandra, R. (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, 2020

    2020

  7. [15]

    E., Prett, D

    Garcia, C. E., Prett, D. M., and Morari, M. Model predictive control: Theory and practice—a survey. Automatica, 1989

    1989

  8. [16]

    Garcia, F. M. and Thomas, P. A meta-mdp approach to exploration for lifelong reinforcement learning. Neural Information Processing Systems, 2019

    2019

  9. [17]

    Building a subspace of policies for scalable continual learning

    Gaya, J.-B., Doan, T., Caccia, L., Soulier, L., Denoyer, L., and Raileanu, R. Building a subspace of policies for scalable continual learning. International Conference on Learning Representations, 2023

    2023

  10. [18]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018

    2018

  11. [19]

    K., et al

    Huang, G.-B., Chen, L., Siew, C. K., et al. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Networks, 2006

    2006

  12. [20]

    Continual model-based reinforcement learning with hypernetworks

    Huang, Y., Xie, K., Bharadhwaj, H., and Shkurti, F. Continual model-based reinforcement learning with hypernetworks. In IEEE International Conference on Robotics and Automation, 2021

    2021

  13. [21]

    A theory of universal artificial intelligence based on algorithmic complexity

    Hutter, M. A theory of universal artificial intelligence based on algorithmic complexity. arXiv preprint arXiv: cs.0004001, 2000

    2000

  14. [22]

    and Cosgun, A

    Isele, D. and Cosgun, A. Selective experience replay for lifelong learning. AAAI Conference on Artificial Intelligence, 2018

    2018

  15. [23]

    Johnson, W. B. and Lindenstrauss, J. Extensions of lipschitz mappings into a hilbert space. Contemporary Mathematics, 1984

    1984

  16. [24]

    R., Hwang, S

    Kang, H., Yoon, J., Madjid, S. R., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with soft-winning subnetworks. International Conference on Machine Learning, 2022

    2022

  17. [25]

    J., Zohren, S., and Roberts, S

    Kessler, S., Parker-Holder, J., Ball, P. J., Zohren, S., and Roberts, S. J. Same state, different task: Continual reinforcement learning without interference. AAAI Conference on Artificial Intelligence, 2022

    2022

  18. [26]

    Kessler, S., Ostaszewski, M., Bortkiewicz, M. P., \. Z arski , M., Wolczyk, M., Parker-Holder, J., Roberts, S. J., and Milo \'s , P. The effectiveness of world models for continual reinforcement learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, 2023

    2023

  19. [27]

    Towards continual reinforcement learning: A review and perspectives

    Khetarpal, K., Riemer, M., Rish, I., and Precup, D. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 2022

    2022

  20. [28]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  21. [29]

    A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al

    Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017

    2017

  22. [30]

    Ai2-thor: An interactive 3d environment for visual ai

    Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017

    2017 arXiv

  23. [31]

    u ttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rockt \

    K \"u ttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rockt \"a schel, T. The nethack learning environment. Advances in Neural Information Processing Systems, 2020

    2020

  24. [32]

    S., and Lin, M

    Liu, Z., Du, C., Lee, W. S., and Lin, M. Locality sensitive sparse encoding for learning world models online. In International Conference on Learning Representations, 2024

    2024

  25. [33]

    and Lazebnik, S

    Mallya, A. and Lazebnik, S. Packnet: Adding multiple tasks to a single network by iterative pruning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

    2018

  26. [34]

    Deep online learning via meta-learning: Continual adaptation for model-based rl

    Nagabandi, A., Finn, C., and Levine, S. Deep online learning via meta-learning: Continual adaptation for model-based rl. International Conference on Learning Representations, 2018

    2018

  27. [35]

    R., De Schutter, B., Wiering, M

    Negenborn, R. R., De Schutter, B., Wiering, M. A., and Hellendoorn, H. Learning-based model predictive control for markov decision processes. IFAC Proceedings Volumes, 2005

    2005

  28. [36]

    and Vidal, R

    Peng, L. and Vidal, R. Block coordinate descent on smooth manifolds: Convergence theory and twenty-one examples. Conference on Parsimony and Learning, 2023

    2023

  29. [37]

    V., and Vidal, R

    Peng, L., Giampouras, P. V., and Vidal, R. The ideal continual learner: An agent that never forgets. International Conference on Machine Learning, 2023

    2023

  30. [38]

    O., and Calandra, R

    Pineda, L., Amos, B., Zhang, A., Lambert, N. O., and Calandra, R. Mbrl-lib: A modular library for model-based reinforcement learning. Arxiv, 2021

    2021

  31. [39]

    Sample-efficient cross-entropy method for real-time planning

    Pinneri, C., Sawant, S., Blaes, S., Achterhold, J., Stueckler, J., Rolinek, M., and Martius, G. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning, 2021

    2021

  32. [40]

    Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents

    Powers, S., Xing, E., Kolve, E., Mottaghi, R., and Gupta, A. Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents. In Proceedings of The 1st Conference on Lifelong Learning Agents, 2022

    2022

  33. [41]

    Learning to learn without forgetting by maximizing transfer and minimizing interference

    Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., and Tesauro, G. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations, 2018

    2018

  34. [42]

    Experience replay for continual learning

    Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Neural Information Processing Systems, 2019

    2019

  35. [43]

    The cross-entropy method for combinatorial and continuous optimization

    Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1999

    1999

  36. [44]

    Curious exploration via structured world models yields zero-shot object manipulation

    Sancaktar, C., Blaes, S., and Martius, G. Curious exploration via structured world models yields zero-shot object manipulation. In Advances in Neural Information Processing Systems, 2022

    2022

  37. [45]

    M., Grabska-Barwinska, A., Teh, Y

    Schwarz, J., Luketina, J., Czarnecki, W. M., Grabska-Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress & compress: A scalable framework for continual learning. International Conference on Machine Learning, 2018

    2018

  38. [46]

    Shalev-Shwartz, S. et al. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 2012

    2012

  39. [47]

    Autonomous reinforcement learning: Formalism and benchmarking

    Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. Autonomous reinforcement learning: Formalism and benchmarking. International Conference on Learning Representations, 2022

    2022

  40. [48]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020

    2020

  41. [49]

    Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings. 1990

    1990

  42. [50]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE International Conference on Intelligent Robots and Systems, 2012

    2012

  43. [51]

    Vitter, J. S. Random sampling with a reservoir. ACM Transactions on Mathematical Software, 1985

    1985

  44. [52]

    and Ba, J

    Wang, T. and Ba, J. Exploring model-based planning with policy networks. International Conference on Learning Representations, 2020

    2020

  45. [53]

    Model predictive path integral control using covariance variable importance sampling

    Williams, G., Aldrich, A., and Theodorou, E. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv: 1509.01149, 2015

    2015 arXiv

  46. [54]

    Continual world: A robotic benchmark for continual reinforcement learning

    Wo czyk, M., Zaj a c, M., Pascanu, R., Kuci \'n ski, ., and Mi o \'s , P. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 2021

    2021

  47. [55]

    Continual task allocation in meta-policy network via sparse prompting

    Yang, Y., Zhou, T., Jiang, J., Long, G., and Shi, Y. Continual task allocation in meta-policy network via sparse prompting. In International Conference on Machine Learning, pp.\ 39623--39638. PMLR, 2023

    2023

  48. [56]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2019

    2019

  49. [57]

    Continual learning through synaptic intelligence

    Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017

    2017

  50. [58]

    The Schur complement and its applications, volume 4

    Zhang, F. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006

    2006

  51. [59]

    ACIL : Analytic class-incremental learning with absolute memorization and privacy protection

    Zhuang, H., Weng, Z., Wei, H., Xie, R., Toh, K.-A., and Lin, Z. ACIL : Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems, 2022

    2022

  52. [60]

    GKEAL : Gaussian kernel embedded analytic learning for few-shot class incremental task

    Zhuang, H., Weng, Z., He, R., Lin, Z., and Zeng, Z. GKEAL : Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  53. [61]

    DS-AL : A dual-stream analytic learning for exemplar-free class-incremental learning

    Zhuang, H., He, R., Tong, K., Zeng, Z., Chen, C., and Lin, Z. DS-AL : A dual-stream analytic learning for exemplar-free class-incremental learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.