{"id":"f64e02d8-deb8-421d-b40f-1ff375014399","arxiv_id":"2507.09177","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An online follow-the-leader world model updated by analytic least squares, combined with CEM planning, solves sequential robotic tasks without forgetting on a new unified-dynamics benchmark.","lead":"This paper builds a continual reinforcement learning agent that learns the environment dynamics with a closed-form online update and then plans actions with model predictive control, avoiding catastrophic forgetting. It also introduces Continual Bench, a benchmark with six robotic manipulation tasks that share one underlying world model, on which the proposed agent matches or beats deep-model baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 1 (Eq. 5) is violated in continuous-control CRL, so the proven no-forgetting regret bound of Theorem 1 does not apply to the experimental setting; the empirical results may stand, but the theoretical claim is unsupported.","rationale":"The reader's weakest_assumption identifies Assumption 1 (Eq. 5), and I agree that this is the load-bearing premise. The proof of Theorem 1 in Appendix E uses Assumption 1 in Lemma 2, Lemma 3, and Proposition 2 to obtain the O(sqrt(K^2 D log T)) regret bound. Without it, the bound does not follow. The assumption is not a technical convenience: in a continuous state-action space, the uniform distance between a new point's rank-one outer product and the empirical average cannot shrink as 1/t because new points are almost never exactly previously observed points. Task switches make this worse by deliberately changing the distribution of visited state-action pairs, which is the defining difficulty of continual RL that the paper aims to address. Remark 1 explicitly limits the assumption to exploration 'within 1/(lambda t) away from the center,' which shrinks to zero, effectively conceding that the assumption fails under the distributional shift the method claims to handle. The empirical results are credible: OA is tested against strong baselines on an open-sourced benchmark with 7 seeds and matches Perfect Memory, and the code is publicly available. The concern is not about the empirical validity of the method but about the strength of the theoretical claim in the abstract. The conditional verdict is appropriate: the authors should temper the 'immune to forgetting by construction' and 'proven regret bound' phrasing, state the limits of Assumption 1, and either verify the assumption on Continual Bench or reposition the theory as applying only when inputs revisit a finite or near-finite set. My recommended concrete check would settle whether the assumption actually holds in the reported experiments; if it fails (as expected), the paper's central theoretical justification is unsupported, even though the practical contribution remains interesting.","tokens_in":29871,"tokens_out":7671,"duration_ms":90857,"concrete_test":"Record the sequence of states and actions collected by OA on Continual Bench, and at each time t compute L(t) = sup_x ||phi(x)phi(x)^T - (1/t) sum_{i<=t} phi(x_i)phi(x_i)^T||_2 over the observed data (or over a Monte Carlo sample of inputs around the observed data) using the paper's Losse features (Appendix A.1) and the chosen lambda. Plot L(t) against 1/(lambda t) over the whole run. If L(t) exceeds 1/(lambda t) at any time, especially around task switches, then Assumption 1 is violated in the exact regime where Theorem 1 is invoked, and the theoretical guarantee cannot be used to support the 'no forgetting' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that the sparse FTL update (Eq. 4) is a no-regret world model (Theorem 1, Eq. 7), which underpins the abstract's assertion that the model is 'immune to forgetting by construction.' The proof of Theorem 1 relies critically on Assumption 1 (Eq. 5), which requires that for every t, the outer product of any possible input's feature map phi(x)phi(x)^T lies within spectral norm 1/(lambda t) of the empirical average (1/t) sum_i phi(x_i)phi(x_i)^T. This is not a mild assumption in continuous-control CRL: the feature map is nonlinear and inputs live in a continuous space, so the empirical covariance of rank-one outer products does not converge uniformly over the entire input space at rate 1/t. At task boundaries, the state-action distribution shifts, so newly visited inputs are precisely those for which the distance in Eq. (5) is large. Remark 1 concedes the assumption requires exploring 'within 1/(lambda t) away from the center in the feature space,' a ball that shrinks to zero radius. Hence Theorem 1's guarantee does not apply to the Continual Bench experiments or to the general CRL setting the paper targets. The sparse update is also only approximately the FTL solution (Proposition 3), so the phrase 'by construction' overstates what is proven. The empirical non-forgetting results may still be valid, but the formal claim of a proven no-forgetting bound under mild assumptions is unsupported: Assumption 1 is doing the work and is violated exactly when forgetting would occur.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Online Agent (OA) for continual reinforcement learning. OA learns a sparse, randomly featured, ridge-regularized linear world model with a closed-form Follow-The-Leader update, and acts by CEM/MPC planning using this model. The authors claim that the sparse online update is no-regret, with a bound of O(sqrt(K^2 D log T)) under assumptions stated as mild, and hence that the world model is immune to forgetting by construction. To evaluate OA, they introduce Continual Bench, a lightweight Mujoco-based benchmark with six Meta-World-style tasks arranged so that a unified dynamics exists across tasks. On this benchmark, OA is reported to maintain performance on all previously seen tasks, matching a Perfect Memory deep-model baseline and outperforming deep-model agents with EWC/SI, Coreset, and fine-tuning.","tokens_in":30201,"tokens_out":8083,"duration_ms":113108,"significance":"If the theoretical claim held as stated, the paper would make a significant contribution: an online, replay-free world model with a logarithmic regret guarantee would be a strong building block for continual model-based RL, and the proposed benchmark would fill a practical gap in CRL evaluation. The empirical design has real strengths: the benchmark is open-sourced, experiments use 7 seeds, baselines share the same planner and hyperparameters, and OA's non-forgetting behavior on Continual Bench is visually and numerically consistent. However, the theoretical guarantee is currently the central advertised contribution, and both the assumptions and the proof have load-bearing problems; the empirical results alone, while promising, do not establish the paper's headline claim of a proven no-regret world model under mild assumptions.","major_comments":[{"comment":"Assumption 1 is not mild and is likely violated exactly in the CRL setting. For a nonlinear feature map on a continuous input space, the supremum over all x of ||phi(x)phi(x)^T - (1/t) sum_i phi(x_i)phi(x_i)^T||_2 does not generally decay as 1/(lambda t); if phi has sparse, roughly disjoint supports for distant inputs, the norm of a new rank-one term can remain bounded away from zero. For large t the right-hand side goes to zero, which would force every possible feature outer product to coincide with the empirical mean. Remark 1 itself concedes that the assumption holds only if new inputs are explored within a 1/(lambda t)-neighborhood of observed data, a ball that shrinks to radius zero. Task switches in continual RL are precisely the moments when newly visited state-action pairs are far from previous data, so Theorem 1 does not apply to the Continual Bench experiments or to the general CRL setting claimed in the abstract.","section":"4.2, Eq. (5), Remark 1"},{"comment":"The proof of Theorem 1 uses the Loewner semidefinite order (⪯) on matrices that are not symmetric and not square. For example, Lemma 2 claims (1 - 1/t) W(t) + Delta_t ⪯ W(t+1) ⪯ W(t) + Delta_t, where W(t), W(t+1), and Delta_t are D x S matrices; the positive-semidefinite order is undefined for nonsymmetric rectangular matrices. The same issue appears in Lemma 3, Proposition 2, and Proposition 3, where expressions such as W(t+1) - fW(t+1) ⪯ K r_M Delta_t are written as if they were PSD inequalities. Since these inequalities are then used to bound the per-step losses and the final regret, the formal proof as written does not establish Theorem 1.","section":"Appendix E.2, Lemma 2; Appendix E.3, Proposition 3"},{"comment":"Assumption 3 is a data-dependent lower bound on K that involves A^{(t)}_{ss}, the feature values at x_t, and y_t, yet K is a fixed architectural hyperparameter in the experiments (Appendix A.3 fixes Lambda and hence the activation ratio). The paper says 'we choose K such that Assumption 3 holds,' but the quantities in the bound are not known before the data are observed, and no verification of Assumption 3 is provided for Continual Bench. Consequently, the proof that the sparse update in Eq. (4) approximates the exact FTL solution, which is essential to the 'no-forgetting by construction' claim, is not connected to the configurations actually evaluated.","section":"4.2, Assumption 3, Eq. (8)"}],"minor_comments":[{"comment":"The notation for A^{(t)} is inconsistent: it is introduced as Phi_{t-1}^T Phi_{t-1}, but Eq. (4) and the surrounding text index A by the current time step as if the new point has already been incorporated; please clarify whether A^{(t)} includes phi(x_t).","section":"Appendix E.1, Lemma 1"},{"comment":"The derivation of the constant K' contains several unclear steps, including the placement of phi(x_t)^T phi(x_t) and the manipulation of y_t^T y_t as scalar factors; these steps should be rewritten carefully because the resulting bound feeds into Proposition 3.","section":"Appendix E.3.1, Proposition 2"},{"comment":"The claim that OA achieves Perfect Memory's performance 'by a much more efficient online update' would be stronger with a direct computation-time or per-step-cost comparison; the current text compares update mechanisms conceptually but does not measure wall-clock time or total optimization cost.","section":"6.3 and Appendix A.5"},{"comment":"The x-axis labels for the buffer-size ablation are difficult to read; please reformat the ticks so that the buffer sizes (1, 5000, 10000, 20000, 141946) are legible.","section":"Figure 6(c)"},{"comment":"Remark 1 should explicitly acknowledge that the stated sufficient condition for Assumption 1 (exploration within a shrinking 1/(lambda t) ball) is incompatible with the distributional shift induced by switching tasks; leaving this as a passing remark obscures the strength of the assumption.","section":"Remark 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution and the Continual Bench environment are solid and could form the basis of a good paper, but the theoretical section is currently a liability. The invalid use of semidefinite inequalities on nonsymmetric matrices is a technical error that might be repairable with norm-based arguments, but the implausibility of Assumption 1 is more fundamental: it is not a mild assumption, and it is unlikely to hold at task boundaries. In revision, the authors should either provide a correct proof under genuinely mild and verifiable conditions, or substantially re-scope the claims so that the paper is presented as an empirical method plus benchmark, with the regret analysis explicitly conditional on assumptions that are checked on the benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is real: an online Follow-The-Leader sparse world model with CEM planning matches the Perfect Memory baseline on Continual Bench across 7 seeds, and the benchmark itself is a useful, lightweight addition to the CRL toolbox. Those results are worth taking seriously. What is genuinely new here is the ridge-regularized sparse update to the FTL rule, the regret analysis attempting to justify it, the planning-only formulation of CRL, and the open-sourced Continual Bench. The ablation showing that sparser models do no worse is also a nice touch.\n\nThe soft spots are not minor, because they sit on the paper's main claim. Assumption 1 (Eq. 5) demands that for every possible input x, the outer product phi(x)phi(x)^T lies within spectral norm 1/(lambda t) of the empirical average of previously seen feature outer products. For nonlinear features in continuous control, there is no reason for uniform convergence at that rate over the entire input space, and at task boundaries the newly visited state-action pairs are exactly where that distance is large. The paper's own Remark 1 concedes the assumption only holds when new points stay near observed data. So Theorem 1's regret bound does not apply to the Continual Bench experiments, and the abstract's \"immune to forgetting by construction\" is not supported by the proof. Separately, the proof uses semidefinite inequalities on nonsymmetric matrix differences (Lemma 2, Proposition 3), which is nonstandard and needs to be checked. The sparse update is also only an approximation to the exact FTL solution (Proposition 3), so even the \"by construction\" phrasing overstates what is proven. None of this kills the empirical finding, but it does mean the paper is currently selling a theory it does not actually have.\n\nThe benchmark is deliberately designed so that a unified dynamics exists; that is fine as a controlled setting, but it leaves the method's behavior under conflicting-dynamics CRL untested. The limitations section mentions some of this, but the abstract and introduction still promise more than the math delivers.\n\nWho is this for: anyone working on model-based RL or continual learning who wants a cheap, reproducible CRL environment and is interested in whether closed-form online updates can sidestep catastrophic forgetting in practice. That reader gets real value from the empirical sections even if the theory is disappointing.\n\nRecommendation: send it to peer review. A serious referee should engage with the paper, but the authors need to either fix the proof, replace the no-regret claim with an honest bound under a less heroic assumption, or explicitly reposition the theoretical section as heuristic motivation. With that revision, the empirical contribution and benchmark make it publishable.","headline":"The benchmark and empirical results are solid, but the advertised no-forgetting guarantee does not cover the actual experimental setting; the paper needs major revision before it earns its abstract.","tokens_in":30744,"tokens_out":1889,"would_cite":true,"duration_ms":26758,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68W27"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that learning a sparse Follow-The-Leader world model online — with ridge regularization and closed-form updates — is no-regret, so planning with the latest model solves each new task while keeping old skills…","keywords":["continual reinforcement learning","catastrophic forgetting","online world models","Follow-The-Leader learning","model predictive control","regret bounds","sparse feature encoding","Continual Bench"],"falsifier":"Train OA on Continual Bench with the task order reversed and record the world model's prediction loss on held-out data from earlier tasks; if the loss on old-task data rises after a task switch beyond the $\\mathcal{O}(\\log T)$ regret drift, the 'immune to forgetting by construction' claim would be contradicted. Alternatively, compute the Assumption 1 quantity at each switch and check whether the sup norm frequently exceeds $1/(\\lambda t)$, which would mean Theorem 1's bound is not in force.","tokens_in":29603,"feed_emoji":"🤖","tokens_out":6331,"duration_ms":71015,"temperature":0.7,"pith_summary":"This paper tries to establish that catastrophic forgetting in continual reinforcement learning can be avoided without replay buffers, task IDs, or per-task networks, by learning one shared world model online and planning with it. The world model is a sparse random-feature network updated in closed form by a Follow-The-Leader rule, so each new experience enters a running least-squares solution instead of perturbing a trained deep network. The paper proves this updater is no-regret, with regret bounded by $\\mathcal{O}(\\sqrt{K^2D\\log(T)})$ under its assumptions, which is what makes the model 'immune to forgetting by construction.' Actions come from a model predictive controller (CEM planner) that re-plans with the current model for whatever reward is presented, so a new task is just a new reward function. On the paper's new Continual Bench environment, this Online Agent matches a perfect-memory deep agent on average performance and achieves lower cumulative regret than deep-world-model baselines with regularization, replay, or architecture-based continual learning techniques.","feed_headline":"No-regret world model solves continual RL without forgetting","feed_subtitle":"A closed-form online update matches a perfect-memory agent while using constant per-step compute.","key_machinery":"The load-bearing object is the sparse Follow-The-Leader world model: $y = W\\sigma(Px)$ with a fixed random projection $P$, a localized sparse feature map $\\phi$, and weights updated in closed form by Eq. (4), $W_s^{(t)} = (A_{ss}^{(t-1)} + \\tfrac{1}{\\lambda}I)^{-1}(B_s^{(t-1)} - A_{ss}^{(t-1)} W_s^{(t-1)})$. The ridge term $\\tfrac{1}{\\lambda}I$ is what gives each local update a unique minimizer and lets the proof bound the gap between the sparse update and the dense least-squares solution, while the sparse activation set $s$ keeps the per-step update cost constant in $K$. A CEM planner with shift initialization, colored noise, and memory converts the model's predictions into actions, so the same learned dynamics serve every task.","core_discovery":"The central claim is that a single agent can continually solve sequentially presented tasks by maintaining an online Follow-The-Leader world model and never retraining on past data. The paper formalizes this as Theorem 1: under Assumptions 1--3, the sparse updater of Eq. (4) satisfies $\\mathrm{Regret}(T) \\le c_W c_y \\sqrt{20(K^2 r_M^2 + 1)D(\\log T + 1)}$, meaning the cumulative prediction loss of the sparse model stays within $\\mathcal{O}(\\sqrt{K^2D\\log T})$ of the best linear predictor in hindsight. The paper also argues that the planner, which only sees the latest model, does not need task boundaries, because the task dependence lives in the externally provided reward function, not in the learned dynamics. Empirically, OA keeps high success on all previously seen tasks in Continual Bench while learning new ones, matching Perfect Memory and surpassing deep model-based agents with SI, Coreset, and PackNet-style continual learning techniques.","pith_inferences":["If Assumption 1 holds broadly enough, the closed-form update strategy transfers to any sequential prediction problem where dynamics are shared and only goals change, including online system identification outside reinforcement learning.","The deterministic FTL model carries no uncertainty estimate; coupling it with calibrated uncertainty and explicit exploration would be a direct extension that the paper itself identifies as future work.","Continual Bench is episodic with explicit task switches; a reset-free version with irreversible states would stress-test whether the no-forgetting property persists when tasks blend together without boundaries.","A simple empirical check would be to log $\\sup_x \\|\\phi(x)\\phi(x)^\\top - \\tfrac{1}{t}\\sum_i \\phi(x_i)\\phi(x_i)^\\top\\|$ at each task boundary; if it frequently exceeds $1/(\\lambda t)$, the conditions for Theorem 1 are violated even if the practical algorithm still performs well."],"forward_implications":["A continual agent can be built with a single shared component — the world model — that updates every step at constant cost, instead of replaying old data to convergence.","With a no-regret world model, each new task is solvable by planning with the current model and the new reward function, so the agent never needs task IDs or task-specific heads.","On Continual Bench, OA keeps success on all six previously seen tasks at the level of a perfect-memory agent while deep baselines degrade, and its average performance grows as tasks accumulate.","The sparse model's utilization rises as tasks accumulate but its update cost stays fixed, so capacity does not exhaust through the iterative-learning blind spots that subspace-based continual learning methods hit.","The benchmark design shows that spatial arrangement, not just temporal concatenation, determines whether a unified dynamics exists and whether forgetting and transfer can be studied at all."],"supporting_citations":[{"why":"Supplies the Follow-The-Leader sparse encoding and the analytic online update rule that OA extends.","marker":"Liu et al. (2024)"},{"why":"Provides the standard FTL regret-bound technique that the proof adapts with Assumption 1.","marker":"Shalev-Shwartz et al. (2012)"},{"why":"Supplies the Schur complement lemma used to bound the gap between sparse and dense solutions.","marker":"Zhang (2006)"},{"why":"Cited as evidence that unregularized rank-deficient updates may not converge, motivating the ridge term.","marker":"Peng & Vidal (2023)"},{"why":"Supplies the Meta-World task primitives and reward subroutines on which Continual Bench is built.","marker":"Yu et al. (2019)"},{"why":"Defines the Continual-World benchmark whose conflicting-dynamics flaw motivates the Continual Bench design.","marker":"Wołczyk et al. (2021)"},{"why":"Contributes the colored-noise and memory modifications used by the CEM planner.","marker":"Pinneri et al. (2021)"}],"fun_headline_variants":["Online world model sidesteps catastrophic forgetting in continual RL","Follow-The-Leader planner achieves no-forgetting continual learning","Sparse online updates match perfect memory in CRL bench","No-replay agent meets perfect-memory baseline on continual tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof's load-bearing assumption is that every new state-action pair is, in feature space, close to the average of all previously seen pairs (within $1/(\\lambda t)$); switching tasks is precisely when that is most likely to fail, and the paper concedes the assumption holds only when new exploration stays near old data.","fun_headline_variants_meta":{"raw":{"variants":["Online world model sidesteps catastrophic forgetting in continual RL","Follow-The-Leader planner achieves no-forgetting continual learning","Sparse online updates match perfect memory in CRL bench","No-replay agent meets perfect-memory baseline on continual tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1559,"prompt_tokens":963,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":528}},"tokens_in":579,"tokens_out":596,"duration_ms":8189,"temperature":1.0,"reasoning_tokens":528,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:03:13.091644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train OA on Continual Bench with the task order reversed and record the world model's prediction loss on held-out data from earlier tasks; if the loss on old-task data rises after a task switch beyond the $\\mathcal{O}(\\log T)$ regret drift, the 'immune to forgetting by construction' claim would be contradicted. Alternatively, compute the Assumption 1 quantity at each switch and check whether the sup norm frequently exceeds $1/(\\lambda t)$, which would mean Theorem 1's bound is not in force.","supporting_citations":[{"cited_title":"S., and Lin, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Follow-The-Leader sparse encoding and the analytic online update rule that OA extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the standard FTL regret-bound technique that the proof adapts with Assumption 1."},{"cited_title":"The Schur complement and its applications, volume 4","cited_arxiv_id":null,"evidence_quote":"Supplies the Schur complement lemma used to bound the gap between sparse and dense solutions."},{"cited_title":"Sample-efficient cross-entropy method for real-time planning","cited_arxiv_id":null,"evidence_quote":"Contributes the colored-noise and memory modifications used by the CEM planner."}],"review_version":1}