{"id":"acf02cf9-a8a5-401c-9d90-5f45792480a8","arxiv_id":"2506.00592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reducing churn in continual RL via C-CHAIN prevents NTK rank collapse and substantially improves learning across four benchmark suites.","lead":"C-CHAIN, a regularizer that penalizes network output changes on data outside the training batch, reduces churn and improves continual reinforcement learning across Gym Control, ProcGen, DeepMind Control, and MinAtar benchmarks. The paper links churn to the rank collapse of the neural tangent kernel and argues that reducing churn preserves plasticity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal NTK-rank mechanism is derived for TD regression but tested on PPO with loss-gradient NTKs, so the link from churn reduction to rank preservation is not yet established.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the theory is built for TD/squared-error regression with scalar outputs, while the experiments use PPO's clipped objective, and the empirical NTK is measured on loss gradients rather than network-output gradients. I agree with this assessment. The paper is a strong empirical contribution: C-CHAIN is evaluated across 24 continual RL settings with multiple baselines, reliable metrics, and ablations, and the performance gains are broadly credible. What is not established is the causal mechanism. The formal bridge between churn and NTK rank in Section 4 relies on the chain-rule structure Δθ = -η E_x[∇θ fθ(x) ∇_{fθ} L], which holds for the squared/TD loss sketched in the paper but not for the clipped PPO surrogate used in the main experiments. Moreover, the Appendix B NTK is a Gram matrix of PPO loss gradients, not of fθ(x) gradients; rank collapse of that matrix is a different empirical statement than rank collapse of the theoretical NTK. A concrete output-NTK measurement would settle whether the observed rank preservation is the claimed mechanism or a correlated side effect. If the test confirms the output-NTK behavior, the causal story is substantially strengthened; if not, the paper should be read as an empirical method paper with a heuristic explanation, which still merits publication but with a conditional verdict. I therefore recommend no change to the reader's CONDITIONAL verdict.","tokens_in":22249,"tokens_out":11869,"duration_ms":126150,"concrete_test":"Recompute the Appendix B NTK using gradients of the policy network's output logits fθ(s) with respect to θ on a fixed reference set of Starpilot states, for both Vanilla PPO and C-CHAIN, and compare approximate rank and sum of absolute off-diagonal values with Figures 6 and 11. If the output-based NTK does not show rank collapse for Vanilla, or C-CHAIN does not preserve its rank, then the reported NTK statistics are an artifact of using loss gradients and the mechanistic claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: reducing churn preserves the rank of the NTK and thereby mitigates plasticity loss. The formal derivation, however, applies to a scalar-output network trained on a fixed-target squared or TD loss (Eqs. 5-12), whereas the main experiments train a stochastic PPO policy with the clipped surrogate objective. For PPO, the parameter update is not expressible as Δθ = -η E_x[∇θ fθ(x) ∇_{fθ} L] as assumed in Eq. 4, because the clipped objective depends on policy ratios and advantages and the policy output is vector-valued; the two-effect decomposition in Eqs. 10-12 therefore does not transfer as written. In addition, the empirical NTK in Appendix B is computed from gradients of the PPO objective (g_i = ∇θ L_PPO), not from gradients of the network output fθ(x) as defined in Eq. 2. Rank collapse of this loss-gradient Gram matrix is not the same object as rank collapse of the theoretical NTK, so the measured evidence for the mechanism is not directly about the quantity the theory explains. If either mismatch is substantive, the empirical wins of C-CHAIN remain credible, but the claimed causal story connecting churn reduction to rank preservation is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies loss of plasticity in continual reinforcement learning through the lens of churn, defined as network output variability on out-of-batch data induced by mini-batch training. The authors use the empirical NTK to argue that rank collapse and churn exacerbate each other, propose a regularizer C-CHAIN that minimizes churn on a reference batch, and show that it improves average performance over vanilla PPO and six prior methods on 24 continual RL environments spanning Gym Control, ProcGen, DMC, and MinAtar. They also report reliable-metric confidence intervals and an NTK analysis intended to support the causal mechanism.","tokens_in":22548,"tokens_out":6036,"duration_ms":54630,"significance":"If the causal story holds, the paper makes a valuable contribution: it identifies churn as a controllable and measurable factor in plasticity loss, proposes a simple add-on regularizer that works across multiple RL algorithms and environments, and provides reproducible code. The empirical results are strong: C-CHAIN achieves the best aggregate scores in both Gym Control and ProcGen, with non-overlapping confidence intervals versus the best baseline on median, IQM, and optimality gap. The two-effect decomposition in Section 4.3 is a useful conceptual framework even if the derivation is heuristic. However, the theory-experiment gap discussed below means the causal mechanism is not yet established.","major_comments":[{"comment":"The two-effect decomposition is derived for a scalar-output network trained on a TD or squared loss, with the specific identity ∇_{Q_θ} L_Q = -(Q_θ(x) - target) leading to Eq. 12. The main experiments use PPO's clipped surrogate objective for a vector-valued stochastic policy; for that loss, ∇_θ(∇_f L) is not -g, and the parameter update is not of the assumed form Δθ = -η E_x[∇_θ f_θ(x) ∇_f L]. Therefore the claimed gradient-decorrelation and step-size adjustment effects are not shown to apply to the actual experimental setting. The paper should either extend the derivation to the PPO objective, present the analysis as a heuristic motivating the method, or test the two-effect decomposition on the TD/DQN setting where it applies.","section":"Section 4.3, Eq. 12"},{"comment":"The empirical NTK in Figures 6 and 7 is computed from gradients of the PPO objective (g_i = ∇_θ L_PPO), not from gradients of the network output f_θ(x) as defined in Eq. 2. Rank collapse of this loss-gradient Gram matrix is a different object from the theoretical NTK, so the measurements do not directly support the claim that churn reduction prevents rank collapse of the NTK in Eq. 2. The authors should recompute the empirical NTK using output gradients, or explicitly justify and clearly label the loss-gradient Gram matrix as a proxy, adjusting the wording in Section 5.2 accordingly.","section":"Appendix B and Eq. 2"},{"comment":"The vicious cycle between rank decrease and churn exacerbation is asserted rather than derived. Equation 7 is a linear recurrence for the error with a fixed N_θ; it does not by itself imply that churn lowers the rank of N_θ or that lower rank increases churn. The text should either provide a formal argument or explicitly state this as a hypothesis supported by the empirical observations.","section":"Section 4.2, Eq. 7"}],"minor_comments":[{"comment":"The caption and surrounding text do not clearly explain the visual notation (e.g., which shaded regions correspond to training versus reference data); consider adding a legend or a more explicit textual description.","section":"Figure 1"},{"comment":"The notation 'G_L is the gradient matrix of ∇_{f_θ} L(θ)' is confusing, and the matrix dimensions in the product N_θ S G_L are not made precise; please clarify the dimensions and the definition of G_L.","section":"Eq. 5"},{"comment":"The sentence 'This term could either dampen or accelerate the regular gradient g ∇_f L_f which depends on the sign of the kernel...' has an ambiguous antecedent for 'which depends'; rephrase to indicate clearly what depends on the sign.","section":"Section 4.3, after Eq. 12"},{"comment":"For MountainCar, the observation noise scale σ = 0.02 is chosen manually after finding σ = 2.0 too large; this per-environment design choice appears only in the appendix and should be mentioned in the main text when describing the setup.","section":"Section A.1"},{"comment":"Given the theory-experiment gap noted in the major comments, the abstract and conclusion use 'demonstrate' for the causal claims; consider softening to 'suggest' or 'provide evidence for' until the mismatch is resolved.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is strong and likely publishable after the theory-experiment mismatch is addressed. The main risk is that the causal mechanism claim overreaches; the authors should either close the gap by extending the derivation to PPO or explicitly recast the NTK analysis as empirical motivation. The code release and reliable metrics are strengths, and the paper is well within the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the empirical study, and it is genuinely good. The authors take the churn-reduction loss from Tang & Berseth 2024, adapt it to continual RL (C-CHAIN), and test it across 24 environments: Gym Control, all of ProcGen, DMC, MinAtar, plus two continual SL baselines. They compare against six methods, report 6–12 seeds, use reliable metrics with confidence intervals, and release code. On aggregate, C-CHAIN beats vanilla and the other baselines on Gym Control and ProcGen, with non-overlapping CIs on the reliable metrics. The ablation separating the projective and orthogonal components is also informative. This is a solid, honest empirical contribution.\n\nThe theory is the soft spot, and the stress-test note lands. The NTK bridging argument is developed for a scalar output under a fixed-target squared loss, where the update is Δθ = -η G^T ∇_f L. But the experiments use PPO with a clipped policy objective and a vector-valued policy; the two-effect decomposition in Eqs. 10–12 does not carry over as written. The empirical NTK in Appendix B is also computed from gradients of the PPO loss, not from gradients of the network output f_θ as defined in Eq. 2. So the measured rank collapse is not the same object the theory explains. The authors do use the NTK measurements predictively rather than to fit constants, so the circularity burden is low. But the causal chain from churn reduction to rank preservation to plasticity is asserted more than demonstrated. The \"vicious cycle\" in Eq. 7 is also heuristic, not derived. These are real gaps, but they don't undermine the empirical claim that C-CHAIN works; they undermine only the stated mechanism.\n\nThe paper is also honest about its limits. C-CHAIN underperforms L2 Init and Weight Clipping on the two MNIST continual settings, and the authors say so. The MountainCar result is weaker and they discuss why. Self-citation is to a prior churn paper that is the direct basis for the loss, which is appropriate.\n\nWho is this for? Researchers working on plasticity loss, continual RL, or RL optimization generally. The empirical finding that a simple regularizer helps across many environments is worth engaging with, even if the theoretical packaging overreaches. I'd want to see the NTK rank measured on network-output gradients for PPO, or a PPO-specific derivation, before fully buying the mechanism. But the paper deserves serious peer review; it is not a desk reject. I'd accept it with a request for a substantial revision of the theory section, and I'd cite it for the empirical result.","headline":"A broad, believable empirical case that churn reduction helps continual RL, burdened by a theory section whose NTK mechanism doesn't line up with the PPO experiments.","tokens_in":23045,"tokens_out":1443,"would_cite":true,"duration_ms":15881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reducing churn — the uncontrolled change in a network's outputs for data outside the training batch — restores plasticity in continual RL, and a simple regularizer beats six prior methods across 24 benchmark settings.","keywords":["loss of plasticity","continual reinforcement learning","churn","neural tangent kernel","C-CHAIN","gradient decorrelation","non-stationary learning","plasticity"],"falsifier":"Train a continual RL agent while counterfactually forcing the empirical NTK to keep high rank without reducing churn (for example, injecting orthogonal gradient directions that stabilize rank but leave out-of-batch outputs free to move); if plasticity still degrades, churn is not the operative cause, and if plasticity is preserved, churn is incidental to rank. A cheaper test: compute the $②$ term for the PPO clipped objective and check whether it equals $-\\bar{g}^\\top g \\cdot g$; if not, the step-size-adjustment interpretation is miscalibrated for the main experiments.","tokens_in":22062,"feed_emoji":"🤖","tokens_out":8180,"duration_ms":66876,"temperature":0.7,"pith_summary":"This paper tries to establish that loss of plasticity in continual reinforcement learning is driven by churn: the way a mini-batch update silently rewrites the network's outputs for data not in that batch. Using the Neural Tangent Kernel (NTK) matrix as a bridge, the authors argue that as training progresses the NTK loses rank, gradients become correlated, churn worsens, and the error dynamics of Equation 7 destabilize in a vicious cycle that is the phenomenon known as plasticity loss. The proposed remedy, C-CHAIN, continually minimizes churn for out-of-batch reference data alongside regular training, and the paper shows this preserves NTK rank, decorrelates gradients, and adaptively rescales step sizes. If correct, churn is not a side effect but a controllable causal lever, and a cheap regularizer can replace resets, clipping, and other interventions. Empirically, C-CHAIN achieves the best aggregate scores on continual Gym Control and ProcGen and beats six related methods across 24 continual RL settings.","feed_headline":"Curbing churn stops plasticity loss in continual RL","feed_subtitle":"One regularizer keeps the NTK full-rank and beats six baselines across 24 benchmark settings.","key_machinery":"The machinery is the empirical Neural Tangent Kernel matrix $N_\\theta(i,j) = \\nabla_\\theta f_\\theta(x_i)^\\top \\nabla_\\theta f_\\theta(x_j)$, whose rank serves as the paper's formal indicator of plasticity, together with the first-order churn identity $C_f(\\bar{x}, \\theta, \\Delta\\theta) \\approx \\nabla_\\theta f_\\theta(\\bar{x})^\\top \\Delta\\theta$. Combining these yields the vector form $C_f(\\theta, \\Delta\\theta) \\approx -\\eta N_\\theta S G_L$, showing that the NTK determines churn independently of the loss; substituting this into the error recursion gives the update matrix $I - \\eta N_{\\theta_t} S_i$ that carries the stability argument. C-CHAIN is the algorithm that minimizes churn on a reference batch disjoint from the training batch, and its gradient decomposition into a kernel-suppression term and a projection term is what supports the two-efficacy claim.","core_discovery":"The central claim is that churn and plasticity loss are two views of the same NTK collapse. The error dynamics $E_i(\\theta_{t+1}) \\approx (I - \\eta N_{\\theta_t} S_i) E_i(\\theta_t)$ show that when all pairs of data points develop correlated gradients, the NTK matrix $N_\\theta$ loses rank, out-of-batch outputs swing uncontrollably with every update, and learning becomes unstable exactly as tasks keep changing. Churn reduction breaks the cycle: C-CHAIN's loss $L_f^{\\mathrm{cr}}(\\theta) = \\tfrac{1}{2}\\mathbb{E}_{\\bar{x} \\in B_{\\mathrm{ref}}}[C_f(\\bar{x}, \\theta, \\Delta\\theta)^2]$ pushes against churn on reference data, and its gradient splits into two effects — a decorrelation term that suppresses off-diagonal NTK entries and a projection term that rescales the regular training gradient by the kernel between reference and training data. The paper demonstrates the rank-preserving effect empirically and shows that the two components add up to the full benefit, with the decorrelation (orthogonal) component carrying most of the weight.","pith_inferences":["The theory's two-effect derivation uses a TD/Q-learning loss, while the main experiments run PPO's clipped objective; if the projection interpretation fails to transfer, the step-size-adjustment half of the story weakens while the decorrelation half may still hold — an asymmetry worth testing directly.","A natural stress test: if churn is the operative cause, then a counterfactual that keeps the NTK full-rank while allowing churn (or reduces churn while keeping rank low) should separate the two mechanisms; the paper's own MNIST results, where C-CHAIN helps less than in RL, hint that churn's role is amplified when the data distribution is shaped by the agent's own policy.","Because churn reduction and parameter-near-initialization methods act on different objects (kernel off-diagonals versus parameter distance), combining C-CHAIN with weight clipping or L2-init-style regularization might compound gains, a hypothesis the paper does not test."],"forward_implications":["If churn is a causal lever, NTK rank collapse is preventable by regularization alone, without resets, weight clipping, or parameter-space constraints.","Because C-CHAIN only needs a reference batch from current data and never needs to know when a task switched, it can be layered onto existing RL algorithms (the paper demonstrates PPO and DoubleDQN) with a single relative-scale hyperparameter.","The decomposition predicts that the decorrelation effect dominates, so cheaper approximations to the Hessian or kernel term could yield most of the benefit.","Better-than-oracle results on continual control suggest churn reduction preserves useful transfer across related tasks rather than merely blocking interference.","The same recipe transfers to continuous control and to a value-based agent, indicating the mechanism is not specific to one objective or action space."],"supporting_citations":[{"why":"Source of the churn-reduction loss and the 'chain effect' argument that C-CHAIN extends from single-MDP RL to continual RL.","marker":"(Tang & Berseth, 2024)"},{"why":"Supplies the empirical NTK definition used as the formal bridge between churn and plasticity.","marker":"(Achiam et al., 2019)"},{"why":"Establishes that NTK rank decrease signals plasticity loss, the phenomenon the paper's theory explains.","marker":"(Lyle et al., 2024)"},{"why":"Introduces policy churn as a measurable phenomenon in deep RL, motivating the analysis.","marker":"(Schaul et al., 2022)"},{"why":"Provides the TRAC code base and the continual Gym Control and ProcGen task setups used in the main comparisons.","marker":"(Muppidi et al., 2024)"},{"why":"Supplies the approximate-rank statistic used to track empirical NTK rank collapse.","marker":"(Kumar et al., 2021)"},{"why":"Primacy-bias hypothesis that early overfitting hampers later learning, which the churn account operationalizes.","marker":"(Nikishin et al., 2022a)"},{"why":"The L2 Init (regenerative regularization) baseline that C-CHAIN outperforms in the continual RL experiments.","marker":"(Kumar et al., 2023b)"}],"fun_headline_variants":["Cut churn to stop plasticity loss in continual RL","Reduce churn, keep RL plastic across tasks","Churn kills plasticity: reduce it for continual RL","Curb churn, preserve plasticity in continual RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The two-effect analysis is derived for a TD/Q-learning loss, but the main experiments train with PPO's clipped policy objective, for which the key gradient identity that produces the step-size-adjustment term does not hold exactly; if that interpretation does not transfer, half the explanatory story is unsupported in the settings where the method is actually evaluated.","fun_headline_variants_meta":{"raw":{"variants":["Cut churn to stop plasticity loss in continual RL","Reduce churn, keep RL plastic across tasks","Churn kills plasticity: reduce it for continual RL","Curb churn, preserve plasticity in continual RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1488,"prompt_tokens":932,"completion_tokens":556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":548,"tokens_out":556,"duration_ms":5874,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:02:08.706635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a continual RL agent while counterfactually forcing the empirical NTK to keep high rank without reducing churn (for example, injecting orthogonal gradient directions that stabilize rank but leave out-of-batch outputs free to move); if plasticity still degrades, churn is not the operative cause, and if plasticity is preserved, churn is incidental to rank. A cheaper test: compute the $②$ term for the PPO clipped objective and check whether it equals $-\\bar{g}^\\top g \\cdot g$; if not, the step-size-adjustment interpretation is miscalibrated for the main experiments.","supporting_citations":[],"review_version":1}