{"id":"b277ab26-5511-41d3-a62b-ab4677256780","arxiv_id":"2411.15235","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CODE-CL combines conceptor matrices with a learnable shared-direction projection to reduce catastrophic forgetting and improve forward transfer in task-incremental learning.","lead":"This paper introduces CODE-CL, a continual learning method that uses conceptor matrices to project gradients away from old-task knowledge while selectively reusing shared directions for new tasks. On standard benchmarks, it reports higher accuracy and better forward knowledge transfer than prior gradient projection methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The shared-direction subspace U_t,and is computed from a single pre-training batch and never updated; as W and M change during the task, the released directions may drift out of alignment, undermining the FWT and forgetting guarantees.","rationale":"The paper provides code, consistent SOTA numbers on four benchmarks, and ablations over K, α, and ε, which is genuine evidence. My concern is not about the honesty of the results but about whether the mechanism that explains them is correct. The claim that CODE-CL promotes forward transfer by releasing shared directions requires the released subspace to remain the shared subspace while the model is being trained. Algorithm 1 computes U_t,and once, from a 125-sample batch, before the epoch loop, and never updates it. The projection I−C_{t−1} also uses a conceptor fixed at the end of the previous task, so the whole gradient constraint is static. In deep networks trained from a changing weight vector, the activation subspace of a task is known to drift; the paper itself updates C_t after the task, acknowledging that the conceptor changes with training. If the drift is rapid, the fixed intersection can become a stale region of activation space, and the method's FWT benefit would be an artifact of a lucky initialization rather than a principled identification of shared structure. The ε ablation only shows that the threshold matters, not that the subspace is accurate. This is the weakest link because every claimed advantage (higher ACC, better FWT, low BWT) depends on it. The test I propose—monitoring principal angles during training and comparing to a recomputed-U baseline—would directly settle whether this premise holds. If it holds, the paper's explanation is validated; if not, the empirical gains may be due to other factors, and the method would need a dynamic subspace estimate. This keeps the verdict at CONDITIONAL pending that check.","tokens_in":15037,"tokens_out":8113,"duration_ms":80139,"concrete_test":"On Split CIFAR100, for a held-out task t>1, record at every training epoch the principal angles between the frozen U_t,and (top-K of C_t,and) and the conceptor of the current model's activations on a fixed validation batch. Also run a variant that recomputes U_t,and every 10 epochs (or at each epoch if feasible) and compare final ACC and BWT. If the median principal angle exceeds about 20° or the recomputed-U variant changes ACC by more than 1 percentage point, the fixed-subspace premise is load-bearing and the current explanation of FWT/forgetting is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central premise is that the top-K singular vectors of C_t,and = C_t,pre ∧ C_t-1 (Algorithm 1 lines 5-8) identify directions that are (a) shared between old tasks and task t and (b) safe to release via M_t. This premise has a temporal component that is never checked: C_t,pre is computed from a batch of 125 activations of the pre-training model (Table 9, bs=125) before any gradient step on task t, and U_t,and is then frozen for the entire epoch loop (lines 15-19). During that loop, W and M_t both change, so the input activations of task t shift. The inferred intersection is only valid for the initial model. If the activation subspace of task t rotates appreciably during training, the released directions may no longer be the true shared directions: they could omit directions that became important for the new task (reducing FWT) or include directions that overlap with old-task knowledge after the shift (increasing forgetting). The paper's only evidence for the reliability of the intersection is the ε ablation (Table 7), which varies the threshold but does not test temporal stability. With only 125 samples to estimate high-dimensional covariance, the intersection is also subject to sampling noise. Thus the method's core mechanism rests on an untested stationarity assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CODE-CL, a continual learning method that uses conceptor matrices to estimate the importance of input-activation directions per layer for previous tasks. For each new task, it computes the intersection between the previous task conceptor and a pre-conceptor of the new task; if the overlap is large enough, it allows gradient updates along the top-K shared directions through a learnable matrix M_t, while projecting the base-weight gradient onto the pseudo-orthogonal complement I-C^{t-1}. The authors claim that this balances stability and plasticity, mitigates catastrophic forgetting, and improves forward knowledge transfer. Experiments on Split CIFAR100, Split MiniImageNet, 5-Datasets, and OL-CIFAR100 report state-of-the-art average accuracy and improved relative FWT over GPM-based baselines, with memory complexity comparable to GPM and TRGP.","tokens_in":15338,"tokens_out":7537,"duration_ms":70431,"significance":"If the reported results hold, CODE-CL is a competitive addition to gradient-projection continual learning, achieving the best average accuracy on three standard benchmarks (77.21%, 71.16%, 93.51%) and substantially higher relative FWT than TRGP, CUBER, and SGP. The method is reproducible in principle because the code is provided, and the conceptor formulation is a clean way to parameterize subspace overlap. However, the paper's own BWT numbers show that CODE-CL forgets more than TRGP, CUBER, and GPM on Split CIFAR100 and Split MiniImageNet, which conflicts with the abstract's 'minimal forgetting' claim. The main contribution is algorithmic and heuristic; there is no theoretical guarantee that the unconstrained M_t updates preserve old-task performance.","major_comments":[{"comment":"The claim of 'minimal forgetting' is not supported by the reported BWT values. On Split CIFAR100, CODE-CL has BWT of -1.1±0.28, which is worse than GPM (-0.2±0.19), TRGP (-0.1±0.18), and CUBER (+0.1±0.11); on Split MiniImageNet, BWT -1.1±0.3 is worse than GPM, TRGP, and CUBER. The text should soften this claim or explicitly discuss that the improved ACC/FWT come at the cost of slightly more forgetting than some prior methods.","section":"Abstract and Table 1"},{"comment":"The shared-direction subspace U_t,and is computed from a single forward pass over a batch of 125 samples before any training on task t, then frozen for the entire epoch loop (lines 3-8 and 15-19). Since W and M_t both change during training, the input activations of task t will shift, so the intersection may no longer represent the true shared directions. The paper does not test this stationarity assumption; the epsilon ablation (Table 7) varies only the threshold, not the timing or sample size. Please provide evidence of subspace stability (e.g., cosine similarity of top directions before/after training) or an ablation that recomputes C_t,and periodically.","section":"Algorithm 1 and Section 3.2.1"},{"comment":"The gradient constraint in Eq. (8) is applied only to the base weight W. The effective weights W_eff = W + W U_t M_t U_t^T are also updated through the unconstrained M_t gradient. Because the columns of U_t lie in the C_t-1 subspace (they come from the intersection C_t,and = C_t,pre ∧ C_t-1), the M_t update directly modifies directions deemed important for previous tasks. Thus Eq. (8) does not guarantee forgetting protection as claimed. The paper should analyze this effect or empirically test the impact of the M_t update on old-task performance (e.g., freezing M_t during training and measuring BWT).","section":"Section 3.2.2, Eq. (8), Algorithm 1 lines 16-19"},{"comment":"The aperture values are inconsistent between the main text and the supplementary material. The main text states α=16 for Split miniImageNet and α=8 for 5-Datasets, but Supplementary Table 9 reports α=8 and α=4, respectively. Since α is a key hyperparameter and the ablation in Figure 4 shows sensitivity, this inconsistency prevents exact reproduction of the reported results. Please correct the discrepancy and state the exact values used for Table 1.","section":"Section 4.1.2 and Supplementary Table 9"},{"comment":"The paper does not explain how task-level accuracy is computed after all tasks are learned. Training on task t uses the effective weights W_eff,t = W + W U_t M_t U_t^T, and W continues to change during later tasks. For evaluation on task i after the full sequence, the model must either re-apply the stored (U_i, M_i) to reconstruct the effective feature extractor or use the final W alone; these two protocols yield different numbers. The memory analysis (Table 3) suggests per-task storage of U_i and M_i, but the evaluation protocol is never stated, which affects the interpretation of every ACC and BWT number. Please clarify the exact evaluation procedure.","section":"Section 4.1.3 and Section 4.2.1 (evaluation protocol)"}],"minor_comments":[{"comment":"Algorithm 2 uses a threshold β to determine the effective rank of conceptors, but β is not listed in Table 9 or defined in the main text. Please specify its value and how it is chosen.","section":"Section 2.1 and Algorithm 2"},{"comment":"The condition 'if Θ(C_t,and)/Θ(C_t-1) > ε then for each layer l ∈ L' is ambiguous: it should be stated explicitly that the capacity ratio and the SVD are computed per layer, and that the learnable M_t is defined per layer as well.","section":"Algorithm 1, line 6"},{"comment":"Several BWT entries from prior works are reported without standard deviations (e.g., DFGP on Split CIFAR100). Please add the missing values or state that they are unavailable.","section":"Table 1"},{"comment":"The FWT metric as defined is a relative FWT against a GPM baseline. This should be stated more prominently in the main text to avoid readers interpreting it as an absolute measure of forward transfer.","section":"Eq. (10) and Table 2"},{"comment":"There are typographical errors in the title ('De ep'), the affiliation ('West Lafayete'), and some citation formatting. These should be corrected in the final version.","section":"Title and affiliations"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning venue and the experimental results are promising. The main issues are not fatal, but the evaluation protocol ambiguity (major comment 5) and the unconstrained M_t update (major comment 3) need to be resolved before the claims can be fully trusted. The hyperparameter inconsistency (major comment 4) is easily fixable but must be addressed. I recommend major revision rather than rejection because the core idea is sound and the reported gains are plausible, but the manuscript as written does not yet substantiate its strongest claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CODE-CL is a legitimate incremental advance: it combines conceptor matrices with a learnable per-direction release for shared subspaces, and the experiments are consistent enough to support the main accuracy claim. The code is public, the ablations cover the key hyperparameters, and the gains over GPM/TRGP/SGP are in the 1–3 point range, which is meaningful in this benchmark suite. That is the part worth taking seriously.\n\nThe soft spots are roughly what the reader flagged, plus one outright error. The BWT numbers do not support \"minimal forgetting\": -1.1% on two benchmarks is clearly worse than GPM, TRGP, and CUBER, so that phrase should go. The claim about beating the Multitask upper bound on 5-Datasets and MiniImageNet is unexplained and needs a comment—there are plausible reasons (regularization, multi-head evaluation, dataset ordering), but the paper offers none. More concretely, Section 4.1.2 and Table 9 disagree: the text gives α=16 for MiniImageNet and α=8 for 5-Datasets, while the table gives 8 and 4, and Table 5 suggests the reported 93.51 ACC for 5-Datasets corresponds to α=8, not α=4. That has to be reconciled before this can be trusted as a state-of-the-art claim.\n\nOn the stress-test concern about U_t,and being computed from a single pre-training batch and frozen: I think that is a legitimate limitation but not a load-bearing flaw. Forgetting is controlled by the fixed projection I - C^{t-1}, which does not depend on U_t,and; the stale intersection can only reduce the effectiveness of forward transfer, not cause forgetting. It is an untested stationarity assumption, and the ε ablation does not address it directly, but it is the kind of thing a good reviewer asks for in a revision rather than a reason to reject. Same for the 125-sample conceptor estimate: worth reporting variance over the sampling batch, but not fatal.\n\nOverall: this deserves a serious referee. The method is new enough, the empirical work is honest in its main numbers, and the flaws are fixable. I would send it to review with a request to fix the inconsistency, soften the forgetting claim, and justify or downplay the multitask comparison.","headline":"Solid incremental CL method with public code and consistent accuracy gains, but the forgetting claims overreach and there is a hyperparameter inconsistency that must be fixed.","tokens_in":15860,"tokens_out":2545,"would_cite":true,"duration_ms":25668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CODE-CL claims that selectively releasing shared old-task directions, while projecting gradients into the pseudo-orthogonal complement, yields both low forgetting and high forward transfer.","keywords":["continual learning","catastrophic forgetting","forward knowledge transfer","gradient projection","conceptor matrices","task-incremental learning","pseudo-orthogonal subspaces","deep learning"],"falsifier":"Train a sequence of two tasks with input subspaces whose overlap is known by construction (e.g., synthetic Gaussian features spanning chosen subspaces). Compute $C_{t,\\mathrm{and}}$ and compare its top-$K$ directions against the true shared subspace. Then flip the released directions: replace $U$ in $W^{\\mathrm{eff}} = W + W U M U^\\top$ with a random orthonormal set of the same dimension while keeping the projection and training otherwise identical. If accuracy on the new task stays high, the mechanism is not actually transferring learned structure; if backward transfer worsens, the intersection estimate is releasing unsafe directions. Either outcome would undermine the central claim.","tokens_in":14827,"feed_emoji":"🧠","tokens_out":8095,"duration_ms":71284,"temperature":0.7,"pith_summary":"Continual learning systems that guard against catastrophic forgetting by freezing old-task gradient directions pay for stability with poor forward transfer: new tasks cannot reuse what older tasks already learned. CODE-CL aims to have both. It represents each layer's past knowledge as a conceptor matrix, a regularized reconstruction of input activations whose singular values encode how important each direction is, and it projects new-task gradients into the pseudo-orthogonal complement of that subspace to avoid overwriting. At the same time, it computes the intersection between the old conceptor and the new task's pre-conceptor; when that overlap is large enough, it allows the network to learn a linear combination of the top shared directions through a learnable matrix. Across Split CIFAR100, Split MiniImageNet, and 5-Datasets, the paper reports accuracy of 77.21%, 71.16%, and 93.51%, respectively, surpassing prior gradient-projection methods and, on the latter two benchmarks, even the multitask training upper bound.","feed_headline":"CODE-CL beats prior continual-learning methods on three benchmarks","feed_subtitle":"Shared directions get reused, not frozen, so models learn new tasks without forgetting old ones.","key_machinery":"The carrying object is the conceptor matrix, a regularized reconstruction matrix for a batch of activation vectors: $C(X,\\alpha)=\\arg\\min_C \\frac{1}{b}\\|X-XC\\|_F^2 + \\alpha^{-2}\\|C\\|_F^2$, with closed form $\\frac{X^\\top X}{b}(\\frac{X^\\top X}{b}+\\alpha^{-2}I)^{-1}$. Reading $C$ through its SVD, $C = U S U^\\top$, the singular values $S_{ii}\\in(0,1)$ encode the importance of the directions $U_{:,i}$, so $C$ acts as a soft projection onto the layer's activation subspace. Conceptors form a Boolean algebra: $\\neg C = I-C$, $C\\wedge B = (C^{-1}+B^{-1}-I)^{-1}$ (with a pseudo-inverse variant for rank-deficient matrices), and $C\\vee B = \\neg(\\neg C \\wedge \\neg B)$. CODE-CL uses $\\wedge$ to locate directions shared by the old and new tasks, uses the top-$K$ singular vectors of that intersection to define the learnable subspace via $W^{\\mathrm{eff}} = W + W U M U^\\top$, and uses $\\neg C_{t-1}$ to enforce pseudo-orthogonal gradient updates, then $\\vee$ to merge old and new knowledge. This gives a single mechanism that decides, per direction, whether to freeze, project away, or actively reuse.","core_discovery":"On the paper's own terms, the central claim is that catastrophic forgetting and forward knowledge transfer are not in conflict once the update direction is chosen per direction rather than per task. A conceptor matrix $C$ is a soft projection built from the layer's input activations; its singular values in $(0,1)$ mark how much each direction matters to the tasks seen so far. Before learning task $t$, CODE-CL forms $C_{t,\\mathrm{and}} = C_{t,\\mathrm{pre}} \\wedge C_{t-1}$, the conceptor intersection of the new task and all previous tasks. If the capacity ratio $\\Theta(C_{t,\\mathrm{and}})/\\Theta(C_{t-1})$ exceeds a threshold $\\epsilon$, the top-$K$ directions of that intersection are released for learning by setting $W^{\\mathrm{eff}} = W + W U M U^\\top$, with $M$ a learned $K\\times K$ matrix, while gradients are constrained to $(I - C_{t-1})\\nabla_W L$. After training, the knowledge is consolidated by the union $C_t = C_{t,\\mathrm{post}} \\vee C_{t-1}$. The paper's experiments conclude that this recipe yields state-of-the-art accuracy, near-zero backward transfer, and the highest forward transfer among compared projection methods.","pith_inferences":["The threshold $\\epsilon$ and the count of free dimensions $K$ are chosen globally, but the paper's own ablations show accuracy climbs with $K$ and falls with larger $\\epsilon$; an editor's inference is that per-layer or per-task adaptive selection of $K$ and $\\epsilon$ could improve the stability-plasticity balance further, a direction the paper does not explore.","The reported forward-transfer metric is relative to GPM as baseline, so the magnitude of the FWT advantage is baseline-dependent; comparing CODE-CL against a common non-projection baseline (e.g., plain SGD or a regularization method) would be an informative test of how much of the gain is due to the released directions versus the projection itself.","Because conceptors are computed from a finite batch of activations, the intersection estimate carries sampling noise; a natural stress test is to compare the method's behavior on identical task pairs when the conceptor is estimated from increasingly small batches, predicting that BWT should degrade or FWT should drop once the intersection becomes unreliable.","The paper's task-incremental multi-head setup means task identity is provided at test time; whether the same direction-release mechanism helps in class-incremental or single-head settings, where the classifier suffers cross-task interference, is a plausible extension the paper does not address."],"forward_implications":["If the main results are right, gradient projection methods can be improved without extra data replay or network expansion: the gains come from selectively freeing shared directions.","CODE-CL's reported accuracy on Split MiniImageNet and 5-Datasets exceeds the multitask upper bound, so sequential learning with controlled transfer can beat joint training in these settings.","The added state per task is $O(TNK+TK^2)$ on top of the $O(N^2)$ conceptors, so the method remains practical for larger layers compared to per-task-gradient methods like CUBER.","Low backward transfer (-1.1%, -1.1%, -0.11% BWT on the three main benchmarks) means the released directions do not, as measured, harm old tasks, supporting the safety of the intersection estimate.","On the overlapping-class benchmark OL-CIFAR100, CODE-CL also leads (76.89% ACC with 6.02% relative FWT), suggesting the mechanism transfers beyond strictly disjoint class splits."],"supporting_citations":[{"why":"Supplies the conceptor-matrix definition, closed form, and the Boolean operations (not, and, or) on which CODE-CL builds.","marker":"[8]"},{"why":"Supplies the fact that gradients lie in the span of input activations, which justifies using activation subspaces to constrain gradient updates.","marker":"[36]"},{"why":"GPM is the baseline projection method CODE-CL extends; it also provides the dataloaders and the reference baseline for relative FWT.","marker":"[23]"},{"why":"TRGP is the trusted-region predecessor whose whole-task selection CODE-CL refines to individual shared directions, and a main comparison point.","marker":"[14]"},{"why":"SGP is the scaled-projection predecessor that CODE-CL compares against to show the benefit of direction-specific rather than uniform scaling.","marker":"[22]"},{"why":"CUBER provides the OL-CIFAR100 benchmark with overlapping classes and is a comparison method for both ACC and FWT.","marker":"[15]"},{"why":"Space Decoupling (SD), combined with TRGP, is a comparison method whose results CODE-CL aims to beat on the main benchmarks.","marker":"[37]"}],"fun_headline_variants":["CODE-CL: Learn new tasks without forgetting, and transfer knowledge too","CODE-CL: One method for less forgetting and more transfer","CODE-CL adapts to correlated tasks, boosting forward transfer","CODE-CL: Reuse shared directions to learn without forgetting","CODE-CL: A conceptor-based fix for catastrophic forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the intersection between the current task's activation directions and the stored summary of previous tasks, estimated from a finite batch with a threshold, reliably flags which directions are both shared and safe to release for learning.","fun_headline_variants_meta":{"raw":{"variants":["CODE-CL: Learn new tasks without forgetting, and transfer knowledge too","CODE-CL: One method for less forgetting and more transfer","CODE-CL adapts to correlated tasks, boosting forward transfer","CODE-CL: Reuse shared directions to learn without forgetting","CODE-CL: A conceptor-based fix for catastrophic forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2371,"prompt_tokens":1028,"completion_tokens":1343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1255}},"tokens_in":644,"tokens_out":1343,"duration_ms":9605,"temperature":1.0,"reasoning_tokens":1255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:05:13.249940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a sequence of two tasks with input subspaces whose overlap is known by construction (e.g., synthetic Gaussian features spanning chosen subspaces). Compute $C_{t,\\mathrm{and}}$ and compare its top-$K$ directions against the true shared subspace. Then flip the released directions: replace $U$ in $W^{\\mathrm{eff}} = W + W U M U^\\top$ with a random orthonormal set of the same dimension while keeping the projection and training otherwise identical. If accuracy on the new task stays high, the mechanism is not actually transferring learned structure; if backward transfer worsens, the intersection estimate is releasing unsafe directions. Either outcome would undermine the central claim.","supporting_citations":[{"cited_title":"Understanding deep learning re- quires rethinking generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the fact that gradients lie in the span of input activations, which justifies using activation subspaces to constrain gradient updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPM is the baseline projection method CODE-CL extends; it also provides the dataloaders and the reference baseline for relative FWT."},{"cited_title":"TRGP: Trust Region Gradient Projection for Continual Learning.In- ternational Conference on Learning Representations , 2022","cited_arxiv_id":null,"evidence_quote":"TRGP is the trusted-region predecessor whose whole-task selection CODE-CL refines to individual shared directions, and a main comparison point."},{"cited_title":"Continual Learning with Scaled Gradient Projection","cited_arxiv_id":null,"evidence_quote":"SGP is the scaled-projection predecessor that CODE-CL compares against to show the benefit of direction-specific rather than uniform scaling."},{"cited_title":"Beyond not-forgetting: continual learning with backward knowledge transfer","cited_arxiv_id":null,"evidence_quote":"CUBER provides the OL-CIFAR100 benchmark with overlapping classes and is a comparison method for both ACC and FWT."},{"cited_title":"Rethinking gradient projection continual learning: Stability/plasticity feature space decou- pling","cited_arxiv_id":null,"evidence_quote":"Space Decoupling (SD), combined with TRGP, is a comparison method whose results CODE-CL aims to beat on the main benchmarks."}],"review_version":1}