{"id":"7f58ab5c-91f5-4ef3-81b4-89fcc0a463c8","arxiv_id":"2505.24816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CL-LoRA adds a fixed random-orthogonal shared LoRA branch for cross-task knowledge and task-specific LoRA branches with block-wise weights, improving rehearsal-free class-incremental learning accuracy at low parameter cost.","lead":"This paper introduces CL-LoRA, a continual learning method that splits low-rank adapters into shared and task-specific parts so a frozen vision model can learn new classes in sequence. It reports top accuracy on three benchmarks with only 0.3% trainable parameters, making the approach cheaper to train and run than prior adapter-based methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference in Eq. (14) compares old-task prototypes computed with the then-current shared up-projection A_s against features produced by the final A_s, but no per-task shared snapshots are stored; unresolved drift makes the reported accuracy and the claimed O(l+(N-l)T) inference advantage…","rationale":"The reader's weakest assumption was that a fixed random orthogonal B_s creates a stable shared subspace. I agree that shared-adapter stability is load-bearing, but the sharper issue is a concrete train/test inconsistency: even with B_s fixed, A_s is continuously updated, and the inference procedure in Eq. (14) uses the final A_s for all tasks while prototypes were stored with earlier A_s states. The paper never stores or uses per-task A_s snapshots and never quantifies drift, although Eq. (10) explicitly permits drift in less important dimensions. This is more specific than the general orthogonality concern and directly affects both the correctness of reported accuracies and the claimed inference-complexity reduction. The proposed check would settle it empirically. I do not think this changes the overall conditional status: the method may still be sound if drift is negligible, but the evaluation protocol must be verified before the central claims are accepted. Parameter accounting in Table 1 is a secondary concern; the prototype/shared drift is the most load-bearing issue.","tokens_in":17808,"tokens_out":14459,"duration_ms":170165,"concrete_test":"Modify the released code on ImageNet-R T=20: after each task i, checkpoint A_s^i in addition to the final A_s^T. At inference, compute z^l for task i using A_s^i instead of A_s^T, and compare final accuracy AT and per-task accuracy against the paper's protocol. Also measure shared drift d_i = ||A_s^i - A_s^T||_F / ||A_s^i||_F and feature drift ||z^l_i(A_s^i) - z^l_i(A_s^T)||_2 on held-out old-task data. If switching to per-task snapshots changes AT by more than about 1 point, or if drift grows with task index, the reported results rely on mismatched prototypes; if drift is negligible, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evaluation protocol has a train/test mismatch for old tasks. During training of task t, prototypes P_t are computed with the shared up-projection A_s^t at that time (Algorithm 1, lines 10-32). At inference, Eq. (14) computes z^l through the current (final) shared adapter for every task, then applies the task-specific branch. Algorithm 1 line 31 stores only task-specific (A_t, B_t) and U_t, not a per-task snapshot of A_s. Since A_s is continuously updated, and gradient reassignment in Eq. (10) explicitly allows dimensions with small previous norms to adapt to new tasks, the features z^l used for old tasks at test are produced by a different shared adapter than the one that produced the stored prototypes. No rehearsal data exists to recompute old prototypes, and the paper reports no measurement of A_s drift or of accuracy under per-task shared snapshots. If the correct inference requires per-task A_s^i snapshots, the complexity becomes O(lT+(N-l)T)=O(NT), eliminating the advertised O(l+(N-l)T) advantage. If the final A_s is used instead, old-task predictions compare mismatched features to mismatched prototypes. Either way, the evaluation protocol and the scalability claim need explicit justification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CL-LoRA, a dual-adapter architecture for rehearsal-free class-incremental learning (CIL) with pre-trained vision transformers. A task-shared LoRA branch with a fixed random orthogonal down-projection and a continuously updated up-projection is placed in the first l transformer blocks to accumulate cross-task knowledge, while task-specific LoRA branches with learnable block-wise scaling factors and an orthogonality penalty are used in the remaining blocks. The method is evaluated on CIFAR-100, ImageNet-R, ImageNet-A, and VTAB, reporting state-of-the-art or competitive accuracy with 0.3% trainable parameters and reduced inference complexity O(l+(N-l)T).","tokens_in":18108,"tokens_out":7428,"duration_ms":86377,"significance":"The paper is well-motivated and addresses a real limitation of adapter-based CIL, namely parameter redundancy and the lack of cross-task knowledge sharing. The design of a stable shared low-rank subspace via a fixed random orthogonal down-projection is interesting and is supported by a clear ablation (Table 4) showing the orthogonality property is essential. The experimental protocol is solid in many respects: ten seeds, standard benchmarks, and ablations for each proposed component. If the two load-bearing issues identified below are resolved, the method could be a useful contribution. However, the current evaluation protocol has an unresolved train/test mismatch for old-task prototypes, and the reported parameter efficiency appears to undercount the total stored parameters.","major_comments":[{"comment":"There is a train/test inconsistency for old-task prototypes. During training of task i, prototypes p_i are computed with the shared up-projection A_s^i at that time, but at inference Eq. (14) passes every test sample through the final shared adapter A_s^T, because Algorithm 1 stores only the task-specific (A_i,B_i) and U_i, not per-task snapshots of A_s. Since A_s is continuously updated across tasks and gradient reassignment explicitly allows some dimensions to change, the features fed to the old task-specific branches are produced by a different shared adapter than the one used to compute the stored prototypes. The sentence in Section 4.4 that the prototypes are 'computed using the same adapter combination during training' is therefore not correct for the shared branch. This issue is unmeasured. If the correct procedure requires per-task A_s^i snapshots, the inference complexity becomes O(lT+(N-l)T)=O(NT), which invalidates the O(l+(N-l)T) scalability claim in Section 5.4. The authors should either store per-task snapshots of the shared adapter and revise the complexity claim, or provide direct evidence that A_s drift is negligible, for example by measuring prototype-feature distance or comparing final-adapter vs snapshot-adapter accuracy.","section":"Section 4.4, Eq. (14), Algorithm 1"},{"comment":"The reported parameter efficiency appears to count only the per-task incremental parameters, not the total number of additional parameters used at inference. With the stated setup (r=10, LoRA on Wq and Wv, l=6, N=12, ViT-B/16), the shared branch has 6 blocks x 2 projections x 768x10 = 92,160 trainable A_s parameters, and each new task adds 6 blocks x 2 projections x (768x10 + 10x768) = 184,320 trainable parameters. For T=20, the total is about 3.78 million parameters, which is roughly 4.4% of the ViT-B/16 backbone, not 0.3%. The comparison methods in Table 1 report total stored parameters, so this undercount makes the parameter-efficiency advantage in the abstract, Section 5.2, and Section 5.4 misleading. The table should report the total number of stored parameters at the end of the task sequence, or clearly state and consistently use a per-task metric.","section":"Table 1, Section 5.1, Sections 5.2 and 5.4"}],"minor_comments":[{"comment":"U_t is defined as a concatenation of scalar block-scaling factors, so U_t is a vector and U_t^T U_i is a scalar; the double sum over j,k is ill-defined. The loss should be written as a single Frobenius norm, or the definition of U_t should be expanded to a matrix if the block weights are intended to be per-head or per-dimension.","section":"Eq. (12)"},{"comment":"The knowledge distillation loss is computed by passing both the current and previous shared-adapter features through the current local classifier h_t^phi. This is closer to a feature-consistency regularizer than to standard distillation from a previous task's classifier. The definition of s_{t-1} and the role of the current classifier should be clarified, since the notation suggests a sum over current classes i in C_t for both distributions.","section":"Eq. (9) and Algorithm 1, line 22"}],"recommendation":"major_revision","confidential_remarks":"The prototype-drift issue and the parameter-counting issue both bear directly on the paper's central claims of accuracy and efficiency. The authors should be asked to provide per-task shared-adapter snapshots or drift measurements, and to correct the parameter accounting. The manuscript is otherwise clearly written and the experiments are reproducible in spirit, but these two points need substantive revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one needs a careful look at the inference protocol before you trust Table 1. The dual-adapter architecture is genuinely interesting: a shared LoRA branch in the first l blocks with a fixed random-orthogonal down-projection, plus task-specific LoRA with learnable block-wise weights in later blocks. That combination is new and the ablation evidence is solid. But the evaluation has a train/test mismatch for old tasks that is load-bearing.\n\nIn Algorithm 1, after each task they store only (A_t, B_t) and U_t. Prototypes for task t are computed during training with the shared up-projection A_s at that time. At inference, Eq. (14) computes z^l for every task using the final shared A_s, then applies the task-specific branch. For old tasks, features and prototypes are produced by different shared adapters, and nothing in the paper measures A_s drift or compares against per-task snapshots. The stress-test note is right: if you store snapshots, inference becomes O(NT) and the advertised O(l+(N-l)T) advantage vanishes; if you use the final A_s, you eat the mismatch. Either way, the reported accuracy and the scalability claim need explicit justification. I disagree with the reader's soundness 7; this is more than a notation error. It could be fixable—store snapshots and see if numbers hold—but as written, the central empirical claim is unverified.\n\nCredit where due: the design choices are well-motivated, the ablations support each component, Table 4 is a striking demonstration that the random-orthogonal down-projection is essential, and the 10-seed reproduction of baselines is good practice. The fixed B_s insight is worth citing on its own.\n\nSofter spots: Eq. (12) has a notation error (U_t as a vector makes the double sum ill-defined). The 'scalable' label overstates the case: task-specific parameters still grow with T and the big-O complexity is still O(NT), just with a better constant. No hyperparameter validation is described, though fixed lambdas across datasets helps.\n\nBottom line: the method deserves serious referee time, but a referee must require the authors to resolve the prototype mismatch, either with snapshot inference or with a measurement of A_s drift. I'd send it to review with that as the top major comment. The design idea is citable; the numbers are not, yet.","headline":"Clever dual-LoRA design, but the inference protocol mismatches training for old tasks and the reported accuracy depends on that mismatch.","tokens_in":18638,"tokens_out":7494,"would_cite":true,"duration_ms":84001,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-adapter LoRA design lets a frozen ViT keep old classes while learning new ones, using only 0.3% trainable parameters and no stored exemplars.","keywords":["class-incremental learning","rehearsal-free continual learning","low-rank adaptation","LoRA","task-shared adapters","task-specific adapters","knowledge distillation","orthogonal projection"],"falsifier":"Run CL-LoRA on a benchmark where each new task changes the input distribution drastically (e.g., VTAB's cross-domain splits); if the fixed shared subspace cannot accommodate the shift, accuracy on earlier tasks should decay despite the distillation loss, which would show the method only handles moderate shifts. More directly, replace $B_s$ with a fixed random non-orthogonal matrix on ImageNet-R with $T=20$: the paper's Table 4 predicts a collapse from about 79% to under 3% final accuracy, so a failure to collapse would falsify the orthogonality claim.","tokens_in":17542,"feed_emoji":"🧠","tokens_out":7849,"duration_ms":77784,"temperature":0.7,"pith_summary":"The paper asks whether a pre-trained vision transformer can keep learning new classes, one task at a time, without storing any examples from earlier tasks. Its answer is CL-LoRA, a dual-adapter scheme that splits the transformer into early blocks that share one continually updated low-rank adapter and later blocks where each task gets its own adapter. The shared adapter fixes a random orthogonal down-projection matrix and only trains the up-projection, so knowledge from all tasks accumulates in one stable low-dimensional subspace; the task-specific adapters add learnable per-block scaling with an orthogonality penalty to keep tasks from interfering. On four standard benchmarks the result is state-of-the-art or competitive accuracy with only 0.3% trainable parameters, and inference cost drops from $O(NT)$ to $O(l + (N-l)T)$ because the shared blocks are run once. If the design holds, it shows that cross-task knowledge sharing and parameter efficiency can be combined in rehearsal-free continual learning.","feed_headline":"Shared LoRA subspace beats task-by-task adapters at 0.3% parameters","feed_subtitle":"Rehearsal-free class-incremental learning keeps old knowledge while adding just 0.3% trainable weights.","key_machinery":"The machinery is a dual-adapter transformer: blocks 1 through $l$ host a task-shared LoRA branch ($A_s B_s$, with $B_s$ a fixed random orthogonal matrix from an SVD of a Gaussian matrix and $A_s$ zero-initialized and continuously updated), and blocks $l+1$ through $N$ host task-specific LoRA branches ($A_t B_t$) multiplied by learnable block-wise scaling factors $\\mu_t$ and constrained by an orthogonality loss between tasks. Around this sit two stabilizers: early-exit knowledge distillation at block $l$, whose gradient is reassigned proportionally to the $\\ell^2$ norms of the previous shared up-projection's weight vectors, and the orthogonality regularization on the block-weight vectors. The fixed $B_s$ plays the load-bearing role: it projects inputs into a low-dimensional space whose geometry is preserved because $B_s B_s^\\top = I$, giving the shared up-projection $A_s$ a stable coordinate system in which to accumulate knowledge.","core_discovery":"In class-incremental learning without rehearsal, the paper claims that a single shared low-rank adapter can carry cross-task knowledge if its down-projection is a fixed random orthogonal matrix $B_s$ (so $B_s B_s^\\top = I$) and only the up-projection $A_s$ is trained, provided the gradient of a knowledge-distillation loss at the early-exit point is reassigned by the $\\ell^2$ norms of the previous task's $A_s$ vectors. The shared adapter sits in the first $l$ transformer blocks, which learn general features, while task-specific adapters with learnable block-wise scaling factors (regularized to be nearly orthogonal between tasks) occupy the remaining blocks. With this split, CL-LoRA achieves the best reported accuracy on ImageNet-R, ImageNet-A, and VTAB, and competitive accuracy on CIFAR-100, at 0.3% trainable parameters; the shared blocks make inference cheaper because each test sample passes through them only once, reducing forward passes from $O(NT)$ to $O(l + (N-l)T)$.","pith_inferences":["The same fixed-orthogonal down-projection idea could be tested as a drop-in replacement for the shared branches in other PEFT continual learners (e.g., prompts or adapters), where the shared component is currently trained without a fixed subspace.","The early-exit position $l$ is currently fixed at 6 for all datasets; the paper suggests it could be tuned from the first few tasks, which would make the method adaptive to datasets with different degrees of distribution shift.","Because the shared subspace is built into the architecture rather than enforced by replay, the approach might extend to online or blurry-boundary CIL where task identity is unknown during training; the paper notes this possibility but does not test it.","One testable prediction: if the orthogonality of $B_s$ matters mainly for geometry preservation, replacing the SVD construction with a QR-based orthogonalization should give identical accuracy; a small ablation could confirm this cheaply."],"forward_implications":["Rehearsal-free CIL can work with a frozen backbone plus a tiny trainable fraction: 0.3% of backbone parameters suffice for strong accuracy on four benchmarks.","Inference on a stream of $T$ tasks costs $O(l + (N-l)T)$ adapter forward passes per test sample instead of $O(NT)$, so the method scales to longer task sequences.","A fixed random orthogonal down-projection outperforms a trainable down-projection at every tested split position $l$, in line with the asymmetry analysis of LoRA (reference [61]).","Very low ranks suffice: $r=1$ already gives strong results on CIFAR-100, and adding more projection matrices ($W_q$, $W_k$, $W_v$) does not reliably help.","The early-exit knowledge distillation plus gradient reassignment is what keeps the shared adapter from drifting to the most recent task; removing it degrades accuracy, especially on longer sequences."],"supporting_citations":[{"why":"Supplies the theoretical motivation that a random untrained down-projection $B$ performs nearly as well as a fine-tuned one, justifying the fixed shared down-projection.","marker":"[61]"},{"why":"Defines LoRA as the low-rank decomposition $(A, B)$ that the shared and task-specific adapters are built from.","marker":"[20]"},{"why":"Baseline InfLoRA, an interference-free task-specific LoRA method that the paper's shared-plus-specific design extends; the comparison shows the gain from sharing.","marker":"[26]"},{"why":"Baseline O-LoRA using orthogonality constraints on task-specific adapters, contrasted with CL-LoRA's fixed shared orthogonal subspace.","marker":"[47]"},{"why":"Baseline EASE, a strong adapter-based rehearsal-free CIL method whose task-specific expansion is what CL-LoRA's shared blocks reduce.","marker":"[58]"},{"why":"Baseline RanPAC, which also uses random projections for rehearsal-free CIL and is the method CL-LoRA beats on ImageNet-R/A while using far fewer parameters.","marker":"[34]"},{"why":"Supplies the gradient-reweighting idea that CL-LoRA adapts into gradient reassignment for shared knowledge preservation.","marker":"[13]"},{"why":"Shows vision transformers process general features early and specific features late, motivating the shared-in-first-blocks / specific-in-later-blocks split.","marker":"[37]"}],"fun_headline_variants":["CL-LoRA: shared and specific adapters cut CIL cost to 0.3%","One shared LoRA plus task adapters: 0.3% parameters for CIL","Rehearsal-free CIL with shared LoRA: 0.3% weights, faster inference","Dual LoRA: shared knowledge, per-task adapters, 0.3% trainable","CL-LoRA halves forward passes, keeps accuracy with 0.3% params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a fixed random orthogonal down-projection matrix $B_s$ creates a stable low-dimensional subspace in which shared knowledge can accumulate without forgetting; if that orthogonality or fixedness is removed, the shared adapter stops working, since random $B_s$ collapses accuracy to roughly 9% on CIFAR-100.","fun_headline_variants_meta":{"raw":{"variants":["CL-LoRA: shared and specific adapters cut CIL cost to 0.3%","One shared LoRA plus task adapters: 0.3% parameters for CIL","Rehearsal-free CIL with shared LoRA: 0.3% weights, faster inference","Dual LoRA: shared knowledge, per-task adapters, 0.3% trainable","CL-LoRA halves forward passes, keeps accuracy with 0.3% params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000976,"raw_usage":{"total_tokens":4178,"prompt_tokens":1006,"completion_tokens":3172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3050}},"tokens_in":622,"tokens_out":3172,"duration_ms":23445,"temperature":1.0,"reasoning_tokens":3050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:12:49.513976+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CL-LoRA on a benchmark where each new task changes the input distribution drastically (e.g., VTAB's cross-domain splits); if the fixed shared subspace cannot accommodate the shift, accuracy on earlier tasks should decay despite the distillation loss, which would show the method only handles moderate shifts. More directly, replace $B_s$ with a fixed random non-orthogonal matrix on ImageNet-R with $T=20$: the paper's Table 4 predicts a collapse from about 79% to under 3% final accuracy, so a failure to collapse would falsify the orthogonality claim.","supporting_citations":[{"cited_title":"Inflora: Interference-free low-rank adaptation for continual learning","cited_arxiv_id":null,"evidence_quote":"Baseline InfLoRA, an interference-free task-specific LoRA method that the paper's shared-plus-specific design extends; the comparison shows the gain from sharing."},{"cited_title":"Or- thogonal subspace learning for language model continual learning","cited_arxiv_id":null,"evidence_quote":"Baseline O-LoRA using orthogonality constraints on task-specific adapters, contrasted with CL-LoRA's fixed shared orthogonal subspace."},{"cited_title":"Expandable subspace ensemble for pre-trained model-based class-incremental learning","cited_arxiv_id":null,"evidence_quote":"Baseline EASE, a strong adapter-based rehearsal-free CIL method whose task-specific expansion is what CL-LoRA's shared blocks reduce."},{"cited_title":"Ranpac: Ran- dom projections and pre-trained models for continual learn- ing","cited_arxiv_id":null,"evidence_quote":"Baseline RanPAC, which also uses random projections for rehearsal-free CIL and is the method CL-LoRA beats on ImageNet-R/A while using far fewer parameters."},{"cited_title":"Gradient reweighting: Towards imbalanced class-incremental learning","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-reweighting idea that CL-LoRA adapts into gradient reassignment for shared knowledge preservation."},{"cited_title":"How do vision transformers work? International Conference on Learning Representa- tions, 2022","cited_arxiv_id":null,"evidence_quote":"Shows vision transformers process general features early and specific features late, motivating the shared-in-first-blocks / specific-in-later-blocks split."}],"review_version":1}