Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SplitLoRA derives the optimal per-module subspace size for gradient-projection continual learning, beating fixed-threshold methods by 2–5%.

desk verdict Broken theory, real empirical gains: SplitLoRA's optimality derivation does not hold up, but the method is a plausible heuristic. read the letter →

arxiv 2505.22370 v3 pith:P4R2JS6O submitted 2025-05-28 cs.LG cs.AI

Haomiao Qiu , Miao Zhang , Ziyue Qiao , Weili Guan , Min Zhang , Liqiang Nie This is my paper
classification cs.LGcs.AI
keywords continuallearninglow-rankadaptationgradientprojectionstability-plasticitytrade-offsingularvaluedecompositionparameter-efficientfine-tuningcatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continual learning models that use gradient projection must decide how many directions of the old-task gradient space to leave free for a new task. SplitLoRA argues that this 'minor subspace' size should be derived per module from a stability-plasticity trade-off, not fixed by a global energy threshold, and it derives a closed-form objective for the optimal size. The method works inside the LoRA parameter-efficient fine-tuning framework: each task's LoRA is built on the minor subspace of previous tasks, so new knowledge is learned in directions that interfere least with old knowledge. If the derivation is correct, continual learners can pick up 2–5% final accuracy on standard vision benchmarks while adding almost no memory or compute.

What carries the argument

The central object is the per-module objective $k \mapsto (t-1)\epsilon_t(k) - \alpha k/d$, where $\epsilon_t(k) = \sum_{i=d-k+1}^{d}\sigma_i / \sum_{i=1}^{d}\sigma_i$ is the fraction of old-gradient energy lying in the $k$ smallest singular directions. Minimizing this objective over $k \in [1,d]$ produces the optimal minor-subspace size $k_t^*$ that balances expected stability loss against expected plasticity loss. The companion mechanism is the projection construction $A_t = \hat U^k_t R$, with $\hat U^k_t$ the last $k$ left singular vectors of the old-task gradient and $R$ a fixed random Gaussian matrix; keeping $A_t$ fixed while optimizing only $B_t$ ensures every LoRA update stays inside the chosen minor subspace.

What would settle it

On a real task stream, record the values $|\langle \Delta W_t, u_i v_i^\top\rangle|$ for each of the $d$ singular directions of $G^{\mathrm{old}}_t$; if their spread is large relative to their mean, the uniform-projection assumption underlying Eq. (12) fails and the derived $k^*$ is not the true minimizer of expected loss.

Watch

Extended reading notes

Core claim

SplitLoRA claims that the size of the minor subspace used for gradient projection should be chosen separately for every LoRA module, rather than by a uniform threshold on cumulative squared singular values. Starting from an $L$-smoothness bound on the total loss change, it models the expected stability loss as $E[L^S_t] = -(t-1)\epsilon_t(k_t)\langle \Delta W_t, G^{\mathrm{old}}_t\rangle$ and the expected plasticity loss as $E[L^P_t] = -(k_t/d)\langle \Delta W_t, G_t\rangle$, under the assumption that the new task update is uniformly distributed across feature directions. Treating the ratio $\alpha = -\langle \Delta W_t, G_t\rangle / \langle \Delta W_t, G^{\mathrm{old}}_t\rangle$ as a fixed hyperparameter, minimizing the sum yields an approximate optimal $k_t^*$ for each module; the method then fixes $A_t = \hat U^k_t R$ with a random Gaussian $R$ and trains only $B_t$, confining LoRA updates to the low-interference minor subspace. The paper reports state-of-the-art final and cumulative accuracy on ImageNet-R, CIFAR-100, and DomainNet, with gains of roughly 2–5% over existing continual learning methods.

Load-bearing premise

The derivation assumes the new task's update direction is uniformly spread across all feature directions, so its expected projection is the same on every singular direction; if real task gradients concentrate on a few directions, the computed optimal subspace size has no theoretical basis.

Editorial extensions

If this is right

  • Different layers of the model receive different subspace sizes; the paper observes that shallower layers keep larger minor subspaces and deeper layers smaller ones, since changes in deep layers affect old-task stability more.
  • As the number of tasks grows, the optimal minor subspace shrinks, so the method automatically shifts toward stability in later tasks.
  • The method adds almost no cost: one extra forward pass per task to update the stored gradient subspace, with memory and runtime close to the fixed-threshold baseline.
  • A single hyperparameter $\alpha$ controls the stability-plasticity trade-off, and reported accuracy is stable across $\alpha \in \{1,5,10,20,30\}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the uniformity assumption behind Theorem 4.2 can be checked directly by recording the empirical projections of $\Delta W_t$ onto the singular directions of $G^{\mathrm{old}}_t$; if those projections are highly non-uniform, the expected-loss formulas hold only approximately.
  • Editorial inference: since $\alpha$ is defined as a ratio of two inner products that are observable during training, one could estimate it online per task instead of fixing it, turning Eq. (14) from a hyperparameter into a measured quantity.
  • Editorial inference: the same splitting argument should transfer to other parameter-efficient modules such as adapters or prompts, because only the low-rank parametrization changes, so a natural test is to replace the LoRA $A$ matrix with a prompt or adapter projection into the minor subspace.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SplitLoRA, a continual learning method that combines LoRA with gradient projection. For each LoRA module, the method computes the SVD of an averaged 'old task' gradient space, selects the size k of the minor (low-singular-value) subspace by minimizing an objective that is claimed to balance stability and plasticity losses, and then initializes the frozen LoRA matrix A from a random projection of that minor subspace so that updates remain confined to it. The authors support the approach with a theoretical analysis (Proposition 4.1, Theorem 4.2) and report state-of-the-art results on ImageNet-R, CIFAR-100, DomainNet, and a seven-benchmark suite. The central claim is that the proposed k* is the theoretically optimal minor-subspace size.

Significance. If the theoretical derivation were sound, the paper would address a real limitation of fixed-threshold gradient-projection methods such as InfLoRA: different layers do appear to benefit from different subspace sizes, as the paper's own Figure 4 illustrates. The empirical results are strong and fairly broad: consistent FAA/CAA gains over several strong baselines on three standard datasets, an additional seven-benchmark comparison, an efficiency analysis (Table 6), and ablations of A initialization and the hyperparameter α (Tables 4 and 5). The method is simple and reproducible. However, the theoretical centerpiece—the derivation of an optimal k*—is not supported by the proofs as written, and the resulting k* depends on a hand-tuned hyperparameter α. The contribution is therefore better characterized as a well-performing heuristic with an invalid optimality argument than as a derived optimum.

major comments (3)
  1. [§4.2, Eq. (7) and Appendix A.1] The proof of Proposition 4.1 does not establish the stated inequality. L-smoothness of each L_i gives L_i(W_t) ≤ L_i(W_{t-1}) + ⟨∇L_i(W_{t-1}), ΔW_t⟩ + (L/2)‖ΔW_t‖², and summing over i = 1..t yields an upper bound with +(t−1)⟨ΔW_t, G_old_t⟩ + ⟨ΔW_t, G_t⟩ + (tL/2)‖ΔW_t‖². The proof in Appendix A.1 instead produces negative inner products and a (t−1)L/2 coefficient with no justification; the sign flip is exactly what allows the two terms to be labeled 'Stability Loss' and 'Plasticity Loss' in Eqs. (8)–(9). Since the optimization objective in Eq. (13) is built from these mislabeled terms, the derivation is invalid at its foundation.
  2. [§4.2, Theorem 4.2 and Appendix A.2] The uniform-direction assumption is incompatible with the derived nonzero expectations. If ΔW_t is uniformly distributed over directions in a rotationally invariant sense, then for any fixed matrix V, E[⟨ΔW_t, V⟩] = 0, so the right-hand sides of Eqs. (11) and (12) would vanish. The proof's step 'E[⟨ΔW_t, u_i v_i^T⟩] = c for all i' and then 'c = ⟨ΔW_t, G_old_t⟩ / Σσ_i' replaces a true expectation with the realized, data-dependent value of the inner product. Thus the uniform assumption does no work, and Theorem 4.2 does not support the objective in Eq. (13).
  3. [§4.2, Eqs. (14)–(15) and §5.3] The claimed 'optimal' k* is not derived in a parameter-free way. The ratio α in Eq. (14) is introduced as a quantity that the text says varies during training, and it is then fixed as a hyperparameter (α = 20), with Table 4 showing the resulting performance across a range of values. Consequently, Eq. (15) minimizes a hand-specified objective whose trade-off coefficient is tuned on the benchmark tasks; the Abstract's and Section 1's claim that SplitLoRA 'derives the optimal partition of the gradient space' is not supported. The paper's own Limitation paragraph concedes that the constant-ratio assumption 'may not be the most principled or optimal solution.'
minor comments (6)
  1. [Appendix A.1, Lemma A.1] The lemma's orthogonality condition is not satisfied by updates projected onto the minor subspace, which by construction has nonzero components of G_old; moreover, the lemma is never invoked in the proof of Proposition 4.1, so its role is unclear.
  2. [Proposition 4.1 and Appendix A.1] The notation G_t is used inconsistently: the proposition statement says G_t = ∇L_t(W_t), while the proof sums gradients ∇L_i(W_{t-1}) evaluated at W_{t-1}; for a uniform Taylor expansion, the current-task term should also be ∇L_t(W_{t-1}).
  3. [Eq. (6)] The denominator of the cumulative singular-value threshold is written as 'P σ2', which is ambiguous; it should read Σ_{i=1}^d σ_i^2.
  4. [Eq. (11)] The expression E[L^S_t] retains the random variable ΔW_t on the right-hand side; as written, the expectation is not well defined unless ΔW_t is treated as fixed or the expectation is explicitly over the data distribution.
  5. [Table 3] The header 'FAA↑CAA↑' omits separators and units; the table would benefit from standard formatting consistent with Tables 1 and 2.
  6. [References] Reference [48] is cited as an anonymous under-review ICLR submission; citing an anonymous work in this form is not ideal for a published reference list.

Circularity Check

2 steps flagged · score 6.0 of 10

The claimed derivation of the optimal subspace size k* reduces to a tuned trade-off weight α and to a fixed-alignment assumption that contradicts the paper's own uniform-direction model; the empirical benchmark gains are external and not circular.

  1. fitted input called prediction [Section 4.2, Eqs. (14)–(15) and the paragraph after Eq. (16)]
    "Noting that ∆Wt and Gt gradually change as training progresses, solving this optimization problem is challenging. To simplify this, we introduce a ratio parameter α: α=−⟨∆Wt,G t⟩/⟨∆Wt,G old t ⟩. ... Since both ∆Wt and Gt change during training, α also varies dynamically. Meanwhile, the update subspace must be determined before training begins for task t. To resolve this mismatch, we treat α as a fixed hyperparameter throughout the learning process. In our experiments, α was set as a hyperparameter, with α= 20 as a general choice."

    The quantity α is not an independent constant: Eq. (14) defines it as the ratio of the alignment of the update with the new-task gradient and the old-task gradient, i.e., the same stability/plasticity quantities that the objective in Eq. (15) is meant to balance, and the update ∆Wt is itself shaped by the subspace size kt being optimized. The paper then fixes α=20 after benchmark experiments (Table 4) and calls the minimizer of Eq. (15) the optimal k*. Thus the claimed 'derived optimum' is actually the minimizer of a criterion whose trade-off weight was tuned on the evaluation data; the optimality claim is equivalent to a hyperparameter choice, not a first-principles result.

  2. other [Theorem 4.2 (Section 4.2) and Appendix A.2, Eqs. (11)–(12) and (18)–(19)]
    "Since the update direction of the new task is unknown, we assume that it is uniformly distributed across all directions. that is to say, ∆Wt has the same expected projection value across different feature directions of Gt, we provide the expected values of the stability loss : E[LS t (Wt)] =−(t−1)ϵ t(kt)⟨∆Wt,G old t ⟩ ... Under the uniform distribution assumption, all expected projections are equal: E[⟨∆Wt,u iv⊤ i ⟩F ] =c,∀i."

    A direction uniformly distributed over the sphere is rotationally invariant, so for every fixed matrix V one has E[⟨∆Wt,V⟩]=0. Under the theorem's own assumption, both E[L^S_t] and E[L^P_t] must vanish. The nonzero expressions in Eqs. (11)–(12) are obtained only by keeping ⟨∆Wt,G^old_t⟩ and ⟨∆Wt,G_t⟩ inside the expectation as fixed constants, and the proof sets c=⟨∆Wt,G^old_t⟩/Σσ_i and substitutes it back. The 'derived' expectation is therefore identical to an assumed nonzero alignment, which is precisely the data-dependent quantity the uniform assumption was introduced to remove. Eq. (15) is consequently not a consequence of the stated assumption; it presupposes the alignment it claims to derive.

full rationale

SplitLoRA's empirical evaluation is self-contained: comparisons are against external baselines and no load-bearing self-citation appears. The circularity is located in the claimed derivation of k*. Eq. (14) defines the trade-off α as a ratio of the very alignment terms that the objective balances, while Eq. (15) uses that same α to select the subspace size kt that determines the projected update; the paper then fixes α=20 after experiments, making the 'optimal' k* a tuned threshold rather than a derived quantity. Independently, Theorem 4.2 assumes a uniformly distributed update direction, under which E[⟨∆Wt,V⟩]=0 for every fixed V, so the expected stability and plasticity losses vanish; the nonzero formulas in Eqs. (11)–(12) are obtained by treating ⟨∆Wt,G^old_t⟩ and ⟨∆Wt,G_t⟩ as fixed constants and defining c=⟨∆Wt,G^old_t⟩/Σσ_i, i.e., by assuming the alignment the theorem claims to derive. Eq. (15) therefore has no valid basis under its own assumption. The reported accuracy gains are external evidence that the heuristic may work, but the central 'first-principles' optimality claim reduces to a tuned hyperparameter and a contrary assumption, giving partial circularity (score 6).

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on several unproven assumptions. The most fragile are the uniform-direction assumption, which makes the expectation formulas difficult to justify, and the constant-alpha assumption, which is acknowledged as a limitation. The method also relies on a hand-set hyperparameter alpha, so the 'optimal' subspace size is not fully derived from first principles.

free parameters (1)
  • alpha (α) = 20 (default; Table 4 explores 1, 5, 10, 30)
    Introduced in Eq. (14) as the ratio of inner products, but treated as a fixed hyperparameter because gradients change during training. The optimal subspace size k* depends directly on alpha in Eq. (15), so the claimed optimality is conditional on a hand-set constant.
assumptions (4)
  • domain assumption Each task loss L_i is L-smooth.
    Used in Proposition 4.1 and its proof to obtain the quadratic upper bound. Standard in optimization theory, but it is an assumption about the loss landscape that is not verified for the actual neural network.
  • ad hoc to paper The update direction of the new task is uniformly distributed across all feature directions.
    Stated in Theorem 4.2 to derive E[L^S] and E[L^P]. If the update direction were truly uniform, expected inner products with fixed gradients would be zero, making the derived formulas vacuous. This is a critical modeling assumption with no empirical support.
  • ad hoc to paper The ratio α = -<ΔW_t,G_t>/<ΔW_t,G_old> is constant during training.
    Needed to make Eq. (15) tractable. The paper acknowledges in the Limitation section that this ratio is assumed fixed, calling it 'not the most principled or optimal solution.'
  • domain assumption Second-order terms in the Taylor expansion of gradients are negligible (Lemma A.1).
    Used to claim that gradients of previous tasks remain unchanged under orthogonal updates. This requires small learning rates and low curvature, which is not guaranteed in training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting." pith.science (2026). https://pith.science/paper/P4R2JS6O

@misc{pith2026250522370,
  author       = {Pith},
  title        = {Pith review of: SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4R2JS6O}},
  note         = {Machine review of arXiv:2505.22370}
}
read the original abstract

Continual Learning requires a model to learn multiple tasks in sequence while maintaining both stability:preserving knowledge from previously learned tasks, and plasticity:effectively learning new tasks. Gradient projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two orthogonal subspaces: a primary subspace and a minor subspace. New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge. However, existing Gradient Projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space. In this work, we consider a continual learning paradigm based on Low-Rank Adaptation, which has gained considerable attention due to its efficiency and wide applicability, and propose a novel approach for continual learning, called SplitLoRA. We first provide a theoretical analysis of how subspace partitioning affects model stability and plasticity. Informed by this analysis, we then introduce an effective method that derives the optimal partition of the gradient space for previously learned tasks. This approach effectively balances stability and plasticity in continual learning. Experimental results on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2505.22370 by the authors.

Figure 1
Figure 1. An overview of our proposed SplitLoRA. During the learning of the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Variation of the performance of different methods during the learning of ImageNet-R. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The impact of α on the stability and plasticity of the model in continual learning. As α increases, stability decreases (higher forgetting) while plasticity improves, illustrating the trade-off between retaining past knowledge and adapting to new tasks. 5.3 Hyperparameter Analysis and Discussion We study the effect of the hyperparameter α on continual learning performance. As shown in [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: We recorded the evolution of the minor subspace size during training on ImageNet-R with [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Under Adam, feeding a modified gradient into both moment accumulators cancels the intended continual-learning protection; feeding only the first moment preserves it.

Reference graph

Works this paper leans on

59 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Continual lifelong learning with neural networks: A review.Neural Networks, 113:54–71, 2019

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural Networks, 113:54–71, 2019

    2019

  2. [2]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations, 2022

    2022

  3. [3]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InProceedings of the International Conference on Machine Learning, pages 2790–2799, 2019

    work page 2019

  4. [4]

    Belongie, Bharath Hariharan, and Ser-Nam Lim

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. InProceedings of the European Conference on Computer Vision, pages 709–727, 2022

    work page 2022

  5. [5]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022

    work page 2022

  6. [6]

    Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909–11919, 2023

    work page 2023

  7. [7]

    A unified continual learning framework with general parameter-efficient tuning

    Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11449–11459, 2023

    work page 2023

  8. [8]

    Inflora: Interference-free low-rank adaptation for continual learning

    Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638–23647, 2024

    work page 2024

Show all 59 references
  1. [9]

    Visual prompt tuning in null space for continual learning.arXiv preprint arXiv:2406.05658, 2024

    Yue Lu, Shizhou Zhang, De Cheng, Yinghui Xing, Nannan Wang, Peng Wang, and Yan- ning Zhang. Visual prompt tuning in null space for continual learning.arXiv preprint arXiv:2406.05658, 2024

    2024 arXiv

  2. [10]

    Taia: Large language models are out-of-distribution data learners.arXiv preprint arXiv:2405.20192, 2024

    Shuyang Jiang, Yusheng Liao, Ya Zhang, Yanfeng Wang, and Yu Wang. Taia: Large language models are out-of-distribution data learners.arXiv preprint arXiv:2405.20192, 2024

    2024 arXiv

  3. [11]

    Knowledge neurons in pretrained transformers.arXiv preprint arXiv:2104.08696, 2021

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers.arXiv preprint arXiv:2104.08696, 2021

    2021 arXiv

  4. [12]

    Transformer feed-forward layers are key-value memories.arXiv preprint arXiv:2012.14913, 2020

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories.arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  5. [13]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors,Proceedings of the Conference on Empirical Methods in Natural Language Processing, pa...

    2021

  6. [14]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4582–4597, 2021

    2021

  7. [15]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. InProceedings of the Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 1–9, 2022. 10

    2022

  8. [16]

    Adapterbias: Parameter-efficient token-dependent representation shift for adapters in nlp tasks

    Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-Yi Lee. Adapterbias: Parameter-efficient token-dependent representation shift for adapters in nlp tasks. InFindings of the Association for Computational Linguistics, pages 2608–2621, 2022

    2022

  9. [17]

    Compacter: Efficient low-rank hypercomplex adapter layers

    Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. InAdvances in Neural Information Processing Systems, pages 1022–1035, 2021

    2021

  10. [18]

    Adaptformer: Adapting vision transformers for scalable visual recognition.Advances in Neural Information Processing Systems, pages 16664–16678, 2022

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition.Advances in Neural Information Processing Systems, pages 16664–16678, 2022

    2022

  11. [19]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. InICML, pages 3987–3995, 2017

    2017

  12. [20]

    Continual learning with node-importance based adaptive group sparse regularization.Advances in Neural Information Processing Systems, pages 3647–3658, 2020

    Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization.Advances in Neural Information Processing Systems, pages 3647–3658, 2020

    2020

  13. [21]

    Memory aware synapses: Learning what (not) to forget

    Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. InECCV, pages 139–154, 2018

    2018

  14. [22]

    Overcoming catastrophic forgetting in neural networks.PNAS, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.PNAS, 114(13):3521–3526, 2017

    2017

  15. [23]

    Online continual learning with maximal interfered retrieval

    Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pages 11849–11860, 2019

    2019

  16. [24]

    Gradient based sample selection for online continual learning

    Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. InAdvances in Neural Information Processing Systems, pages 11816–11825, 2019

    2019

  17. [25]

    Exploring example influence in continual learning.Advances in Neural Information Processing Systems, pages 27075–27086, 2022

    Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning.Advances in Neural Information Processing Systems, pages 27075–27086, 2022

    2022

  18. [26]

    Loss decoupling for task-agnostic continual learning

    Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems, 2023

    2023

  19. [27]

    Progressive neural networks.arXiv preprint arXiv:1606.04671, 2016

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks.arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  20. [28]

    Steven C. Y . Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647–13657, 2019

    2019

  21. [29]

    Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting

    Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. InProceedings of the International Conference on Machine Learning, pages 3925–3934, 2019

    2019

  22. [30]

    Continual learning of context-dependent processing in neural networks.Nature Machine Intelligence, 1(8):364–372, 2019

    Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks.Nature Machine Intelligence, 1(8):364–372, 2019

    2019

  23. [31]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. InAISTATS, pages 3762–3773, 2020

    2020

  24. [32]

    Gradient projection memory for continual learning

    Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. InICLR, 2021. 11

    2021

  25. [33]

    Trgp: Trust region gradient projection for continual learning.arXiv preprint arXiv:2202.02931, 2022

    Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning.arXiv preprint arXiv:2202.02931, 2022

    2022 arXiv

  26. [34]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022

    2022

  27. [35]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conf...

    2021

  28. [36]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186, 2019

    2019

  29. [37]

    Transfer without forgetting

    Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Si- mone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In Proceedings of the European Conference on Computer Vision, pages 692–709, 2022

    2022

  30. [38]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19068–19079, 2023

    2023

  31. [39]

    Introducing language guidance in prompt- based continual learning

    Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt- based continual learning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11...

    2023

  32. [40]

    Personalized federated continual learning via multi-granularity prompt

    Hao Yu, Xin Yang, Xin Gao, Yan Kang, Hao Wang, Junbo Zhang, and Tianrui Li. Personalized federated continual learning via multi-granularity prompt. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4023–4034, 2024

    2024

  33. [41]

    Orthogonal subspace learning for language model continual learning.arXiv preprint arXiv:2310.14152, 2023

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning.arXiv preprint arXiv:2310.14152, 2023

    2023 arXiv

  34. [42]

    Gradient-based editing of memory examples for online task-free continual learning.NeurIPS, 34:29193–29205, 2021

    Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing of memory examples for online task-free continual learning.NeurIPS, 34:29193–29205, 2021

    2021

  35. [43]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv preprint arXiv:2308.03303, 2023

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning.arXiv preprint arXiv:2308.03303, 2023

    2023 arXiv

  36. [44]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. InECCV, pages 631–648, 2022

    2022

  37. [45]

    Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.arXiv preprint arXiv:2310.07234, 2023

    Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.arXiv preprint arXiv:2310.07234, 2023

    2023 arXiv

  38. [46]

    Evolving parameterized prompt memory for continual learning

    Muhammad Rifki Kurniawan, Xiang Song, Zhiheng Ma, Yuhang He, Yihong Gong, Yang Qi, and Xing Wei. Evolving parameterized prompt memory for continual learning. InAAAI, volume 38, pages 13301–13309, 2024

    2024

  39. [47]

    Vector quantization prompting for continual learning

    Li Jiao, Qiuxia Lai, Yu Li, and Qiang Xu. Vector quantization prompting for continual learning. arXiv preprint arXiv:2410.20444, 2024

    2024 arXiv

  40. [48]

    S-lora: Scalable low-rank adaptation for class incremental learning.Under review at International Conference on Learning Representations (ICLR), 2025

    Anonymous. S-lora: Scalable low-rank adaptation for class incremental learning.Under review at International Conference on Learning Representations (ICLR), 2025. 12

    2025

  41. [49]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. InProceedings of the IEEE/CVF International Co...

    2021

  42. [50]

    Learning multiple layers of features from tiny images.Master’s thesis, University of Tront, 2009

    A Krizhevsky. Learning multiple layers of features from tiny images.Master’s thesis, University of Tront, 2009

    2009

  43. [51]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406–1415, 2019

    2019

  44. [52]

    Continual diffusion: Continual customization of text-to-image diffusion with c-lora.archiveEprint: 2304.06027, 2024

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora.archiveEprint: 2304.06027, 2024

    2024 arXiv

  45. [53]

    Slca: Slow learner with classifier alignment for continual learning on a pre-trained model

    Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. InICCV, 2023

    2023

  46. [54]

    Imagenet-21k pretraining for the masses

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. InNeurIPS, 2021

    2021

  47. [55]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2018

    2018

  48. [56]

    Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. InICLR, 2022

    2022

  49. [57]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021

    2021

  50. [58]

    Continual learning with pre-trained models: A survey.arXiv preprint arXiv:2401.16386, 2024

    Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey.arXiv preprint arXiv:2401.16386, 2024. 13 A Appendix A.1 Proof of Proposition 4.1 Before proving Proposition 4.1, we first establish a supporting lemma. ...

    2024 arXiv

  51. [59]

    Last-Acc

    The updated parameter is given by: Wt =W j −η˜gt. Further assume that the second-order term ηHj ˜gt in the Taylor expansion of ∇Lj can be ignored. Then, the gradient of task j remains unchanged: ∇Lj(Wt) =∇L j(Wj). Proof. Since Lj is twice-differentiable, we apply the first-ord...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.