Pith. sign in

REVIEW 4 major objections 5 minor 54 references

LoRA-Based Continual Learning with Constraints on Critical Parameter Changes

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that under orthogonal LoRA tuning the parameters most critical to previous tasks still drift, and that freezing the top-p important parameter matrices—via LoRAC-IPC, an orthogonal LoRA composition with importance-based…

desk verdict Solid experiments and a plausible combination of known LoRA ideas, but the freezing mechanism is under-specified: 'frozen' matrices may still change through global omega updates, so the central IPC claim needs a clearer update rule. read the letter →

arxiv 2504.13407 v1 pith:I7CPU6ZV submitted 2025-04-18 cs.CV cs.AI

Shimou Ling , Liang Zhang , Jiangwei Zhao , Lili Pan , Hongliang Li This is my paper · ORCID
classification cs.CVcs.AI
keywords continuallearningclass-incrementalcatastrophicforgettingLoRAlow-rankadaptationorthogonalcompositionparameterimportancerehearsal-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Under orthogonal LoRA tuning, the parameters most critical to earlier tasks still change noticeably once later tasks are learned, so orthogonal constraints alone do not stop forgetting. The paper proposes LoRAC-IPC: compose sequentially learned LoRA modules on a pre-trained Vision Transformer, enforce orthogonality among their subspaces through QR decomposition and a regularization loss, and after each task freeze the top-p parameter matrices ranked by a smoothed sensitivity-uncertainty importance score. On several rehearsal-free continual learning benchmarks the method reports the best or near-best accuracy, with headline gains of 6.35% higher accuracy and 3.24% lower forgetting on Split CIFAR-100 with a Sup-21K backbone. If correct, this gives a simple memory-free recipe for keeping pre-trained model fine-tuning stable across long task sequences.

What carries the argument

The load-bearing identity is the LoRA composition $W_t = W_0 + \omega_1\Delta W_1 + \cdots + \omega_t\Delta W_t$, with each $\Delta W_\tau$ reparameterized through QR decomposition as $\Delta W_\tau = Q_\tau K_\tau$, where the columns of $Q_\tau$ are orthonormal. A regularization loss $\mathcal{L}_{\mathrm{ortho}}(\tilde{Q}_t) = \|\tilde{Q}_t^\top \tilde{Q}_t - I\|^2$ on the concatenated matrix $\tilde{Q}_t = [Q_1,\ldots,Q_t]$ forces subspaces learned for different tasks to be mutually orthogonal. The IPC component then ranks each parameter matrix by its average importance score $\bar{I}\cdot\bar{U}$ and freezes the top-p matrices before the next task, holding fixed the parameters that mattered most for the current task. The $\omega$ coefficients are updated with a low learning rate to preserve some plasticity, which slightly relaxes the orthogonality constraint.

What would settle it

Track every matrix frozen after task $t$ through all later tasks: if any of its entries changes by more than numerical precision after training task $t+1$, the freeze is not being enforced and the claimed mechanism is not what produces the gains. A simpler check is to hold $\omega$ fixed at 1 for all past LoRA modules; if accuracy and forgetting barely change, the low-rate $\omega$ updates claimed to aid plasticity play no measurable role.

Watch

Extended reading notes

Core claim

The paper's central discovery is that LoRA-level orthogonality does not protect the parameters that matter: even when the columns of the learned LoRA factors are mutually orthonormal, the parameter matrices most sensitive to the current task's loss still change substantially after later tasks are learned. The authors attribute this to the low-rank nature of LoRA—the low-rank subspace cannot represent the original parameter space, so orthogonal low-rank solutions do not imply orthogonal full matrices. Their fix is to freeze, before each new task, the top-p parameter matrices selected by the importance score $S(w_{t,ij}) = \bar{I}(w_{t,ij}) \cdot \bar{U}(w_{t,ij})$, a product of exponentially smoothed gradient sensitivity and smoothed uncertainty, while composing past LoRA modules with learnable weights and QR-decomposed orthonormal factors. The reported outcome is rehearsal-free continual learning that surpasses previous methods on Split CIFAR-100, Split ImageNet-R, 5-datasets, and Split DomainNet, with the largest gains on Split CIFAR-100 (6.35% higher accuracy, 3.24% lower forgetting under Sup-21K).

Load-bearing premise

The whole gain rests on trusting the sensitivity-uncertainty score to pick the truly critical parameter matrices, and on assuming that freezing those matrices is not silently undone by the later low-learning-rate updates to the weighting coefficients $\omega$.

Editorial extensions

If this is right

  • On Split CIFAR-100 with the Sup-21K backbone, LoRAC-IPC reports 6.35% higher average accuracy and 3.24% lower forgetting than previous rehearsal-free methods.
  • On Split ImageNet-R it reports 1.24% higher accuracy than a strong projection-based baseline, and on the self-supervised MoCo-1K backbone it beats a full fine-tuning method by 2.39%.
  • On the 5-datasets benchmark it reaches 95.58% average accuracy with 0.03% forgetting (95.77% and 0.01% with batch-wise task-ID inference), close to the joint-training upper bound.
  • The ablation attributes 1.96% accuracy to LoRA composition, another 4.43% to the orthogonality loss, 0.45% to the important-parameter constraints, and 0.89% to task-ID inference on Split CIFAR-100.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper tests the freeze-first-then-compose recipe only with LoRA; the same importance-selection and freezing logic could plausibly be applied to adapters or prompts composed orthogonally, a transfer the paper does not claim.
  • Because the importance score is computed on mini-batches of the current task, the top-p selection may be noisy early in training; using importance estimated after convergence or with validation data is a natural robustness check the paper does not perform.
  • The method relies on task-ID inference at test time for its best numbers; in boundary-free or blurry continual learning where task boundaries are unknown, the gains would likely shrink unless the inference step is replaced.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LoRAC-IPC, a rehearsal-free continual learning method for pre-trained Vision Transformers. It combines sequentially learned LoRA modules in a weighted composition, imposes an orthogonality regularization on the QR-decomposed projection matrices, and proposes to freeze the most important parameter matrices (selected by a smoothed sensitivity–uncertainty score) before learning subsequent tasks. It also introduces a task-ID inference scheme and a task-adaptive classifier adjustment. On Split CIFAR-100, Split ImageNet-R, 5-datasets, Split DomainNet, and two multimodal datasets, the method reports state-of-the-art or competitive accuracy and forgetting.

Significance. If the mechanism is correctly specified, LoRAC-IPC is a simple and strong rehearsal-free baseline. The empirical study is extensive: three PTMs, four standard benchmarks, two multimodal datasets, ablations of rank, trade-off lambda, and component analysis. The paper provides code, and the central observation that orthogonal LoRA tuning does not prevent changes in important parameters is interesting and potentially useful. However, the description of the freezing mechanism is internally ambiguous, and the current text does not guarantee that the claimed constraints are operative as stated.

major comments (4)
  1. [Section 3.3, Eq. (1), Eq. (11)] The interaction between the IPC freezing and the global update of the omega coefficients is unspecified. Equation (1) defines every parameter matrix W as a sum over all LoRA modules scaled by omega. In task t+1, Section 3.3 states that omega_1,...,omega_t are updated with a low learning rate. Under Eq. (1), this update changes every previously frozen matrix unless an explicit mask or snapshot is used. Equation (11), the total training loss, contains no masking, stop-gradient, or constraint term that would prevent the new LoRA module Delta W_{t+1} from contributing to the frozen matrices, nor does it express the freezing as a constraint on the optimization. The authors must specify the exact update rules: are frozen matrices snapshotted and excluded from the composition for subsequent tasks, is there a per-matrix stop-gradient on the omega coefficients, or is the new LoRA module added only to non-frozen matrices? This is load-bearing because the central claim and the IPC versus Random result in Table 4 are attributed to the freezing mechanism.
  2. [Section 3.2, Eqs. (6)-(10)] The importance score is defined for a trainable parameter w_{t,ij}, but Eq. (9) averages it over the composed matrix W_t, whose elements are not themselves trainable parameters (they are sums of pre-trained weights and LoRA updates). The manuscript does not state whether the gradient in Eq. (6) is taken with respect to the composed matrix elements (via the chain rule through LoRA factors and omega coefficients) or with respect to the LoRA factors themselves, nor does it specify how the omega coefficients enter the importance computation. Without this clarification, the selection of the top-p matrices is not reproducible, and the comparison with the Random baseline in Table 4 does not establish that the importance estimates are meaningful.
  3. [Section 5.1.3 and Appendix A] The top-p percentage is set to 5% or 10% empirically, but the manuscript does not report which value is used for which dataset and backbone, nor does it provide a sensitivity analysis over p. Since p directly controls the number of frozen matrices and trades off plasticity against stability, the absence of this information makes it difficult to judge how robust the reported gains are to this hyperparameter. Please report the p values used in all experiments and add a sensitivity study (e.g., varying p as in the rank and lambda studies).
  4. [Section 4.1, Eq. (13)] The task-ID inference uses the feature extractor f(x, Theta_1), but Theta_1 is not defined as a stored snapshot. After task 1, IPC freezes only selected matrices, and later tasks update both the omega coefficients and the remaining matrices, so the current model after all tasks is not equal to Theta_1. Please clarify whether a copy of the model immediately after task 1 is saved and used for prototype and covariance estimation, or whether the current composed model is used. This matters because the reported accuracy with task-ID inference (e.g., 92.86 on Split CIFAR-100 in Table 1) depends on the correctness of the task-ID inference.
minor comments (5)
  1. [Section 3.2, Eq. (7)] The stated ranges 'beta1 > 0, beta2 < 1' are imprecise; for the exponential moving average to be stable, one normally requires 0 < beta1 < 1 and 0 < beta2 < 1. The experiments use 0.85 for both, so please correct the conditions.
  2. [Figure 1 and Section 3.2] The caption of Figure 1(b) describes the right columns as 'variation in important parameters', but the text defines importance via the sensitivity score while the variation is an L2 norm of parameter change. Please clarify the relationship between the two quantities, or revise the figure caption to match the text.
  3. [Table A.7] The header 'PTM Mehods' contains a typo, and the entries for Split DomainNet are missing for the Sup-21K backbone, which makes the reproducibility of the experimental setup incomplete. Please correct the header and report the weight types and ranks for all datasets used with each backbone.
  4. [Table 1 caption] The footnote markers '†' and '‡' appear next to several method names, but the caption does not unambiguously indicate which rows each marker applies to. Please make the footnotes explicit.
  5. [Section 5.3, Figure 5] The qualitative claim that 'with IPC, the positions of important parameters do not overlap largely' is supported only by visual inspection of the yellow boxes. A quantitative overlap measure (e.g., Jaccard index between the sets of top-p matrices across tasks) would strengthen this claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: LoRAC-IPC's accuracy and forgetting results are measured on external benchmarks, and no prediction reduces to a fitted constant; the only minor concern is in-house dataset self-citation.

full rationale

The derivation chain is empirical rather than formal. Importance scores (Eqs. 6-10) are computed from current-task loss gradients and used to select parameter matrices to freeze; this is a standard regularization heuristic, and the reported metrics (Avg. Acc, Forgetting) come from held-out test splits of external benchmarks (Split CIFAR-100, Split ImageNet-R, 5-datasets, Split DomainNet) against external baselines. There is no equation in which a predicted quantity equals a fitted input by construction, and no uniqueness theorem or prior result by the same authors is invoked to force the design. Citations to O-LoRA, InfLoRA, LoRA-FA, and PLATON are external building blocks. Two auxiliary experiments use in-house datasets (UESTC-MMEA-CL [53], ARIC [54]); these are not load-bearing for the central SOTA claims and do not make the derivation circular. A separate correctness concern is noted: Eq. (1) defines every composed matrix as W0 + sum_i omega_i Delta W_i, while Sec. 3.3 updates all previous omega coefficients with a low learning rate; without masks in code, this global update would alter matrices claimed to be frozen. This is an implementation/consistency ambiguity, not a circularity, because it does not make any reported result equivalent to its input by definition. Score 2 reflects only the minor self-citation of in-house datasets; the central derivation is self-contained against external benchmarks.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on two empirical heuristics: orthogonal LoRA composition reduces interference, and sensitivity-based importance identifies matrices worth freezing. Both are motivated by prior work but validated only through the paper's own experiments. There are no new physical entities; the free parameters are dataset-specific hyperparameters that are tuned per benchmark.

free parameters (5)
  • Orthogonal loss trade-off lambda = 1.0 (Split CIFAR-100, Sup-21K), 1e-6 (5-datasets), 0.01 (Split ImageNet-R), varies per dataset and PTM
    Chosen per dataset to balance cross-entropy and orthogonality; sensitivity is shown in Fig. 9 but the working values are dataset-specific.
  • LoRA rank R = 32 (CIFAR-100 Sup-21K), 16 (CIFAR-100 Sup-21K*/MoCo), 64 (ImageNet-R Sup-21K*), etc.
    Rank controls the low-rank subspace and is selected per dataset and PTM; Fig. 8 shows a sweet spot but no selection criterion is formalized.
  • Top-p important matrix fraction = 5% or 10%
    The percentage of ViT parameter matrices frozen after each task; stated as empirically selected, with no ablation over p.
  • Importance smoothing beta1, beta2 = 0.85
    Exponential moving average coefficients for importance and uncertainty, taken as defaults from PLATON [43].
  • Learning rate for omega coefficients = Not reported
    The weight coefficients on past LoRA modules are updated at a low learning rate; the exact value is not stated in the main text or appendix.
assumptions (6)
  • domain assumption Orthogonal LoRA tuning mitigates forgetting (OGD principle).
    The paper builds on [26,27,28], assuming orthogonality of LoRA subspaces reduces interference, but then shows it is insufficient.
  • domain assumption Sensitivity-based importance (Eq. 6, product of weight and gradient) identifies parameters critical to a task.
    Importance measure borrowed from PLATON [43]; assumed to rank parameter matrices by their effect on task loss.
  • domain assumption Freezing the top-p most important parameter matrices preserves pre-task performance without unduly harming plasticity.
    This is the core heuristic of IPC; the paper validates experimentally but provides no theoretical guarantee.
  • standard math QR decomposition of A_t yields a valid orthonormal basis for the column space of delta W_t.
    Used in Eq. 2 to reparameterize delta W_t = Q_t K_t.
  • domain assumption Class prototypes and Mahalanobis distance computed with Theta_1 reliably infer task identity.
    Task ID inference in Eq. 13 follows [42]; assumes the first-task feature extractor generalizes across tasks.
  • domain assumption Pre-trained ViT image representations are effective for downstream CL tasks.
    The method assumes a strong PTM (Sup-21K etc.) as base, a common assumption in PTM-based CL.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRA-Based Continual Learning with Constraints on Critical Parameter Changes." pith.science (2026). https://pith.science/paper/I7CPU6ZV

@misc{pith2026250413407,
  author       = {Pith},
  title        = {Pith review of: LoRA-Based Continual Learning with Constraints on Critical Parameter Changes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I7CPU6ZV}},
  note         = {Machine review of arXiv:2504.13407}
}
read the original abstract

LoRA-based continual learning represents a promising avenue for leveraging pre-trained models in downstream continual learning tasks. Recent studies have shown that orthogonal LoRA tuning effectively mitigates forgetting. However, this work unveils that under orthogonal LoRA tuning, the critical parameters for pre-tasks still change notably after learning post-tasks. To address this problem, we directly propose freezing the most critical parameter matrices in the Vision Transformer (ViT) for pre-tasks before learning post-tasks. In addition, building on orthogonal LoRA tuning, we propose orthogonal LoRA composition (LoRAC) based on QR decomposition, which may further enhance the plasticity of our method. Elaborate ablation studies and extensive comparisons demonstrate the effectiveness of our proposed method. Our results indicate that our method achieves state-of-the-art (SOTA) performance on several well-known continual learning benchmarks. For instance, on the Split CIFAR-100 dataset, our method shows a 6.35\% improvement in accuracy and a 3.24\% reduction in forgetting compared to previous methods. Our code is available at https://github.com/learninginvision/LoRAC-IPC.

Figures

Figures reproduced from arXiv: 2504.13407 by the authors.

Figure 1
Figure 1. The degree of variation in important parameters with orthogonal con [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Continual Learning with Orthogonal LoRA Composition and Important Parameter Constraints. The upper illustrates the workflow of Important Parameter Constraints (IPC). Upon completion of training for the current task, parameter matrices important to the current task are constrained to remain unchanged in continual learning. The lower shows the framework for Orthogonal LoRA Composition, consisting of three components: … view at source ↗
Figure 3
Figure 3. Parameter Adjustment for Task Adaptive Prediction. We use the fea [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Delta parameter absolute values of the model on each task. Based on the Sup-21k* pre-trained model learned sequentially on Split CIFAR-100 using [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The bar graphs on the left depict the variation in accuracy of LoRAC and LoRAC-IPC across various tasks on Split CIFAR-100. The right half shows [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: t-SNE visualization of representation with LoRA-FT (left) and with [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 9
Figure 9. Figure 9: The average accuracy and forgetting of LoRAC w/ [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 7
Figure 7. Figure 7: Comparison of Nearest Mean Classifier (NMC), LoRA-FT, LoRAC [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Results of LoRAC w/o TII on Split CIFAR-100 with different rank R. 0.75, 1.0, and 10.0. While for the 5-datasets, the trade-off val￾ues are set to 0.0001, 0.001, 0.01, 0.1, 1.0, and 10. The fig￾ure illustrates a relatively consistent change in the performance across bo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 43 canonical work pages

  1. [1]

    Thrun, A lifelong learning perspective for mobile robot control, in: Intelligent robots and systems, Elsevier, 1995, pp

    S. Thrun, A lifelong learning perspective for mobile robot control, in: Intelligent robots and systems, Elsevier, 1995, pp. 201–214

    work page 1995

  2. [2]

    Z. Chen, B. Liu, Lifelong machine learning, Synthesis Lectures on Arti- ficial Intelligence and Machine Learning 12 (3) (2018) 1–207

    work page 2018

  3. [3]

    McCloskey, N

    M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in: Psychology of learning and motivation, V ol. 24, Elsevier, 1989, pp. 109–165

    work page 1989

  4. [4]

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, R. Hadsell, Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016)

    arXiv 2016

  5. [5]

    Fernando, D

    C. Fernando, D. Banarse, C. Blundell, Y . Zwols, D. Ha, A. A. Rusu, A. Pritzel, D. Wierstra, Pathnet: Evolution channels gradient descent in super neural networks, arXiv preprint arXiv:1701.08734 (2017)

    arXiv 2017

  6. [6]

    W. Sun, Q. Li, J. Zhang, D. Wang, W. Wang, Y .-a. Geng, Exemplar- free class incremental learning via discriminative and comparable parallel one-class classifiers, Pattern Recognition 140 (2023) 109561

    work page 2023

  7. [7]

    Z. Fu, Z. Wang, X. Xu, D. Li, H. Yang, Knowledge aggregation networks for class incremental learning, Pattern Recognition 137 (2023) 109310

    work page 2023

  8. [8]

    Zenke, B

    F. Zenke, B. Poole, S. Ganguli, Continual learning through synaptic intel- ligence, in: International Conference on Machine Learning, PMLR, 2017, pp. 3987–3995

    work page 2017

Show all 54 references
  1. [9]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Over- coming catastrophic forgetting in neural networks, Proceedings of the na- tional academy of sciences 114 (13) (2017) 3521–3526

    2017

  2. [10]

    R. Wu, H. Liu, Z. Yue, J.-B. Li, C.-W. Sham, Hyper-feature aggregation and relaxed distillation for class incremental learning, Pattern Recogni- tion 152 (2024) 110440

    2024

  3. [11]

    X. Li, S. Wang, J. Sun, Z. Xu, Memory e fficient data-free distillation for continual learning, Pattern Recognition 144 (2023) 109875

    2023

  4. [12]

    Rolnick, A

    D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, G. Wayne, Experience replay for continual learning, Advances in Neural Information Processing Systems 32 (2019)

    2019

  5. [13]

    Riemer, I

    M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y . Tu, G. Tesauro, Learning to learn without forgetting by maximizing transfer and mini- mizing interference, arXiv preprint arXiv:1810.11910 (2018)

    2018 arXiv

  6. [14]

    Buzzega, M

    P. Buzzega, M. Boschini, A. Porrello, D. Abati, S. Calderara, Dark expe- rience for general continual learning: a strong, simple baseline, Advances in neural information processing systems 33 (2020) 15920–15930

    2020

  7. [15]

    J. Song, J. Chen, L. Du, Rebalancing network with knowledge stability for class incremental learning, Pattern Recognition 153 (2024) 110506

    2024

  8. [16]

    Z. Wang, Z. Zhang, C.-Y . Lee, H. Zhang, R. Sun, X. Ren, G. Su, V . Perot, J. Dy, T. Pfister, Learning to prompt for continual learning, in: Pro- ceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 139–149

    2022

  9. [17]

    Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y . Lee, X. Ren, G. Su, V . Perot, J. Dy, et al., Dualprompt: Complementary prompting for rehearsal-free continual learning, in: European Conference on Computer Vision, Springer, 2022, pp. 631–648

    2022

  10. [18]

    Y . Wang, Z. Huang, X. Hong, S-prompts learning with pre-trained trans- formers: An occam’s razor for domain incremental learning, Advances in Neural Information Processing Systems 35 (2022) 5682–5695

    2022

  11. [19]

    J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Ar- belle, R. Panda, R. Feris, Z. Kira, Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning, in: Pro- ceedings of the IEEE /CVF Conference on Computer Vision and...

    2023

  12. [20]

    L. Wang, J. Xie, X. Zhang, M. Huang, H. Su, J. Zhu, Hierarchical decom- position of prompt-based continual learning: Rethinking obscured sub- optimality, Advances in Neural Information Processing Systems (2023)

    2023

  13. [21]

    Q. Gao, C. Zhao, Y . Sun, T. Xi, G. Zhang, B. Ghanem, J. Zhang, A unified continual learning framework with general parameter-efficient tuning, in: Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, 2023, pp. 11483–11493

    2023

  14. [22]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-e fficient transfer learn- ing for nlp, in: International Conference on Machine Learning, PMLR, 2019, pp. 2790–2799

    2019

  15. [23]

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models, in: International Conference on Learning Representations, 2022

    2022

  16. [24]

    Lester, R

    B. Lester, R. Al-Rfou, N. Constant, The power of scale for parameter- efficient prompt tuning, arXiv preprint arXiv:2104.08691 (2021)

    2021 arXiv

  17. [25]

    X. L. Li, P. Liang, Prefix-tuning: Optimizing continuous prompts for gen- eration, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (V olume 1: Long Papers),...

    2021

  18. [26]

    X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, X.-J. Huang, Orthogonal subspace learning for language model continual learning, in: Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 10658–10671

    2023

  19. [27]

    Liang, W.-J

    Y .-S. Liang, W.-J. Li, Inflora: Interference-free low-rank adaptation for continual learning, in: Proceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 23638–23647

    2024

  20. [28]

    Farajtabar, N

    M. Farajtabar, N. Azizan, A. Mott, A. Li, Orthogonal gradient descent for continual learning, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 3762–3773

    2020

  21. [29]

    L. Wang, X. Zhang, H. Su, J. Zhu, A comprehensive survey of continual learning: theory, method and application, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  22. [30]

    G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural networks 113 (2019) 54–71

    2019

  23. [31]

    Zhang, L

    L. Zhang, L. Zhang, S. Shi, X. Chu, B. Li, Lora-fa: Memory-e fficient low-rank adaptation for large language models fine-tuning, arXiv preprint arXiv:2308.03303 (2023)

    2023 arXiv

  24. [32]

    Zhang, M

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, T. Zhao, Adaptive budget allocation for parameter-e fficient fine-tuning, in: The Eleventh International Conference on Learning Representations, 2023

    2023

  25. [33]

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, S.- N. Lim, Visual prompt tuning, in: European Conference on Computer Vision, Springer, 2022, pp. 709–727

    2022

  26. [34]

    Z. Li, L. Zhao, Z. Zhang, H. Zhang, D. Liu, T. Liu, D. N. Metaxas, Steer- ing prototype with prompt-tuning for rehearsal-free continual learning, arXiv preprint arXiv:2303.09447 (2023)

    2023 arXiv

  27. [35]

    Huang, C.-F

    W.-C. Huang, C.-F. Chen, H. Hsu, OVOR: OnePrompt with virtual outlier regularization for rehearsal-free class-incremental learning, in: Interna- tional Conference on Learning Representations, 2024

    2024

  28. [36]

    J. Qiao, X. Tan, C. Chen, Y . Qu, Y . Peng, Y . Xie, et al., Prompt gradient projection for continual learning, in: The Twelfth International Confer- ence on Learning Representations, 2023

    2023

  29. [37]

    Z. Gao, J. Cen, X. Chang, Consistent prompting for rehearsal-free contin- ual learning, in: Proceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28463–28473

    2024

  30. [38]

    A. Roy, R. Moulick, V . K. Verma, S. Ghosh, A. Das, Convolutional prompting meets language models for continual learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2024, pp. 23616–23626

    2024

  31. [39]

    Zhou, H.-J

    D.-W. Zhou, H.-J. Ye, D.-C. Zhan, Z. Liu, Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need, arXiv preprint arXiv:2303.07338 (2023)

    2023 arXiv

  32. [40]

    Zhang, L

    G. Zhang, L. Wang, G. Kang, L. Chen, Y . Wei, Slca: Slow learner with classifier alignment for continual learning on a pre-trained model, in: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 19148–19158

    2023

  33. [41]

    Zhou, H.-L

    D.-W. Zhou, H.-L. Sun, H.-J. Ye, D.-C. Zhan, Expandable subspace en- semble for pre-trained model-based class-incremental learning, in: Pro- ceedings of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 23554–23564

    2024

  34. [42]

    M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, A. van den Hengel, Ranpac: Random projections and pre-trained models for con- tinual learning, Advances in Neural Information Processing Systems 36 (2024)

    2024

  35. [43]

    Zhang, S

    Q. Zhang, S. Zuo, C. Liang, A. Bukharin, P. He, W. Chen, T. Zhao, Platon: Pruning large transformer models with upper confidence bound of weight importance, in: International Conference on Machine Learning, PMLR, 2022, pp. 26809–26823

    2022

  36. [44]

    Krizhevsky, Learning multiple layers of features from tiny images, Master’s thesis, University of Tront (2009)

    A. Krizhevsky, Learning multiple layers of features from tiny images, Master’s thesis, University of Tront (2009). 13

    2009

  37. [45]

    Hendrycks, S

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, et al., The many faces of robustness: A critical analysis of out-of-distribution generalization, in: Proceedings of the IEEE /CVF International Conference on Computer Visio...

    2021

  38. [46]

    Ridnik, E

    T. Ridnik, E. Ben-Baruch, A. Noy, L. Zelnik-Manor, Imagenet-21k pre- training for the masses, arXiv preprint arXiv:2104.10972 (2021)

    2021 arXiv

  39. [47]

    Ebrahimi, F

    S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, M. Rohrbach, Adversarial continual learning, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, Springer, 2020, pp. 386–402

    2020

  40. [48]

    X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, B. Wang, Moment match- ing for multi-source domain adaptation, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406–1415

    2019

  41. [49]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An im- age is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2020

    2020

  42. [50]

    Janson, W

    P. Janson, W. Zhang, R. Aljundi, M. Elhoseiny, A simple baseline that questions the use of pretrained-models in continual learning, in: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Appli- cations, 2022

    2022

  43. [51]

    Z. Li, D. Hoiem, Learning without forgetting, IEEE transactions on pat- tern analysis and machine intelligence 40 (12) (2017) 2935–2947

    2017

  44. [52]

    J. S. Smith, Y .-C. Hsu, L. Zhang, T. Hua, Z. Kira, Y . Shen, H. Jin, Con- tinual diffusion: Continual customization of text-to-image di ffusion with c-lora, Transactions on Machine Learning Research (2023)

    2023

  45. [53]

    L. Xu, Q. Wu, L. Pan, F. Meng, H. Li, C. He, H. Wang, S. Cheng, Y . Dai, Towards continual egocentric activity recognition: A multi-modal ego- centric activity dataset for continual learning, IEEE Transactions on Mul- timedia (2023)

    2023

  46. [54]

    L. Xu, F. Meng, Q. Wu, L. Pan, H. Qiu, L. Wang, K. Chen, K. Geng, Y . Qian, H. Wang, et al., Aric: An activity recognition dataset in class- room surveillance images, arXiv preprint arXiv:2410.12337 (2024). 14

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.